L2 Regularization & Weight Decay

A network with enough capacity can memorise its training set, noise included. L2 regularization pushes it toward small weights, which tend to give smoother functions that generalise. After this lesson your program computes the penalty, its gradient, and the two ways of shrinking weights during training.

The penalty

Add a term to the loss that grows with the squared size of the weights:

L_total = L_data + (lambda / 2) * ( w1^2 + w2^2 + ... )

lambda sets how much you care: 0 switches the penalty off, larger values squeeze harder. The 1/2 is a convention that cancels the 2 produced by differentiating a square. Biases are usually left out of the penalty.

Its gradient is a pull toward zero

Differentiating (lambda / 2) * w^2 gives lambda * w. Backprop already supplies the data gradient g for each weight; regularizing means adding lambda * w to it before the update:

w  <-  w - lr * ( g + lambda * w )

Read the extra term as a spring attached to zero: it pulls each weight back in proportion to its own size, so large weights are pulled hard and tiny ones barely at all. Weights shrink but do not become exactly zero. (The L1 penalty, lambda * sum |w|, pulls with constant force and does drive weights to exact zeros, which is why it produces sparse models.)

Why it is also called weight decay

Set the data gradient to zero and the step becomes w <- w - lr * lambda * w = w * (1 - lr * lambda): every weight is multiplied by a number just under 1. The weights decay geometrically. For plain SGD, "add lambda*w to the gradient" and "multiply by 1 - lr*lambda" are the same operation.

Decoupled weight decay

They stop being the same for adaptive optimizers such as Adam. Adam divides the whole gradient, penalty included, by the running size of the gradients. If a weight's gradients are around 100 and its penalty term is 0.01, the penalty contributes about one ten-thousandth of the step: heavily-used weights are barely regularized. The fix, used in AdamW, is to decouple the two: let the optimizer handle g, and apply the shrink w <- w - lr * lambda * w separately, so every weight decays by the same fraction regardless of its gradient history. This is why AdamW is the standard for transformers.

A worked trace

w = [2, -1], lambda = 0.5.

penalty   = 0.5/2 * (4 + 1)       = 1.25
gradient  = 0.5 * [2, -1]         = [1.0, -0.5]
decay     with lr = 0.1:  w * (1 - 0.1*0.5) = w * 0.95 = [1.9, -0.95]
L2 step   with data gradient g = [0.3, 0.3], lr = 0.1:
          2  - 0.1*(0.3 + 0.5*2)    = 1.87
          -1 - 0.1*(0.3 + 0.5*(-1)) = -0.98

Notice the second weight: its data gradient pushes it down, the penalty pulls it up, and the net move is toward zero.

Common mistakes

• Subtracting a constant (w - lr*lambda) instead of a fraction of w. Decay is multiplicative and never flips a sign.
• Forgetting the 1/2 in the penalty value, or putting it on the gradient.
• Penalising the bias, which this course does not.
• Using the penalty value where the gradient is needed (or the reverse).
• Updating the weights but not remembering them for the next command.

Your exercise

The input sets WEIGHTS and LAMBDA, then asks for REG_LOSS, REG_GRAD, WD_STEP (decay only) and L2_STEP (decay plus a data gradient). Steps change the stored weights. Implement the four functions.