Why regularize at all

A network with enough capacity can drive training loss to near zero by memorizing the training set — fitting noise instead of the underlying pattern. L2 regularization (a.k.a. weight decay in its simplest form) discourages this by penalizing large weights, pushing the model toward smaller, smoother weight values that tend to generalize better.

The L2 penalty term

We add a penalty to the loss function proportional to the sum of squared weights:

L_total = L_data + λ/2 · Σ w_i²

λ (lambda) controls how strongly we penalize large weights — λ = 0 disables regularization entirely; larger λ shrinks weights more aggressively. The 1/2 factor is a convention that makes the derivative clean (it cancels the 2 that falls out of differentiating a square).

Gradient of the penalty

Since we add λ/2 · w_i² to the loss for every weight, the gradient contribution for each weight is:

∂/∂w_i (λ/2 · w_i²) = λ · w_i

In practice this means: whenever you compute a weight's gradient from backprop, you add λ · w_i to it before the optimizer step. Left unregularized, only the data-driven "pull" acts on a weight; with L2, there's also a constant force pulling every weight toward zero in proportion to its own current value.

Decoupled weight decay

There's a subtlety worth understanding: implementing L2 as "add λ·w to the gradient, then let the optimizer do its normal update" is not the same as directly shrinking the weight. For plain SGD they're equivalent, but for adaptive optimizers like Adam (which rescale gradients by a running second-moment estimate), mixing the penalty into the gradient interacts with that rescaling in unintended ways.

Decoupled weight decay sidesteps this by applying the shrinkage directly to the weight, independent of the gradient-based update:

w ← w - lr · λ · w      # equivalent to: w ← w · (1 - lr·λ)

This is a pure multiplicative shrink — every weight loses a fixed fraction of its magnitude each step, regardless of what the optimizer's adaptive scaling does with the data gradient. This is the idea behind AdamW, one of the most common optimizer configurations in modern training.

What you're building

Your exercise computes all three pieces independently so you can see how they relate:

• REG_LOSS — the scalar penalty value added to the loss.
• REG_GRAD — the vector you'd add to each weight's backprop gradient (classic L2-via-gradient).
• WD_STEP — the direct multiplicative shrink (decoupled weight decay), which doesn't touch the gradient at all.

Watch the order of operations in WD_STEP: it's w - lr·λ·w, not w - lr·λ. A common bug is treating λ like a fixed decrement rather than a proportional shrink — decoupled weight decay always scales with the weight's own current value, so it never pushes a weight past zero or flips its sign in one step.