Why plain SGD isn't enough

Vanilla gradient descent updates a parameter with theta -= lr * grad. That works, but it has two problems in practice:

• Noisy gradients (from mini-batches) make the trajectory zig-zag instead of heading straight for a minimum.
• A single global learning rate is a poor fit for every parameter — some directions of the loss surface are steep, others are almost flat, and a fixed step size is either too aggressive for the steep ones or too timid for the flat ones.

Adam (Adaptive Moment Estimation) fixes both by keeping a running estimate of the gradient's mean (first moment) and its uncentered variance (second moment), and using them to rescale every parameter's step individually.

The two moving averages

For each parameter, Adam tracks two exponential moving averages of the gradient g:

m = β1·m + (1-β1)·g        # momentum — smoothed direction
v = β2·v + (1-β2)·g²       # smoothed magnitude of the gradient

m behaves like classical momentum: it dampens oscillation and keeps moving in a consistent direction even when individual gradients are noisy. v accumulates how large the gradient has recently been (element-wise squared), which lets Adam shrink the step in directions where gradients are consistently large, and take bigger relative steps where they're small.

Bias correction

Because m and v start at zero, they are biased toward zero during the first few steps (early on, m is mostly 0 scaled by (1-β1), not the true average). Adam corrects for this using the step count t:

m̂ = m / (1 - β1^t)
v̂ = v / (1 - β2^t)

As t grows, β1^t and β2^t shrink toward zero, so the correction fades out and m̂ ≈ m, v̂ ≈ v. Without this correction, the optimizer would take artificially small steps at the very start of training.

The parameter update

theta = theta - lr · m̂ / (√v̂ + ε)

ε (typically 1e-8) exists purely to prevent division by zero when v̂ is tiny. Notice the update is not scaled by the raw gradient magnitude directly — dividing by √v̂ normalizes the step so that parameters with historically large gradients get smaller effective steps, and vice versa. This is what makes Adam "adaptive" per-parameter.

Implementation shape

Your optimizer needs persistent state across steps: theta, m, v, and t, plus hyperparameters lr, β1, β2, ε. Each STEP <grad> call must, in order:

1. Increment t.
2. Update m and v with the new gradient.
3. Compute bias-corrected m̂, v̂.
4. Apply the update to theta.

A common bug is computing β1^t using the pre-increment t — make sure you increment the step counter before computing the correction terms, since the first step should use t=1, not t=0 (which would make 1 - β1^0 = 0 and divide by zero).

This lesson works with a single scalar parameter so you can trace every number by hand, but the exact same four-line update is applied element-wise to every weight and bias in a real network — this is the optimizer almost every modern deep learning framework defaults to.