Layer Normalization

LayerNorm normalizes activations within a single token across the embedding dimension.

Per token: subtract mean, divide by std. Then scale + shift with learned gamma, beta.

Why?

• Stabilizes training.
• Prevents activations from drifting to extreme values.
• Each layer sees inputs with similar statistics.

Compare to BatchNorm:

• BatchNorm: normalize across batch dimension. Used in CNNs.
• LayerNorm: normalize across feature dimension. Used in NLP/Transformers.

LN doesn't depend on batch size → works for any batch size at inference (even batch=1).

Pre-norm vs post-norm:

• Post-norm (original Transformer): LN AFTER residual.
  • x = x + Attention(x); x = LN(x)
• Pre-norm (modern): LN BEFORE residual.
  • x = x + Attention(LN(x))

Pre-norm is more stable for deep networks (100+ layers). Standard now.

RMSNorm:

• Skip the mean centering, only scale by RMS.
• Simpler, ~similar effect.
• Used by Llama.

Why these matter:

• Without normalization: deep networks blow up or vanish.
• With it: 100+ layer transformers train stably.

Normalization is one of those "simple math" techniques whose impact was hard to predict and harder to overstate.