Normalizing across features instead of across the batch

Batch normalization (previous lesson) has a weakness: it needs a reasonably sized batch to compute a meaningful mean and variance, and its behavior at train time (batch statistics) differs from inference time (running statistics). That mismatch and batch-size dependency makes it awkward for sequence models like RNNs and Transformers, where batches can be small, variable-length, or (during generation) size 1.

Layer normalization solves this by normalizing across the feature dimension of a single sample, instead of across the batch. Every example is normalized independently, using only its own features — no dependency on batch size, and identical behavior in training and inference.

The formula

For one sample with D features:

μ  = (1/D) Σ x_i                          # mean over features
σ² = (1/D) Σ (x_i - μ)²                    # variance over features
y_i = γ_i · (x_i - μ)/√(σ² + ε) + β_i      # per-feature scale & shift

Compare this directly to batch norm:

The formula shapes look almost identical — mean, variance, normalize, scale-and-shift — but the axis you reduce over is completely different, and that's the entire conceptual difference between the two techniques.

Why γ and β are per-feature vectors here

Unlike the single-feature batch norm exercise, layer norm's γ and β are vectors of length D — one scale and one shift per feature. Since layer norm computes one mean/variance per sample but applies the affine transform per feature, each feature index i needs its own learnable γ_i, β_i so the network can still express "this feature should have larger variance than that one" even after normalization forces every feature into the same distribution within a sample.

Implementation notes

• If GAMMA/BETA are never set, default to all-ones and all-zeros respectively (i.e., normalization only, no additional affine transform).
• The mean and variance are computed once per sample, using every feature of that sample — not per-batch, and not per-feature-across-samples. If you're processing a batch of multiple samples, you'd repeat this whole computation independently for each row.
• Same numerical-stability role for ε as in batch norm: it prevents a divide-by-zero when a sample's features happen to all be equal (variance = 0).

Why it matters for Transformers

Layer normalization is the default normalization used throughout Transformer architectures (both the original "Attention Is All You Need" formulation and virtually every LLM since) precisely because sequence lengths and batch composition vary, and because at inference/generation time you're often normalizing a single token's activations at a time — exactly the regime where batch norm would need special-cased running statistics but layer norm just works, unchanged, because it never looked at the batch dimension in the first place.