The problem: internal covariate shift

As a network trains, the distribution of activations at each layer keeps shifting because the parameters of every earlier layer are changing too. Each layer is effectively trying to learn against a constantly moving target, which slows training and forces the use of small, cautious learning rates. Batch normalization stabilizes this by explicitly re-centering and re-scaling activations to have zero mean and unit variance — computed fresh from the current mini-batch — at every forward pass.

The forward-pass formula

For a single feature, computed across all B samples in a batch:

μ  = (1/B) Σ x_i                    # batch mean
σ² = (1/B) Σ (x_i - μ)²             # batch variance (population, divide by B)
x̂_i = (x_i - μ) / √(σ² + ε)         # normalize to zero mean, unit variance
y_i = γ · x̂_i + β                   # learnable scale & shift

Two details are easy to get wrong:

1. Population variance, not sample variance. Divide by B, not B-1. This is a common source of subtle numerical mismatches when porting between frameworks or hand-rolling the formula.
2. γ and β are learned parameters, not fixed constants. If normalization always forced zero mean/unit variance, the network would lose the ability to represent, say, a feature that's more useful with a non-zero mean. γ/β let the network undo the normalization if that's what minimizes the loss — they give it the option to recover the original distribution, while still getting all of BN's optimization benefits (stable gradients, faster convergence, some regularization effect from the batch-level noise).

Why divide by the batch, not per-sample

Batch norm normalizes each feature across the batch dimension — i.e., for a fixed feature/channel, look at how that one number varies across all the examples currently in the mini-batch. This is different from layer norm (covered in the next lesson), which normalizes across the feature dimension for a single example instead. Mixing these two up is the single most common bug when implementing normalization layers.

Numerical stability: epsilon

ε (typically 1e-5) is added inside the square root purely to prevent division by zero when a batch happens to have (near) zero variance — e.g., if every sample in the batch had the exact same value for a feature. It's small enough to be negligible in the normal case but keeps the computation well-defined always.

What you're implementing

This lesson's exercise strips batch norm down to a single feature (a 1-D batch of scalars) so you can trace the arithmetic exactly:

1. Compute the batch mean and (population) variance.
2. Normalize every element using those batch statistics.
3. Apply the affine transform γ·x̂ + β.

In a full network, this same computation runs independently for every channel/feature, and at inference time the "batch" mean/variance is replaced by a running average accumulated during training (since you often don't have a full batch, or even any batch, at inference time) — but the forward-pass math for a single feature, which this exercise covers, is identical either way.