Why initialization matters

Before any training happens, weights have to start somewhere. Get it wrong and the network can fail before the first gradient step ever helps:

• Too large: activations grow layer over layer, saturating nonlinearities (e.g. tanh/sigmoid output near ±1, where gradients vanish) or overflowing entirely.
• Too small: activations shrink toward zero as they pass through each layer, and gradients vanish during backprop — deep layers barely learn.
• All zeros: every neuron in a layer computes the exact same function and receives the exact same gradient — the network can never break the symmetry between them.

The goal of a good initialization scheme is to pick a scale for the random weights such that the variance of activations (and gradients) stays roughly constant as signals flow through many layers.

Xavier/Glorot initialization

Designed for layers followed by symmetric activations like tanh or sigmoid. The idea: if a layer has fan_in inputs and fan_out outputs, keeping the variance of the output equal to the variance of the input (in both the forward and backward directions) requires scaling weights by a factor derived from fan_in + fan_out:

uniform bound:  U(-a, a),  a = √(6 / (fan_in + fan_out))
normal std:     σ = √(2 / (fan_in + fan_out))

Both formulas target the same variance — 2 / (fan_in + fan_out) — just expressed for two different distributions. (For a uniform distribution on [-a, a], the variance is a²/3; solving a²/3 = 2/(fan_in+fan_out) gives the 6 inside the square root.)

Kaiming/He initialization

tanh/sigmoid are symmetric around zero, but ReLU zeroes out half its input on average, which cuts the effective variance in half. Kaiming initialization (He et al.) accounts for this by using only fan_in and doubling the constant:

uniform bound:  U(-a, a),  a = √(6 / fan_in)
normal std:     σ = √(2 / fan_in)

This is the standard choice for networks using ReLU or its variants (LeakyReLU, GELU, etc.) — which is most modern feedforward and convolutional networks.

What to keep in mind

• These formulas only compute the scale (the bound of a uniform range, or the standard deviation of a normal distribution) — the actual sampling still requires a random draw within that scale. The exercise focuses purely on getting the scale formula right, since that's the part learners most often mix up.
• Xavier uses both fan_in and fan_out because it's balancing variance preservation in both the forward pass and the backward pass simultaneously. Kaiming uses only fan_in because it's derived assuming you care primarily about the forward pass variance (through ReLU), though a fan_out variant exists too.
• Double-check which formula uses 6 and which uses 2 — it's easy to transpose them. As a sanity check: Kaiming's constants are always Xavier's constants with fan_out dropped, not doubled from scratch — both use 6 for the uniform bound and 2 for the normal std, only the denominator changes.

Getting initialization right is one of the cheapest wins in deep learning: it costs nothing at inference time, requires no extra hyperparameter search, and can be the difference between a network that trains in minutes and one that never converges at all.