Xavier & Kaiming Initialization

Before the first gradient step, every weight needs a starting value, and a bad choice can ruin training before it begins. After this lesson your program computes the standard initialization scales and predicts whether a deep stack of layers will explode or fade.

Three ways to start badly

• All zeros. Every neuron in a layer computes the same function and receives the same gradient, so they stay identical forever. Random values are needed to break the symmetry.
• Too large. Activations grow layer after layer; saturating functions like tanh pin at +-1 where their gradient is zero, or the numbers overflow.
• Too small. Activations shrink toward zero layer after layer, and so do the gradients. The deep layers barely learn.

The variance argument

Take a layer whose output is y = w1*x1 + ... + wn*xn, with n inputs of variance Var(x) and independent weights of variance Var(w). Then

Var(y) = n * Var(w) * Var(x)

So each layer multiplies the signal's variance by the gain n * Var(w). Over L layers the variance is multiplied by gain^L. A gain of 1 keeps the signal steady, 1.1 makes it explode (1.1^50 is about 117), 0.9 makes it fade. The goal of initialization is gain = 1.

Xavier (Glorot) and Kaiming (He)

Xavier initialization balances the forward and backward passes by averaging the fan-in n_in and fan-out n_out, and suits symmetric activations like tanh:

Var(w) = 2 / (n_in + n_out)
normal:   std = sqrt( 2 / (n_in + n_out) )
uniform:  bound a = sqrt( 6 / (n_in + n_out) ), weights drawn from [-a, a]

A uniform distribution on [-a, a] has variance a^2 / 3, which is where the 6 comes from.

ReLU zeroes half of its inputs, so only half the signal's second moment survives each layer: the gain becomes n * Var(w) / 2. Kaiming initialization compensates by doubling the variance and using only the fan-in:

normal:   std = sqrt( 2 / n_in )
uniform:  bound a = sqrt( 6 / n_in )

A worked trace

A stack of ReLU layers, each 100 wide (n_in = n_out = 100).

Xavier:   std = sqrt(2/200) = 0.1000   gain = 100 * 0.01 * 0.5 = 0.5
          after 10 layers the variance is 0.5^10 = 0.001 of the input's: the signal has faded
Kaiming:  std = sqrt(2/100) = 0.1414   gain = 100 * 0.02 * 0.5 = 1.0
          after 10 layers: 1.0

And a uniform bound for a different shape, n_in = 50, n_out = 150: Xavier gives sqrt(6/200) = 0.1732.

Without the ReLU factor (a linear stack) Kaiming's gain would be 2, so 10 layers multiply the variance by 1024. That is why the right scheme depends on the activation.

Common mistakes

• Swapping the constants. Both schemes use 2 for the normal's variance and 6 for the uniform bound; only the denominator differs (n_in + n_out for Xavier, n_in for Kaiming).
• Forgetting the square root, which returns a variance instead of a standard deviation.
• Using the uniform bound as a standard deviation.
• Applying Kaiming's gain without the ReLU halving, or Xavier with ReLU and then wondering why deep nets fade.

Your exercise

Six commands. Four return a scale, and SIGNAL returns the variance multiplier of a stack of identical layers. The starter parses and prints; implement the five functions.