Dropout: Stochastic Regularization

Dropout switches off a random part of the network on every training step, so no neuron can rely on a particular partner being present. After this lesson your program produces a reproducible dropout mask from a seed, and applies it with the correct rescaling.

The idea

Large networks co-adapt: neuron A learns to fix the mistakes of neuron B, and the pair fits the training data in a way that does not generalise. Dropout breaks the partnership by zeroing each activation independently with probability p on every forward pass during training. Each step trains a different thinned network, and the final network behaves like an average of exponentially many of them.

Inverted dropout

If a fraction p of activations is zeroed, the expected total activation shrinks by the factor 1 - p. There are two ways to compensate. Inverted dropout does it during training:

training:   y = 0               if the element is dropped
            y = x / (1 - p)     if it is kept
evaluation: y = x               (nothing happens)

Check the expectation with x = 1 and p = 0.5: half the time the output is 0, half the time it is 2, so the average is 1, exactly what evaluation produces. Because the scaling lives in training, the deployed network needs no special code at all. Every modern framework does it this way.

Making randomness reproducible

Tests need the same mask for the same seed on every machine, so this exercise uses a linear congruential generator (LCG), the simplest pseudo-random source:

state  <-  ( state * 1664525 + 1013904223 )  mod 2^32
u      =   ( state >> 8 ) / 2^24             a number in [0, 1)

Start with state = seed mod 2^32, advance once, and take one u per call. The shift by 8 uses the high 24 bits, which are far better distributed than the low bits of an LCG. Use unsigned 32-bit arithmetic or a mask so the product wraps correctly; in a language with overflow checks, use wrapping operations.

Trace with seed 10:

state 1030549473  ->  u = 4025583 / 16777216 = 0.2399
state  797165516  ->  u = 3113927 / 16777216 = 0.1856
state 2863095995  ->  u = 11183968 / 16777216 = 0.6666
state  163339998  ->  u = 638046 / 16777216 = 0.0380

From draws to a mask

Element i uses the i-th draw. It is dropped when u < p and kept otherwise. With p = 0.2 the four draws above give the mask 1 0 1 0 (kept, dropped, kept, dropped). Apply it to x = [4, 8, 2, 6] with scale 1 / 0.8 = 1.25:

output = [ 4*1.25, 0, 2*1.25, 0 ] = [5, 0, 2.5, 0]

Edge cases and common mistakes

• p = 0: nothing is dropped and the scale is 1, so the output equals the input.
• p = 1: everything is dropped. The scale 1 / (1 - p) would divide by zero, so only compute it for a kept element (there are none).
• Evaluation mode must not touch the generator or scale anything.
• Scaling by 1 / p instead of 1 / (1 - p), or dropping when u > p.
• Shifting the sequence by one draw, or restarting the generator mid-vector. Draw exactly one number per element, in order.
• Forgetting to switch dropout off when serving a model. This is the most common deployment bug.

Your exercise

Commands set INPUT, P and SEED, then ask for TRAIN, EVAL, MASK or RAND. Every TRAIN, MASK and RAND starts a fresh generator from the seed. Implement uniforms, mask and apply_dropout.