The idea behind dropout

Large networks can co-adapt: neurons learn to rely on very specific combinations of other neurons, which makes the whole system brittle to noise and prone to overfitting. Dropout breaks this up by randomly zeroing out a fraction of activations on every forward pass during training, forcing the network to learn redundant, robust representations that don't depend on any single unit being present.

At test time we want a deterministic, full-strength network — no randomness. The challenge is making training-time and inference-time outputs consistent in expected magnitude.

Inverted dropout

There are two ways to implement dropout, and modern frameworks universally use inverted dropout:

1. At train time, for each activation x, draw a random u ~ Uniform(0,1). If u < p (the drop probability), zero it out. Otherwise, keep it but rescale by 1/(1-p).
2. At eval time, do nothing — pass activations through unchanged.

train: y_i = 0            if u_i < p
       y_i = x_i / (1-p)  otherwise
eval:  y_i = x_i

Rescaling by 1/(1-p) during training is the "inverted" part: it keeps the expected sum of activations the same whether or not dropout is active, so no correction is needed at inference time. (The older, non-inverted approach scales at eval time instead — inverted dropout is preferred because it keeps eval-time code path simple and fast.)

Making randomness reproducible

Real dropout uses a proper random number generator, but for testing and grading we need determinism: the same seed must always produce the same mask. This exercise uses a classic linear congruential generator (LCG) — the same technique behind many textbook rand() implementations:

state = (state * 1664525 + 1013904223) mod 2^32
u = (state >> 8) / 2^24        # squeeze the top bits into [0, 1)

Each call advances state and derives one uniform value. Using the upper bits (via >> 8) rather than the low bits is standard LCG practice — the low-order bits of an LCG have short, poor-quality periodicity, while the high-order bits are much better distributed.

Building the mask correctly

For a vector of n inputs, you need to draw exactly n values from the LCG in order, one per element, then apply the threshold test elementwise. Getting the order right matters: the first draw corresponds to the first element, and so on — shifting the sequence by even one call produces a completely different (and incorrect) mask.

Edge cases to watch:

• p = 0: every draw is ≥ 0, so nothing is ever dropped, and the scale factor 1/(1-0) = 1 — output equals input exactly.
• p → 1: almost everything is dropped, and the scale factor blows up; well-formed training loops never actually push p to 1.
• EVAL mode must never consult the RNG or apply scaling — it's a pure passthrough, regardless of whatever p and SEED are currently set to.

This mirrors how a real deep learning framework's dropout layer behaves: same layer object, different behavior depending on model.train() vs model.eval() mode, with the RNG only ever touched in training mode.