Residual Connections

Stacking 24 transformer blocks without residual connections produces a network that does not train. Gradients vanish, attention layers fail to learn, and the loss curve flatlines.

The fix from ResNet (He et al. 2016) is one of the most important ideas in deep learning:

output = input + sublayer(input)

Identity short-circuit added around every sublayer (one around attention, one around the FFN).

Why this works

1. Direct gradient path. Backprop through y = x + f(x) distributes the gradient: dy/dx = 1 + df/dx. The 1 is a highway — even if df/dx is tiny, the input-side gradient survives. With 24 stacked blocks the multiplicative effect (factor of ~1) keeps signals alive end-to-end.

2. Identity is a safe default. If a sublayer is useless, the network can learn to output near-zero from f(x), and the residual just passes x through. Compare to a chain of dense layers: every layer must learn the identity, which is surprisingly hard.

3. Composability. Each block adds a "delta" on top of the running representation, like applying a sequence of edits to a draft. The model can learn a small refinement at every layer rather than reinventing the representation each time.

Block shape

Pre-norm transformer block (modern):

x = x + Attention(LayerNorm(x))
x = x + FFN(LayerNorm(x))

Post-norm (original Vaswani):

x = LayerNorm(x + Attention(x))
x = LayerNorm(x + FFN(x))

Pre-norm is the modern default because it's much easier to train deep stacks — the residual path bypasses LayerNorm entirely, so the input gradient really does flow through unscaled.

Don't forget the projection

The output of multi-head attention has shape (T, d_model), same as the input. The final W_O projection is what makes the dimensions match for the residual addition.

For the FFN, the input is d_model, the hidden is 4 * d_model, and the output is d_model again — the structure is hourglass-shaped so the residual works.