Transformer Block

One transformer block = attention + FFN + residuals + norms.

Residual connections:

• x = x + sublayer(x).
• Crucial: gradient flows directly through.
• Without: deep networks fail to train.

Layer norm position:

• Pre-norm: norm before sublayer (modern, more stable).
• Post-norm: norm after residual (original).

Stacking blocks:

Output shape: (T, vocab_size). Each position predicts next token's distribution.

Training: shift labels by 1 (predict token i+1 from positions 0..i).

• Inputs: tokens[:-1].
• Labels: tokens[1:].
• Cross-entropy loss between predicted distribution and true next token.

Inference (text generation):

1. Encode prompt → tokens.
2. Forward → next-token logits.
3. Sample (greedy / top-k / nucleus / temperature).
4. Append to context.
5. Repeat.

Key sizes (per layer of Llama 70B):

• d_model = 8192.
• n_heads = 64.
• FFN hidden = 28672.
• Per layer: ~860M params (the SwiGLU FFN alone is 3 × 8192 × 28672 ≈ 700M).
• 80 layers → ~68B params — nearly all of the 70B lives in the transformer blocks.

The architecture is simple. Scaling it to billions of params + trillions of tokens is the engineering feat.