Causal Masking

A GPT-style language model predicts token t+1 from tokens 0..t. If we let position t peek at future tokens, training cheats and inference becomes impossible (those tokens don't exist yet).

Causal masking zeroes out the attention scores for future positions before softmax.

The mechanism

After computing scores S = Q @ K^T / sqrt(d_k), set S[i, j] = -inf for every j > i. Then softmax — the -inf entries become 0, so position i only attends to positions 0..i.

In code (PyTorch):

mask = torch.triu(torch.ones(T, T), diagonal=1).bool()
scores.masked_fill_(mask, float('-inf'))
weights = softmax(scores, dim=-1)

Why -inf not zero?

If you set future scores to 0 instead of -inf, softmax would still give those positions exp(0)/sum weight — a non-trivial fraction. We need them to contribute zero probability mass, which requires exp(-inf) = 0.

Encoder vs Decoder

• Decoder (GPT) — causal mask. Token t only sees 0..t. Used for generation.
• Encoder (BERT) — no mask. Every token sees every other token. Used for understanding (classification, embeddings).
• Encoder-decoder (T5, original Vaswani) — encoder is bidirectional, decoder is causal, plus cross-attention from decoder to encoder.

Training trick: predict every position at once

The causal mask is what makes transformer training so efficient. We don't need a separate forward pass per target token — we run the model once on the full sequence, and the mask guarantees that the prediction for position t+1 only depended on positions 0..t. We compute cross-entropy loss at every position simultaneously, then sum.

This is teacher forcing: at training time we feed the true previous tokens, not the model's own predictions. At inference time we autoregress one token at a time.