Positional Encoding

Attention is set-invariant: shuffle the tokens and the output shuffles too. Pure token embeddings have no notion of order — to a vanilla attention layer, "the cat sat" and "sat the cat" look identical.

Positional encodings fix this by adding a per-position vector to each token embedding.

Sinusoidal (Vaswani 2017)

The original transformer used fixed sinusoidal codes:

PE(pos, 2i)   = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

Properties:

• Deterministic — no training needed.
• Extrapolates (in theory) to sequences longer than seen at training time.
• Adjacent positions get similar codes; very different positions get very different ones.

The embedding table is added element-wise: x_t = token_emb[token_id_t] + PE(t).

Learned (GPT-2, BERT)

Just allocate a (max_seq, d_model) parameter and learn the codes. Simpler, often slightly better, but does not extrapolate past the trained sequence length.

RoPE — Rotary (Llama, GPT-NeoX, ChatGPT family)

The modern default. RoPE rotates the Q and K vectors in pairs by an angle that depends on the position. Mathematically nice property: the dot product q_m . k_n only depends on the relative distance m - n. Translation-equivariant and extrapolates well.

ALiBi (BLOOM, MPT)

Skip explicit position info entirely. Instead, add a position-dependent bias to attention scores: closer tokens get a bigger bias. Tiny code change, no extra params.

Practical guidance

For this course we will implement sinusoidal — it's the cleanest math and matches the original paper.