Positional Encoding

Lesson 2 gave every position its own learned vector. That table has a hard edge (it ends at the training context length) and it costs parameters. Your program will now compute position information from a formula instead, in the two forms that matter: the sinusoidal vector of the original transformer, and the rotation (RoPE) that modern models apply inside attention.

Sinusoidal encoding: a clock with many hands

Vaswani et al. give position pos a vector of D numbers built from sines and cosines of different frequencies:

PE(pos, 2k)   = sin( pos / 10000^(2k/D) )
PE(pos, 2k+1) = cos( pos / 10000^(2k/D) )

Elements come in sin/cos pairs that share one angle; k counts pairs. Pair 0 has angle pos, so it turns once every 6.28 positions, like a fast second hand. Pair D/2 - 1 has a divisor near 10,000 and turns once in about 60,000 positions, like an hour hand. Reading all the hands at once identifies a position much as a clock face identifies a time, and positions that are close have similar vectors.

Worked trace, D = 4, pos = 2:

pair k=0:  divisor 10000^(0/4) = 1       angle = 2.00   sin = 0.9093   cos = -0.4161
pair k=1:  divisor 10000^(2/4) = 100     angle = 0.02   sin = 0.0200   cos =  0.9998
PE(2) = [0.9093, -0.4161, 0.0200, 0.9998]

The result is added to the token embedding exactly as lesson 2 added the learned row: x[t] = tok[id] + PE(t). Nothing is trained. With an odd D the last element is the sine of a pair that has lost its cosine, which the formula handles without special cases.

Where sinusoids fall short

The vector tells attention where a token is, but attention has to learn to read it, and what it needs is the distance between two tokens. Sinusoids carry that information only indirectly, and models trained on them cope poorly with lengths they never saw.

RoPE: put the position into the rotation

Rotary embeddings (Su et al.; Llama, GPT-NeoX, most open models today) skip the addition. They take the query and key vectors inside attention, cut them into pairs (a, b), and rotate each pair by an angle that grows with the token's position:

(a, b)  ->  ( a*cos(angle) - b*sin(angle),  a*sin(angle) + b*cos(angle) )
angle   =   pos / 10000^(2k/D)          (same frequencies as above, k = pair index)

A rotation keeps the length of a pair and turns its direction. The payoff comes when two rotated vectors meet in a dot product. Take D = 2 and q = k = (1, 0). Rotated to positions m and n they become (cos m, sin m) and (cos n, sin n), and

dot = cos m * cos n + sin m * sin n = cos(m - n)

Positions (5, 3) and (9, 7) give the same score, cos 2 = -0.4161, because only the offset matters. The same identity holds for every pair in a longer vector, so the attention score between two tokens depends on their relative position by construction, with no extra parameters.

Other schemes you will meet

• Learned absolute (GPT-2, BERT): lesson 2. Simple, but ends at position P.
• ALiBi (BLOOM, MPT): no vectors at all. A penalty proportional to distance is subtracted from each attention score.

Common mistakes

• Using the element index i where the pair index k = i // 2 belongs. The sin and cos of a pair must share one angle; with i in the exponent every element gets its own frequency.
• Swapping the signs in the rotation. Rotating (1, 0) by a small positive angle must raise the second coordinate.
• In RoPE, taking D from the model instead of from the vector being rotated, or forgetting that the exponent is 2k / D, not k / D.

Your exercise

Implement the sinusoid and the rotation. The starter parses the commands and already builds DOT on top of your rotation, so a wrong ROPE shows up there too.