Embeddings & Sinusoidal Positional Encoding

Networks compute on vectors of real numbers, but text arrives as integer token ids, and a sequence model must also know the order of those tokens. After this lesson your program turns token ids into vectors and stamps each position with a unique pattern of sines and cosines.

Embeddings: a table you index

An embedding table is a matrix with one row per token in the vocabulary and d columns. Looking up token t means returning row t. There is no arithmetic. During training the rows are ordinary weights, updated by backpropagation, so tokens that appear in similar contexts drift toward similar vectors; that is where the famous geometry of word vectors comes from.

The lookup is mathematically identical to multiplying a one-hot vector by the table: a one-hot row [0, 0, 1, 0] times the matrix picks out row 2. Implementations use the direct row fetch because the multiplication would waste time and memory on a vocabulary of tens of thousands.

Order needs its own signal

Self-attention (the attention chapter, coming up) treats its inputs as a set: shuffle the tokens and, apart from the shuffle, the outputs are the same. Something must tell the model that one token came before another. The original transformer adds a fixed vector PE[pos] to each token's embedding.

The sinusoidal formula

For position pos and dimension index k in a model of width d, let pair = k // 2 (integer division):

angle      = pos / 10000 ^ (2*pair / d)
PE[pos][k] = sin(angle)   if k is even
           = cos(angle)   if k is odd

Dimensions 0 and 1 share the fastest frequency, dimensions 2 and 3 a slower one, and so on, with wavelengths growing geometrically from about 6 positions up to tens of thousands. Fast pairs distinguish neighbours, slow pairs distinguish far-apart positions, exactly as the digits of a clock separate seconds from hours.

A worked trace

d = 4 and pos = 3. For dimensions 0 and 1, pair = 0, the divisor is 10000^0 = 1. For dimensions 2 and 3, pair = 1, the divisor is 10000^(2/4) = 100.

PE[3] = [ sin(3), cos(3), sin(0.03), cos(0.03) ] = [0.1411, -0.9900, 0.0300, 0.9996]

Position 0 is always [0, 1, 0, 1, ...], since sin 0 = 0 and cos 0 = 1. The model's input at a position is the embedding plus the encoding. With embedding [0.5, -0.5, 1, 0] at position 3 the sum is [0.6411, -1.4900, 1.0300, 0.9996].

Why sines and cosines

• No parameters to learn, and it extends to positions longer than anything seen in training.
• Relative offsets are easy. For each pair of dimensions, shifting the position by k is a fixed rotation of that pair, so attention can learn to look "3 tokens back".
• Similarity falls with distance. The dot product PE[i] . PE[j] depends only on |i - j|, and PE[i] . PE[i] is always d / 2 for even d, because sin^2 + cos^2 = 1 in each pair.

Many models learn positional vectors as ordinary weights instead, or use rotary encodings, but the idea of injecting position as extra numbers is the same.

Common mistakes

• Using k instead of k // 2 in the exponent, so neighbouring dimensions get different frequencies.
• Swapping sine and cosine, or using sine for every dimension.
• Writing 2*pair/d with integer division in a language that truncates.
• Computing 10000 ^ x with an integer power function.
• Adding the encoding to the table itself instead of to each looked-up vector.

Your exercise

Commands define the table and the tokens, then ask for LOOKUP, POSENC, ADD or SIM. The starter parses everything and combines your pieces. Implement lookup and pos_encoding.