Embeddings

Your program will take a list of token ids and produce the matrix the first transformer block reads: one row per token, each row the token's meaning plus its position in the sequence.

Token embeddings: a table lookup

A token id is just an integer; it has no geometry. The model needs a vector it can add, scale and compare, so it keeps a table with one learned row per vocabulary entry.

token table, V rows x D columns          (V = vocabulary size, D = model width)

 id 0  ->  [ 1.0   0.0 ]
 id 1  ->  [ 0.0   2.0 ]
 id 2  ->  [ 3.0   1.0 ]

embed(2) = row 2 = [3.0, 1.0]

That is the whole operation: row id of the table. Mathematically it is a one-hot vector (all zeros, a 1 at position id) multiplied by the table, which is why you will see the layer called "a linear layer on one-hot input". Nobody builds the one-hot; indexing is the same answer without the V-wide multiplication.

The rows are parameters. They start random (GPT-2 draws them with standard deviation 0.02) and are trained with everything else, which is how tokens that appear in similar contexts drift toward similar vectors. The table has V x D numbers: 50,257 x 768 is about 38.6 million for GPT-2 small.

Position embeddings: which slot am I in?

Attention (lesson 5) treats its input as a set. Shuffle the rows and the outputs shuffle the same way, so on its own "dog bites man" and "man bites dog" are indistinguishable. The fix is to put the position into the vectors themselves before attention sees them.

GPT-2 does it with a second learned table, pos, with one row per position 0..P-1, where P is the context length (1024 for GPT-2). The input to the first block is the sum, row by row:

x[t] = tok[id[t]] + pos[t]

Worked trace with the table above and pos = [[0.5, 0.5], [0, -1]], tokens 2,2:

t=0:  tok[2] + pos[0] = [3.0, 1.0] + [0.5,  0.5] = [3.5, 1.5]
t=1:  tok[2] + pos[1] = [3.0, 1.0] + [0.0, -1.0] = [3.0, 0.0]

The same token id gives two different rows, because they sit in different slots. Without pos both rows would be [3.0, 1.0] and the model could never tell the two occurrences apart. Adding keeps the width at D; concatenating would double it and every later weight matrix with it. Lesson 4 replaces the learned table with a formula.

The two ways a lookup fails

• An id outside 0..V-1. A real tokenizer never produces one, so it signals a bug upstream and should stop the run, not guess. Be especially careful with -1: in several languages a negative index counts from the end, so table[-1] silently returns the last row and the bug stays hidden.
• More tokens than position rows. The position table has exactly P rows. A learned table has nothing for position P, which is why GPT-2 cannot read a 1025th token. Lesson 14 shows the usual remedy: keep only the last P tokens.

Common mistakes

• Looking up the token row and forgetting the position row, so repeated tokens produce identical rows.
• Using pos[id] instead of pos[t]: the position is the index in the sequence, not the token's id.
• Letting a negative id read the last row, or clamping a too-large id to V-1, instead of reporting an error.
• Reading the position table past its end when the sequence is longer than P.

Your exercise

Given both tables and a token list, print the sum rows, or the single word ERROR when an id is out of range or the sequence is longer than the position table. The starter parses the tables and prints; you write embed.