What Transformers Solve

Your program will turn text into the integers a language model actually reads, and turn them back into text. That is step zero of every GPT: the model never sees letters, only token ids. Over these 14 lessons you build every later stage, and in the last one you run a tiny GPT end to end.

The one job: predict the next token

A language model is a single function. Give it the tokens so far, and it returns a score for every token that could come next.

tokens so far:   t  o  _  b  e  _  o  r  _  n  o  t  _  t  o  _
model returns:   a score for every token in the vocabulary
                 high for "b", low for "z", ...

Generation is that function in a loop: pick a token from the scores, append it, call it again. Training scores the same function against text a human wrote.

Before 2017 the function was a recurrent network (RNN, LSTM) that read one token at a time and squeezed everything seen into one hidden state. Token 500 cannot be computed until tokens 1 to 499 are done, so a GPU's parallelism goes unused, and a fact from token 3 must survive 497 rewrites of the state to reach token 500.

The transformer (Vaswani et al., "Attention Is All You Need", 2017) removes both. In its attention step every position reads every earlier position directly, and all positions are computed at the same time. The price is memory: the pairwise scores form a T x T table for a sequence of T tokens.

The map you will build

text --tokenizer--> ids                                        lesson 1
ids --> token vector + position vector                         lessons 2, 4
    --> N times:  [ normalize, attention, add back ]           lessons 3, 5, 6, 8, 9
                  [ normalize, feed-forward, add back ]        lessons 3, 7, 9
    --> one block, assembled                                   lesson 10
    --> a score per vocabulary entry ("logits")
logits --> loss against the true next token (training)         lesson 11
logits --> choose a token (generation)                         lessons 12, 13
all of it, running a tiny GPT                                  lesson 14

Each exercise reads plain text and prints numbers, so every stage is testable alone and is the real operation, only small.

Stage one: text to ids

A tokenizer is a fixed table between pieces of text and integers. Production models use byte-pair encoding with 50,000 or more pieces, so a common word is one token. The simplest tokenizer, used by Karpathy's nanoGPT on Shakespeare, is character-level: one id per distinct character.

The vocabulary is the distinct characters of the training text in sorted order (by character code: space, then digits, then capitals, then lowercase). Worked trace for the corpus banana bread:

distinct characters, sorted:   ' '  a  b  d  e  n  r
ids:                            0   1  2  3  4  5  6

encode "bean"  ->  b=2 e=4 a=1 n=5   ->  2,4,1,5
decode 6,1,5   ->  r a n             ->  "ran"

Two edge cases are decisions you make, not accidents. A character the corpus never contained has no id, so encoding gives it -1. An id outside 0..V-1 has no character, so decoding gives it ?. Real byte-level tokenizers avoid the first case by making all 256 byte values part of the vocabulary.

Why the order is fixed

Row 3 of the embedding table (next lesson) is trained to mean h. If a later run numbers h as 9, every lookup lands on the wrong row and the trained weights are noise. Where set or map iteration order is unspecified, or randomised on purpose, you must sort explicitly.

Common mistakes

• Numbering characters in order of first appearance. For the corpus dcba that numbers d=0, c=1, b=2, a=3, so dcba encodes as 0,1,2,3, while the sorted vocabulary abcd encodes it as 3,2,1,0.
• Letting a negative id index from the end. In some languages vocab[-1] silently returns the last character, so decoding -1 must be checked, not assumed to fail.
• Trimming the text. A space is a character with an id. Everything after the first space of the command line is the payload, spaces included.

Your exercise

Build the vocabulary from CORPUS, then answer each ENCODE and DECODE command with one output line, in the order they appear. The starter splits each line into a command and its payload and prints the results, so you write only the three pieces: building the vocabulary, encoding a text, decoding a list of ids.