Production LLMs

Your program will run a complete, tiny GPT: token and position embeddings, a stack of the blocks you built, a final normalisation, an output head, and a greedy generation loop that slides a context window. Everything a production model does is this, plus scale and engineering, and the second half of this lesson maps the difference.

The whole model

ids (at most CTX of them)
  -> x[t] = tok[id[t]] + PE(t)                 lessons 2 and 4: look up, add the position
  -> block 1 ... block L                       lesson 10: attention + FFN, pre-norm, causal
  -> LayerNorm the final vectors               lesson 3
  -> logits[v] = x_last . tok[v]               one score per vocabulary entry
logits -> argmax -> next id                    lesson 12

Three details are new.

Weight tying. The output head reuses the embedding table. The logit of token v is the dot product of the final vector with v's embedding, so the model predicts the token whose embedding best matches the vector it ended up with. It saves V x D parameters (38.6 million for GPT-2 small) and often helps quality.

Only the last row. Generation needs the scores for the next token only, so only the last position's vector goes through the head. The earlier rows are still needed: they were the keys and values of attention.

The window. The model holds at most CTX positions. When the sequence grows beyond that, keep only the last CTX tokens and run the model on them. The positions restart at 0 inside the window. This is the same crop nanoGPT does with idx[:, -block_size:].

Parameters: V x D for the table, plus per block 4 D^2 for attention and 2 D F for the FFN. GPT-2 small is V = 50257, D = 768, F = 3072, L = 12: 38.6M + 12 x 7.1M, about 124M.

Weights from the seed

The seeded generator is the one from lesson 10. Draw the embedding table first (V rows of D numbers), then each layer's weights in the order of lesson 10, layer 1 first. The position vectors are computed, not drawn.

What a production model adds

• Scale and data. Billions of parameters and trillions of tokens, trained on thousands of GPUs with the loss of lesson 11. Then supervised fine-tuning on instruction examples and RLHF or DPO against human preferences turn a text predictor into an assistant. LoRA fine-tunes with about 1 percent of the parameters; QLoRA adds a 4-bit base.
• A KV cache. Keys and values of past tokens never change, so a server keeps them and computes only the new token's. Per token that is 2 x layers x kv_heads x head_width x bytes. Llama 2 70B (80 layers, 8 KV heads thanks to grouped-query attention, width 128, 2-byte numbers) needs 320 KiB per token, or 1.25 GiB for a 4096-token conversation. With all 64 heads it would be 8 times that.
• Quantisation. Weights stored in 8 or 4 bits shrink memory 2 to 4 times for a small loss in quality (GPTQ, AWQ, GGUF).
• Serving. Continuous batching and paged KV memory (vLLM) keep GPUs full; speculative decoding trades a small draft model for latency. Latency (time to first token) and throughput (tokens per second across users) pull in opposite directions.
• Around the model. Retrieval-augmented generation puts fetched documents into the prompt to reduce invention; tool use has the model emit structured calls that code executes. Prompt injection, jailbreaks and training-data leakage are the risks.

Common mistakes

• Taking positions from the whole sequence instead of from the start of the window.
• Skipping the final LayerNorm before the head.
• Drawing the weights in the wrong order: the embedding table comes first.
• Not cropping the prompt when it is already longer than the window.

Your exercise

Write forward (embeddings, blocks, final norm, head) and generate (window and greedy loop). The starter supplies the seeded weights, the position vectors and the block you wrote in lesson 10.