Build a Transformer (GPT-style)
Build a tiny GPT in NumPy: token + position embeddings, layer normalization, scaled dot-product self-attention, multi-head attention with causal mask, feed-forward networks, residual connections, sampling (greedy, top-k, top-p), and the architecture behind ChatGPT, Claude, Llama.
advanced14 lessons4 chapters14 graded exercisesPython
No sign-up needed for lesson 1 · certificate on completion · sign up to save progress
What you’ll have built
Chapter by chapter. Every step is a graded exercise.
- Building Blocks
- Attention
- Full Transformer
- Production
Starts in Python — solve in the language you choose in the editor, same tests either way.
Curriculum
4 chapters, 14 lessons. Each lesson is a short read, one graded exercise, and a quiz.
- 1What Transformers SolveRead · exercise · quiz
- 2EmbeddingsRead · exercise · quiz
- 3Layer NormalizationRead · exercise · quiz
- 4Positional EncodingRead · exercise · quiz
Loading reference solution…