What Transformers Solve

The Transformer architecture (Vaswani et al. 2017, "Attention Is All You Need") replaced RNN/LSTM for sequence modeling. Powers:

• GPT family (Claude, ChatGPT, Llama, Gemini).
• BERT (search, classification).
• T5, BART (translation, summarization).
• Vision Transformers (ViT).
• Audio (Whisper).
• Multimodal (CLIP, GPT-4V).

Why?

• Parallelizable: process all tokens at once (vs RNN sequential).
• Long-range dependencies: attention attends globally.
• Scalable: just stack more layers + bigger.

Core idea: self-attention.

• Each token "looks at" all others to figure out its meaning.
• Weighting via learned attention scores.

Compare:

• RNN: process word-by-word, hidden state carries context. Slow, hard to remember long sequences.
• Transformer: process all words at once, attend across the whole sequence.

Architecture overview (decoder-only, like GPT):

input tokens
  ↓
embed + position encoding
  ↓
[Layer 1: attention + FFN]
[Layer 2: attention + FFN]
...
[Layer N]
  ↓
linear + softmax
  ↓
next-token probability distribution

Each layer:

1. Multi-head self-attention: each token attends to others.
2. Feed-forward network: per-token nonlinearity.
3. Residual connections + layer norm: deep stability.

Sizes (2024 LLMs):

• GPT-2 small: 124M params.
• GPT-3: 175B params.
• Llama 70B: 70B params.
• GPT-4: ~1T params (rumored).

We'll build a tiny GPT (~10M params) on character-level Shakespeare. Karpathy's nanoGPT is the reference. By the end, you understand what's inside ChatGPT.

Reference: "Attention Is All You Need" paper, nanoGPT (Karpathy), 3blue1brown attention video.