Training a Language Model

Training data: a big corpus of text.

• Common Crawl: ~petabyte of web text.
• Books, Wikipedia, GitHub.
• Filter for quality (deduplication, language detection, content filtering).

Tokenization:

• BPE (byte-pair encoding) or sentence-piece.
• Vocab ~50000.
• Each token = 1-3 chars typically.

Training objective: next-token prediction (autoregressive).

• Input: token sequence.
• Predict: next token.
• Loss: cross-entropy.

Hardware:

• GPT-3: 1024 GPUs for ~30 days.
• Llama 70B: 2048 A100s.
• GPT-4: ~25000 GPUs (rumored).

Data parallel: each GPU has full model, processes different batch. Tensor parallel: split model across GPUs (for huge models). Pipeline parallel: layers across GPUs in pipeline.

Mixed precision:

• FP16 forward + backward.
• FP32 master weights.
• 4x memory savings, 2-3x speedup.
• Some loss in numerical stability.

Optimizers:

• AdamW (Adam + weight decay): standard for LLMs.
• Lion: simpler, sometimes faster.

Learning rate schedule:

• Warmup: 0 → max_lr over first 1-2k steps.
• Cosine decay: max_lr → 0 over training.

Effective batch size: 1M+ tokens per step (gradient accumulation across many GPUs).

Loss curves:

• Start: ~10 (cross-entropy of uniform over 50k vocab).
• After many tokens: ~2-3 for small models, ~1.5 for big ones.
• Lower = better.

Scaling laws (Kaplan et al. 2020, Chinchilla 2022):

• Performance scales with: parameters × compute × data.
• For best loss: ~20 tokens per parameter.
• Common rule: bigger model + more data > both.

Cost:

• GPT-3 training: ~$5M (2020 prices).
• GPT-4: ~$100M.
• Llama 2 70B: ~$5M (Meta).
• Open source has narrowed the gap.

Datasets:

• The Pile: 800GB curated.
• RedPajama: open Llama dataset replication.
• Common Crawl filtered: hundreds of TB.

Quality matters:

• Deduplicate (saves cost + improves model).
• Filter low-quality (Reddit upvote tier).
• Mix code, books, papers, web for diverse capability.