Inference & Sampling

Generation: produce one token at a time.

Sampling strategies:

Greedy (deterministic):

• Pick argmax. No randomness.
• Repetitive output.

Temperature:

• Divide logits by temperature τ before softmax.
• τ < 1: sharper distribution (more deterministic).
• τ > 1: flatter (more random).
• τ = 0: equivalent to greedy.

Top-k:

• Keep top k logits, mask rest, sample from remaining.

Top-p (nucleus):

• Sort by probability.
• Keep smallest set with cumulative > p.
• Sample from those.
• More natural-sounding text than top-k.

Beam search:

• Track top-k partial sequences.
• Pick highest cumulative log-prob.
• More fluent but less diverse.

KV-cache (critical optimization):

• Each new token needs attention to ALL previous.
• Computing K, V for all previous each step = quadratic cost.
• Cache K, V from prior steps, only compute for new token.
• Speedup: 10-100x for long sequences.

Memory:

• KV-cache size = layers × heads × seq_len × d_k × 2 bytes (FP16).
• Llama 70B at 4096 ctx: ~80 GB just for KV cache!
• Mitigations: GQA (group query attention), paged attention, quantization.

Speculative decoding:

• Use a smaller "draft" model to propose tokens.
• Verify with big model (parallel, single pass).
• 2-3x speedup.

Modern serving:

• vLLM: paged attention + continuous batching.
• TGI (Hugging Face): production text generation server.
• llama.cpp: CPU/GPU inference of GGML models.
• Tensor parallelism for huge models.