Top-K Sampling

Greedy decoding (always argmax) is deterministic and boring. It collapses to repeating loops and dull outputs. To generate diverse text we sample from the model's distribution.

But sampling from the full softmax distribution puts non-trivial mass on every vocab token — including thousands of obviously-wrong choices. Top-K and top-p clip the long tail before sampling.

Top-K

1. Compute logits.
2. Keep the K largest logits, set the rest to -inf.
3. Softmax over the survivors → a clean distribution over K tokens.
4. Sample one.

Typical K for English text: 40-100. Much smaller and the model gets too rigid; much larger and you re-introduce the long-tail nonsense.

Top-P (nucleus)

Similar idea, but the set size adapts to the distribution:

1. Sort tokens by descending probability.
2. Take the smallest prefix whose cumulative probability > p.
3. Sample from that nucleus.

Where top-K always picks K tokens, top-P picks 1 token if the distribution is peaked and 200 if it's flat. This often produces better text. Typical p: 0.9-0.95.

Temperature

Before any top-K/top-P, divide logits by T:

• T < 1: sharpens — closer to argmax / greedy.
• T = 1: no change.
• T > 1: flattens — more chaotic, more random.

OpenAI / Anthropic APIs expose temperature and top_p directly.

Practical defaults

Implementation cost

Sorting is O(V log V) per token; partial sort / topk is O(V log K). Either is negligible compared to the transformer forward pass itself — sampling is not the bottleneck.