Top-K and Top-P Sampling

Watch the model score many possible continuations, then narrow them before choosing the next token.

Top-k keeps only the k most probable tokens, and top-p (nucleus sampling; Holtzman et al., 2019) keeps the smallest set of top tokens whose probabilities add up to at least p. Both set every other token's probability to zero, renormalise the survivors so they sum to one, and then sample. Top-k uses a fixed candidate count; top-p adapts the count to how confident the model is. Top-p is called nucleus sampling because it keeps the "nucleus" of the distribution: the threshold is a cumulative probability, not a fixed number of tokens like k.

A vocabulary has tens of thousands of tokens. Even for a clear context, the model gives each unlikely token a small but non-zero probability. Individually those slivers are tiny; together they can be large: if the 50,000 least likely tokens hold 3 percent of the total mass, then about one token in thirty-three is drawn from that tail, and across a 300-token reply the chance of at least one off-topic draw becomes substantial (1 − 0.97³⁰⁰ is nearly 1 for independent draws). A single odd token can derail the sentence, because every later token is conditioned on it.

The chart below plots that formula for any tail mass you choose; the 3 percent case is the default.

Truncation removes the tail before sampling. It differs from temperature, which re-weights every token but keeps all of them eligible. Temperature changes how much more likely good tokens are; truncation makes bad ones impossible.

Nucleus sampling sorts tokens by probability and keeps the smallest set whose cumulative probability reaches p.

def top_p_filter(probs, p):
    order = argsort(probs, descending=True)
    cumulative = 0.0
    keep = []
    for token in order:
        keep.append(token)
        cumulative += probs[token]
        if cumulative >= p:                       # smallest prefix with mass >= p
            break
    filtered = zeros_like(probs)
    filtered[keep] = probs[keep]
    return filtered / filtered.sum()

Take a toy distribution (illustrative values) over tokens A to F: [0.40, 0.25, 0.15, 0.10, 0.06, 0.04], already sorted.

keep A, B, C           mass = 0.40 + 0.25 + 0.15 = 0.80
renormalise (divide by 0.80):
   A = 0.40 / 0.80 = 0.5000
   B = 0.25 / 0.80 = 0.3125
   C = 0.15 / 0.80 = 0.1875          (sum = 1.0000)
D, E, F are now impossible

cumulative: A 0.40 -> B 0.65 -> C 0.80   (>= 0.80, stop)
keep A, B, C           -> same set as top-k = 3; renormalised [0.5000, 0.3125, 0.1875]

(In floating point 0.40 + 0.25 + 0.15 may land at 0.7999999..., which could force D to be included.
 Implementations differ; do not rely on exact equality.)

cumulative: A 0.40 -> B 0.65 -> C 0.80 -> D 0.90   (>= 0.90, stop)
keep A, B, C, D        mass = 0.90
renormalise (divide by 0.90):
   A = 0.4444   B = 0.2778   C = 0.1667   D = 0.1111   (sum = 1.0000)

The coincidence that top-k = 3 and top-p = 0.80 agree is just this distribution; with a different distribution they would not. After filtering and renormalising, sampling is an ordinary weighted draw over the survivors: with top-k = 3, A is chosen 50 percent of the time, even though it started at 40 percent, because the removed tail's probability was redistributed proportionally.

The key difference between the methods appears when the model's confidence changes from step to step.

With [0.92, 0.04, 0.02, 0.01, 0.01], top-k = 3 retains three tokens (mass 0.98), keeping two candidates the model considers very unlikely; top-p = 0.90 retains one token, because 0.92 already exceeds 0.90, so the next token is effectively deterministic. That matches the situation: the model is confident, so there is nothing sensible to choose between.

With ten equally likely tokens, top-k = 3 keeps just three and discards 70 percent of the mass, which throws away seven perfectly good options. Top-p = 0.90 keeps nine, reflecting how uncertain the model is. That is the practical argument for nucleus sampling: the candidate set grows when there are many reasonable continuations (an open-ended creative opening) and shrinks when there is essentially one (the closing parenthesis of a function call). A fixed k cannot do both.

Combine this with the temperature lesson's example. With logits [2, 1, 0] and T = 2 the probabilities are about [0.506, 0.307, 0.186]. Applying top-k = 2 keeps A and B (mass 0.813) and renormalises to about [0.622, 0.378]. Reversing the order, filtering on T = 1 probabilities and then scaling, would give a different outcome, which is exactly why you should read your provider's documentation: the order and the set of supported parameters differ, and some hosted APIs expose only top-p, only top-k, or neither.

The explorer below runs this whole pipeline on the distributions used in this lesson. Pick a distribution, then change the temperature, k and p. For example, choose Logits [2, 1, 0], set T = 2 and k = 2 to reproduce the [0.622, 0.378] result above, or choose Peaked with top-p enabled to see a single survivor.

Tighter truncation (small k, low p) increases coherence and repeatability, and can produce repetitive text: the same safe phrases, or loops where the most likely continuation is the one just written. Looser truncation increases variety and the chance of an odd token. Truncation can also remove a correct but low-probability token: if the model's top choices for a rare product name are wrong and the right token is at 2 percent, a top-p of 0.9 may have excluded it. Nothing about these filters checks facts; they reshape which of the model's own guesses can appear. A false statement with 70 percent probability survives any truncation setting.

Structured extraction benefits from little randomness, but the better protection is schema validation and constrained decoding where available. Code generation has strict syntax, so tail tokens are often invalid, but multiple correct solutions exist and sampling several candidates and running tests can beat any single greedy answer. Creative writing usually wants a wider nucleus, with a penalty on repetition. Customer support needs consistent policy language, yet natural phrasing; here a moderate setting plus verification of claims matters more than the exact value. These are hypotheses to test, not rules; models are trained differently and some are tuned to work best at their default settings.

Compare four configurations on the same model, the same prompts, and the same metrics: greedy (or temperature 0), top-k only, top-p only, and a combination with a moderate temperature. For each prompt, draw several samples per configuration.

CONFIGS = {
  "greedy":    dict(temperature=0),
  "top_k_40":  dict(temperature=1.0, top_k=40),      # only if the endpoint supports top_k
  "top_p_0.9": dict(temperature=1.0, top_p=0.9),
  "combined":  dict(temperature=0.7, top_p=0.9),
}
for name, params in CONFIGS.items():
    for case in test_set:                      # fixed prompts, fixed order
        outs = [call(model, case.prompt, **params) for _ in range(8)]
        record(name, case.id,
               task_score = mean(score(case, o) for o in outs),          # correctness or rubric
               distinct   = len(set(normalise(o) for o in outs)),        # diversity
               invalid    = sum(not valid_format(o) for o in outs),      # parse / schema failures
               tokens     = sum(usage(o) for o in outs))                 # cost
# Report spread across the 8 samples as well as the mean; decide pass thresholds before running.

Sampling controls change which tokens appear. The next lesson turns to a different question: why a fluent, well-sampled answer can still be unsupported.