Temperature

Watch how changing temperature reshapes the candidate distribution for the next token. The model weights stay the same; only decoding changes.

LLM temperature is a decoding setting that controls how random token selection is: it divides the model's logits by a positive number T before softmax. T below 1 sharpens the distribution, concentrating probability on the top tokens; T above 1 flattens it. For three tokens with logits [2, 1, 0], the probabilities are about [0.67, 0.24, 0.09] at T = 1, [0.87, 0.12, 0.02] at T = 0.5, and [0.51, 0.31, 0.19] at T = 2. Temperature changes how a token is selected from the model's scores; it does not change the weights, knowledge, or ranking of tokens.

Ask a model for "a tagline for a coffee shop" three times and you may receive "Fresh coffee. Better mornings.", "Where every cup starts a conversation.", and "Brewed for the curious." None of these is a malfunction. At the first token the model had a distribution over many acceptable openings; the runtime drew one at random, in proportion to probability, and every later token was conditioned on that draw. The Next-Token Prediction lesson explains how that distribution comes about. Temperature controls how concentrated it is at the moment of selection.

With logits z[i] and temperature T > 0:

P(i) = exp( z[i] / T ) / sum_j exp( z[j] / T )

Why does dividing sharpen or flatten? Dividing by T less than 1 stretches the gaps between logits (a gap of 1 becomes 2 when T = 0.5), and because softmax is exponential in the gaps, larger gaps give a more lopsided distribution. Dividing by T greater than 1 shrinks the gaps, so the exponentials become similar and probabilities even out. The order of tokens is never altered, since dividing by a positive number preserves the order. The model has not learned or forgotten anything; the same scores are simply read with different sensitivity.

Use the same three toy logits as the softmax example in the Next-Token Prediction lesson. Tokens A, B, C have logits 2, 1, 0 (illustrative, not from a real model).

scaled logits  = [2, 1, 0]
exponentials   = [7.389, 2.718, 1.000]    sum = 11.107
probabilities  = [0.665, 0.245, 0.090]

scaled logits  = [2/0.5, 1/0.5, 0/0.5] = [4, 2, 0]
exponentials   = [54.598, 7.389, 1.000]   sum = 62.987
probabilities  = [0.867, 0.117, 0.016]

scaled logits  = [2/2, 1/2, 0/2] = [1.0, 0.5, 0]
exponentials   = [2.718, 1.649, 1.000]    sum = 5.367
probabilities  = [0.506, 0.307, 0.186]

Drag the slider below to see the same computation at any temperature. The values are recomputed from the logits [2, 1, 0], so the three presets reproduce the numbers above.

Notice that A stays on top in all three cases: ranking is unchanged. What changes is how often B and C get picked: C goes from 1.6 percent of draws at T = 0.5 to 18.6 percent at T = 2. At T = 0.5, picking A is roughly 7 to 1 over B; at T = 2 it is under 2 to 1. Over a 200-token reply where there are hundreds of such decisions, those percentages compound into visibly different text.

You cannot literally divide by zero, so T = 0 is an API convention. In most implementations it means "use the greedy argmax": always pick the highest-probability token, which is the limit of the formula as T shrinks toward 0 (the top token's probability approaches 1). Behaviour at zero varies by implementation: some providers treat it as greedy argmax, some enforce a small minimum, and some reject it, so check the documentation for your endpoint.

Even with greedy decoding, identical outputs are not guaranteed. GPU floating-point operations are not perfectly associative, so batching, hardware differences, or kernel selection can change the last decimal places of logits and flip a near-tie. Providers may update model versions behind a name, and some mix requests in a batch in ways that affect numerics. A seed parameter, where offered, makes sampling repeatable on a best-effort basis; its semantics and the allowed temperature range (commonly 0 to 1 or 0 to 2) are provider-specific. For tests, record the exact model version and settings, and design them to tolerate small output variations.

Temperature acts locally, one token at a time. Consider a prompt "Write a one-line product tagline for a coffee shop:" with a first-token distribution (illustrative) of "Fresh" 0.40, "Where" 0.25, "Brewed" 0.15, and others 0.20.

After "Fresh", the model's next-token distribution is conditioned on … Fresh; after "Where", on … Where. These are different distributions, so the two runs do not merely differ in one word, they follow different paths. This is also why raising temperature increases diversity at the level of whole responses more than the per-token probabilities suggest, and why you cannot recover the "same text with one word changed".

Temperature is usually one of several decoding controls. The common conceptual pipeline is: logits → temperature scaling → softmax → candidate filtering (Top-K and Top-P) → renormalise → sample. Implementations differ in the order (some apply top-k to logits before temperature), in which parameters are exposed, and in whether recommended combinations exist. Some providers advise changing temperature or top-p but not both. Repetition and frequency penalties adjust logits for tokens already used. When two controls interact, a setting that looks identical on paper (temperature 0.8, top-p 0.9) can behave differently across providers, so test on the actual endpoint.

"Use 0.2 for code and 0.8 for creative writing" is folklore. To pick a setting for your task, run an experiment. Pick three tasks with different requirements and measure them separately:

• Structured extraction (invoice fields to JSON). Metrics: schema validity rate, field-level accuracy against labelled ground truth, retries needed. You expect variance to hurt here, but verify.
• Customer-response drafting. Metrics: rubric score by reviewers (policy-correct, tone, completeness), number of unsupported claims, and variance across repeated runs of the same input.
• Brainstorming product names. Metrics: number of distinct acceptable ideas per call (de-duplicated), human-rated novelty, and rate of unusable outputs.

Hold everything else constant: same model version, same prompts and test set, same other decoding settings. Vary only temperature, and make several calls per input so you can see within-input variance. Record cost (tokens, retries) because a setting that needs more retries is not free. Decide pass criteria before you look at the results to avoid fitting the threshold to the data. This lesson deliberately gives no result: outcomes depend on model, task, and prompt.

import json, time, statistics

TEMPS = [0.0, 0.3, 0.7, 1.0]
RUNS_PER_INPUT = 5

def run_experiment(client, model, prompt_template, cases, evaluator, out_path):
    with open(out_path, "a") as log:
        for T in TEMPS:
            for case in cases:
                for run in range(RUNS_PER_INPUT):
                    resp = client.generate(model=model, temperature=T, max_tokens=400,
                                           messages=prompt_template.render(case))
                    result = evaluator(case, resp.text)      # e.g. {"schema_ok": True, "fields_correct": 7/8}
                    log.write(json.dumps({
                        "ts": time.time(), "model": resp.model_version,   # exact version, not just the alias
                        "temperature": T, "run": run, "case_id": case.id,
                        "input_tokens": resp.usage.input_tokens, "output_tokens": resp.usage.output_tokens,
                        "seed": None,                                     # provider-specific; record if supported
                        "output": resp.text, **result}) + "\n")

# Analysis: group by (task, temperature); report mean and spread of each metric,
# plus pairwise agreement between runs on the same input (diversity).

Rather than memorising values, ask three questions about the task. Is there one right output? Extraction, classification, and tool-argument generation have a correct answer, so lower temperature (or greedy) usually reduces random variation, and you should confirm that with your metrics. Is variety itself valuable? Brainstorming and varied drafts benefit from sampling more of the distribution, up to the point where coherence falls. What does a bad sample cost? If a wrong token can move money or break a parser, use constraints such as structured output and validation, and do not rely on temperature to prevent errors.

Two limits are worth stating plainly. Low temperature does not make output true: greedy decoding picks the most probable continuation, which can be a confident error, and it makes the same mistake every time. High temperature does not make output original: it spreads probability over tokens the model already ranks as plausible, so it can yield odd combinations of familiar ideas as easily as new ones, plus more errors. Temperature tunes the risk of sampling away from the mode; it neither adds knowledge nor guarantees creativity.

Temperature reshapes every probability. The next lesson looks at a different control that removes candidates altogether.