Next Token Prediction

Watch a sentence grow one token at a time. The same loop powers short answers, long essays, and code generation.

Next-token prediction is the task at the core of every LLM: given a sequence of tokens, the model outputs a probability distribution over its vocabulary for the next position. The network first produces one raw score (a logit) per vocabulary token; softmax converts those scores into probabilities that sum to one; and a decoding rule picks a token. Training teaches the scores by penalising low probability on the true next token, and generation repeats the pick-and-append step until the sequence ends.

This lesson works the whole mechanism with actual numbers so that none of it stays abstract. For the broader picture of where this loop sits in a request, see the LLM Overview.

Suppose a log-analysis assistant has read the text "The server returned status". Many continuations are plausible: 200, 404, 500, code, :. A good language model should assign most of its probability to a handful of these and very little to nonsense such as banana. Which tokens deserve the mass depends on the context: after "The health check failed and the server returned status" the mass should shift toward error codes.

Precisely, a token is an element of a fixed vocabulary of size V (tens of thousands to a few hundred thousand; see Tokens Explained). Given the tokens seen so far, x[1], …, x[t], the model outputs a conditional distribution:

P( x[t+1] = v | x[1], x[2], ..., x[t] )     for every v in the vocabulary
                                             (V probabilities, all >= 0, summing to 1)

Note that the model predicts tokens, not "words". A token may be a word, a fragment such as "ication", a punctuation mark, or a byte. Saying "predicts words" hides precisely the details (spacing, rare words split into fragments) that explain odd behaviour in practice.

After the transformer layers, the last position holds a hidden-state vector h with, say, a few thousand dimensions. A final linear layer, the unembedding or output projection, multiplies it by a matrix W with one row per vocabulary token (many models tie this matrix to the input embedding table). Each product is a single number: the logit for that token.

h      : vector of d numbers      (hidden state at the last position)
W      : V x d matrix             (one row per vocabulary token)
logits = W @ h                    -> V numbers, one raw score per token

Logits are unnormalised scores. They can be negative, they have no fixed range, and only differences between them carry meaning. A logit of 9.1 for " 500" does not mean 91 percent; it means " 500" scores higher than tokens with lower logits, by amounts that softmax will translate into probabilities. They are also not measures of truth: a model can assign a high logit to a fluent falsehood because the context pattern favours it.

Softmax exponentiates each logit and divides by the sum of the exponentials:

p[i] = exp(z[i]) / sum_j exp(z[j])

Take a toy vocabulary of three tokens with logits z = [2, 1, 0]. These numbers are chosen to make the arithmetic visible; they are not outputs of a real model.

exp(2) = 7.389
exp(1) = 2.718
exp(0) = 1.000
sum    = 7.389 + 2.718 + 1.000 = 11.107

p = [7.389/11.107, 2.718/11.107, 1.000/11.107]
  = [0.665, 0.245, 0.090]          (adds to 1.000)

Two properties matter. First, normalisation: exponentials are always positive, and dividing by their sum forces a valid probability distribution. Second, only relative gaps matter. Adding 100 to every logit multiplies every exponential by e^100, which cancels in the ratio, so [102, 101, 100] gives the same probabilities. A gap of 1 between logits multiplies the odds by e ≈ 2.718 (here 7.389 / 2.718), and a gap of 2 multiplies them by e² ≈ 7.389. This exponential sensitivity is why a small change to scores, such as the scaling in the Temperature lesson, can sharply change the distribution.

Pretraining converts every sequence in the corpus into many prediction problems. Take the four-token sequence A B C D:

At each position the model sees the true previous tokens, never its own guesses. That is teacher forcing. Because attention is causal (position 2 cannot see position 3), all three problems are computed in a single forward pass, which is what makes pretraining on trillions of tokens feasible.

The loss: cross-entropy on the target token

For one position, let p_target be the probability the model assigned to the true next token. The loss is −ln(p_target). The loss averaged over all positions in a batch is what the optimiser minimises.

p_target = 0.8  ->  loss = -ln(0.8) = 0.223    (model was fairly sure of the right token)
p_target = 0.1  ->  loss = -ln(0.1) = 2.303    (model gave the right token little mass)
p_target = 1.0  ->  loss = 0                    (perfect, never actually reached)
p_target -> 0   ->  loss -> infinity            (confidently wrong is punished heavily)

Lower loss therefore rewards putting more probability on the token that actually appeared. Because probabilities sum to one, raising the target's probability necessarily lowers the others'.

What the gradient does, numerically

Continue the earlier toy example, and suppose the true next token was B (second position). The model gave B probability 0.245, so the loss is −ln(0.245) = 1.406. For softmax followed by cross-entropy, the gradient of the loss with respect to each logit has a pleasingly simple form: predicted probability minus the one-hot target.

p        = [0.665,  0.245,  0.090]
target   = [0,      1,      0    ]
dL/dz    = p - target = [ 0.665, -0.755, 0.090 ]

gradient descent moves each logit opposite to its gradient:
   z[A] decreases (it took 0.665 of the mass but was wrong)
   z[B] increases (the true token)
   z[C] decreases slightly

Over many examples, the weights drift toward configurations that give high probability to what actually follows in text. After a typical step the learning rate scales these updates; too large and training diverges, too small and it crawls. The broader training process, including validation and checkpoints, is in the LLM Overview.

At inference there is no true next token. The model's own output becomes part of its input. A three-iteration trace with illustrative numbers (a real vocabulary has tens of thousands of entries, so only the top few are shown) makes this explicit:

1. 1

   Iteration 1

   Context: The server returned status. Distribution (toy): " 500" 0.46, " 200" 0.30, " 404" 0.14, other 0.10. Greedy selection picks " 500". Context becomes The server returned status 500.
2. 2

   Iteration 2

   The full new context is processed (efficiently, thanks to the KV cache). Distribution: "," 0.38, "." 0.33, " (" 0.17, other 0.12. Sample "." (a greedy decoder would take ","). Context: The server returned status 500.
3. 3

   Iteration 3

   Distribution: "<end>" 0.55, " The" 0.25, "\n" 0.12, other 0.08. Pick the end-of-sequence token and stop. The finished text is The server returned status 500.

Step through the same three iterations as a picture: the context grows by one token each time, and each new context has its own distribution.

Each distribution depends on the tokens chosen earlier, so a mistaken choice changes every later distribution. That is the root of error propagation in generation: the model is conditioning on its own previous output as though it were reliable input. Generation ends when the model emits an end-of-sequence token, when a configured stop string appears, or when a maximum token count is reached; the last case can truncate an answer mid-sentence.

By the chain rule of probability, the probability of a whole sequence is the product of its per-token conditionals. If a model gave " 500" probability 0.6, then "." given that prefix 0.7, then the end token 0.5, the sequence probability is:

P("500 . <end>") = 0.6 x 0.7 x 0.5 = 0.21

in log space (what implementations sum, to avoid underflow):
ln 0.6 + ln 0.7 + ln 0.5 = -0.511 - 0.357 - 0.693 = -1.561   ( exp(-1.561) = 0.21 )

Choosing the highest-probability token at each step (greedy decoding) maximises each factor locally, which does not maximise the product. A tiny example: at step 1, token A has probability 0.5 and B has 0.4. After A the best next token has only 0.3 (the model is uncertain), but after B the best next token has 0.9. Greedy picks A and ends with at most 0.5 × 0.3 = 0.15, while the B path reaches 0.4 × 0.9 = 0.36. Searching for better whole sequences (beam search) exists, but modern chat systems mostly sample, partly because maximum-likelihood text can be repetitive and bland.

Because of this multiplicative structure, long sequences have tiny absolute probabilities even when each step was likely; comparing sequences of different lengths by raw probability therefore favours short ones, which is why per-token averages (perplexity) are used instead.

Greedy decoding takes the argmax: deterministic apart from tie-breaking and numerical noise. Sampling draws a token at random in proportion to its probability, so the 0.245-probability token in the earlier example is chosen about a quarter of the time. Sampling is what lets the same prompt produce different valid continuations. Two knobs reshape the distribution before the draw: Temperature sharpens or flattens it, and Top-K and Top-P remove unlikely candidates first. The details and their arithmetic are in those lessons, and are deliberately not repeated here.

Predicting the next token well across a vast corpus requires the model to learn grammar, code syntax, common facts, and how explanations are structured, because all of these reduce prediction error. That is why a single objective yields broadly useful language behaviour. But the objective rewards matching the distribution of the training text, not checking claims against the world. If a false statement is common in the corpus, or a plausible-sounding citation fits the pattern of the surrounding text, a high-probability continuation can be wrong. Probability here is the model's learned estimate of what text is likely, not a calibrated probability that a statement is true. See Hallucinations for what to do about it.

Inspecting log probabilities, where supported

Some APIs and most open-source runtimes can return the log probability of chosen tokens and of the top alternatives. This is useful for debugging, for flagging low-confidence spans, and for classification tasks where you compare the probabilities of candidate answers. Option names differ by provider, so this is a sketch rather than a specific API:

resp = client.generate(prompt="The server returned status",
                       max_tokens=3, logprobs=True, top_logprobs=3)

for step in resp.token_logprobs:
    chosen = step.token
    p = math.exp(step.logprob)                    # back to a probability
    alts = [(a.token, math.exp(a.logprob)) for a in step.top_alternatives]
    if p < 0.3:
        flag_for_review(chosen, alts)             # model was torn between candidates

Generation can only attend to what is in the input. The next lesson looks at that constraint: how much fits, what to do when it does not, and how to budget it.