LLM Overview

How LLMs work, in one pass: an LLM splits input text into tokens, converts those tokens into numeric vectors, pushes the vectors through stacked Transformer layers that use self-attention to relate every token to every other token, and projects the result into a probability distribution over the next token — then repeats that same step, one token at a time, until the response is complete. This overview connects the major pieces of that pipeline so the rest of the section has a common mental model.

A large language model (LLM) is a neural network trained on sequences of text to estimate which token is likely to come next, given the tokens so far. An LLM works by applying that estimate repeatedly. It picks one token, appends it to the input, and estimates again until a stop condition is reached. The reply is generated rather than looked up, and this autoregressive loop is the dominant design, not the only one. The model (weights), the serving runtime that executes it, and the application around it are three separate things.

The rest of this chapter follows a single realistic request from the moment a developer presses Enter to the moment the last token streams back, then goes backwards to explain how the model got its abilities in the first place. Mathematical detail lives in the lessons it links to; this page is about how the pieces fit together so that you can reason about an LLM-backed system as an engineer.

To jump ahead: Tokens Explained covers tokenization, and Context Window covers how much text a model can consider at once. Deeper dives on next-token prediction, fine-tuning and model serving have their own lessons.

A developer is on call. A service logs QueryTimeoutError: statement exceeded 30000 ms and she opens the company's AI assistant in her IDE. She types: "Why would this Postgres query time out?" and pastes the query and the error. The assistant starts replying within a second, and the answer appears word by word.

Three things about this interaction surprise people the first time they look closely. First, the model never saw the sentence the way she typed it: it saw a list of integers. Second, the application sent far more than her message: a hidden instruction block, the earlier turns of the conversation, perhaps the open file. Third, the answer was not retrieved from anywhere. At no point does a finished paragraph exist, waiting to be returned. The paragraph exists only because the loop kept running.

Here is what the application sends and what the model actually receives. The application builds a structured request; the serving layer flattens it into one token sequence using the model's chat template.

{
  "model": "assistant-large",
  "messages": [
    {"role": "system", "content": "You are a senior database reliability assistant. Be concise."},
    {"role": "user",   "content": "Why would this Postgres query time out?\n\nSELECT ... FROM orders o JOIN ...\nError: statement exceeded 30000 ms"}
  ],
  "max_tokens": 600,
  "temperature": 0.2
}

<start_of_system> You are a senior database reliability assistant. Be concise. <end>
<start_of_user>   Why would this Postgres query time out? SELECT ... <end>
<start_of_assistant>
  -> [ 2938, 17, 8841, 302, ... ]   a few hundred to a few thousand integers

"The LLM" is used loosely for at least six different things, and confusing them leads to bad designs. It helps to separate them:

• Weights: the learned numbers (parameters), a large file. Frozen once training ends.
• Tokenizer: a fixed text-to-integer mapping that ships with the weights. Not a neural network.
• Inference runtime: the software and GPUs that execute the weights and manage memory such as the KV cache.
• Application: ordinary code that builds prompts, calls the runtime, parses the output, and enforces rules.
• Tools: functions the application exposes (SQL runner, search, ticket API) that the model can ask to have called.
• Retrieved data: documents or records the application places into the input at request time.

Formally, the weights define a conditional probability distribution over the next token: P(token | all previous tokens). Chaining those conditionals gives a probability for a whole sequence. Many text-generating LLMs, including the best-known assistants, are autoregressive transformer models of the decoder-only kind, which is the architecture this chapter describes. Other architectures exist (encoder-decoder models, state-space and hybrid models, diffusion-based text models), so treat "autoregressive transformer" as the dominant design, not a law.

A boundary worth memorising: the model alone does not remember last week's conversation, does not know your database schema, and cannot run a query. If the assistant appears to do these things, application code is storing, retrieving, or executing on its behalf and then feeding the results back as more tokens.

Neural networks do math on numbers, so the first job is turning characters into numbers. A tokenizer splits text into tokens, pieces that are frequent in the training text: whole common words, word fragments, punctuation, or even single bytes for rare strings. Each token has a fixed integer ID in the model's vocabulary, which typically holds tens of thousands to a few hundred thousand entries. The details are covered in Tokens Explained.

text      : "Why does my Postgres query time out?"
tokens    : ["Why", " does", " my", " Post", "gres", " query", " time", " out", "?"]
token IDs : [ 5195,   1587,   856,   5398,  16591,    3319,    892,   704,   30 ]   <- made-up numbers

Each ID then indexes a row in an embedding table, producing a vector of a few thousand numbers. At this stage the vector for " out" is the same wherever the word appears; the table is just a lookup. Position information is added as well, because the transformer would otherwise see an unordered bag of vectors (modern models commonly use rotary position encodings applied inside attention rather than a separate vector added at the start, but the purpose is the same).

The vectors pass through a stack of transformer layers. Each layer lets every position gather information from earlier positions and then transforms the result. By the final layer the vector at the last position is no longer "the word out"; it is a summary of what the whole context implies should come next. A final linear projection multiplies that vector by a matrix with one row per vocabulary token, giving one raw score, a logit, for every possible next token. The lesson on Next-Token Prediction turns those logits into probabilities and shows the arithmetic, so this page does not repeat it.

A transformer layer has two sub-blocks, each wrapped with a residual connection and normalisation. The derivations are in Transformer Basics and Attention Mechanism; here is the engineering intuition. The architecture comes from Vaswani et al., 2017, “Attention Is All You Need” (arXiv:1706.03762).

Causal self-attention. Every position produces three vectors: a query ("what am I looking for?"), a key ("what do I contain?"), and a value ("what do I contribute if someone attends to me?"). A position compares its query with the keys of all earlier positions (and itself), turns the similarities into weights that sum to one, and takes the weighted sum of the values. "Causal" means a position may not look at later positions, which is what makes next-token training and generation consistent: when predicting token t+1, the model has only seen tokens up to t. Many attention heads run in parallel so that different heads can track different relationships.

Feed-forward network. After attention mixes information across positions, a small two-layer network is applied to each position independently. Much of a model's parameter count sits here, and it is where a lot of stored knowledge and pattern-matching appears to live, although interpretability research is still working out exactly how.

Residual paths and normalisation. Each sub-block adds its output to its input rather than replacing it, and activations are normalised. This keeps gradients well-behaved through dozens of layers, and it lets later layers refine an existing representation instead of rebuilding it.

Context changes what a token means

Consider two sentences the assistant might see: "The connection pool is exhausted" and "The connection was refused by the firewall". The token "connection" has the same embedding in both. After a few layers of attention it does not: in the first sentence it has absorbed "pool" and "exhausted", pulling it toward resource management; in the second it has absorbed "refused" and "firewall", pulling it toward networking. The vector at the final position of a long prompt is therefore the product of attention over everything before it, which is exactly what is needed to decide, for example, whether the next token should be about connection limits or about firewall rules.

Everything above is the machinery. The abilities come from what is written into the weights during training, and that happens in stages.

1. 1

   Collect and curate a corpus

   Web pages, books, code, papers, and licensed or synthetic data. This is where rights, privacy, and consent questions enter: what may be used, what personal data must be removed, and what has to be excluded. Quality filtering and deduplication follow, because repeated documents are memorised more easily and over-weight whatever they say.
2. 2

   Train a tokenizer and tokenize everything

   The vocabulary is learned from the corpus (for example by byte-pair encoding) and then frozen. The corpus becomes a very long stream of token IDs.
3. 3

   Pretrain with next-token prediction

   Slice the stream into sequences. For every position, the model predicts the token that follows; the loss measures how much probability it gave the true token. Because the true next token is always available from the text itself, no human labelling is required. This is teacher forcing: the model is always shown the real prefix, never its own earlier guesses. See Next-Token Prediction for the loss.
4. 4

   Validate and checkpoint

   A held-out slice of data measures loss on text the model has not trained on. Checkpoints are saved regularly, so a diverging run can be rolled back and the best weights selected.
5. 5

   Post-train for assistant behavior

   Supervised fine-tuning on instruction/response examples, then preference-based optimisation, teach the base model to behave like an assistant. See Fine-Tuning and Model Alignment.

Two ideas deserve emphasis. The first is the difference between parameters and memorised text. A model with tens or hundreds of billions of parameters is trained on trillions of tokens, so the weights cannot be a verbatim archive; they are a compressed, lossy statistical summary. Some passages, especially heavily duplicated ones, can nevertheless be reproduced nearly verbatim, which is one reason deduplication and privacy filtering matter. The second is generalisation: the model must assign good probability to text it has never seen, which forces it to learn structure (syntax, facts that recur, how stack traces relate to causes) rather than only surface strings.

Data quality shapes the result directly. If a corpus contains many confident but wrong answers about a framework's API, the model has learned that such text is likely. If it contains little text about a niche topic, the model's distribution there is poorly informed, and it will still produce fluent output because fluency was what training rewarded.

Inference starts after the weights are frozen. For each request the runtime executes the following loop.

1. 1

   Assemble messages

   The application builds the message list; the chat template flattens it to one string with role markers.
2. 2

   Tokenize

   The string becomes token IDs. Billing, latency, and the context limit are all measured in these tokens.
3. 3

   Prefill

   The whole prompt is processed in one parallel pass. Every layer computes keys and values for every prompt position and stores them in the KV cache. The last position yields the first set of logits.
4. 4

   Choose a token

   A decoding rule picks one token from the distribution: greedy, or sampling shaped by temperature and top-k / top-p.
5. 5

   Append and decode

   The chosen token is appended. Only that one new position is processed; earlier positions are not recomputed because their keys and values are already cached.
6. 6

   Stop

   Generation ends when the model emits an end-of-sequence token, a stop string is matched, or the max token limit is hit.

The KV cache is why this is practical. Without it, producing token 500 would require reprocessing tokens 1 to 499 from scratch. With it, each decode step does a small amount of new computation but must read the whole cache, so decode is typically limited by memory bandwidth rather than arithmetic. Prefill is the opposite: lots of parallel work, usually compute-bound. That is why the delay before the first token (time to first token) grows with prompt length, while the speed of the stream afterwards (time per output token) is comparatively steady. The cache also consumes GPU memory in proportion to context length and the number of concurrent requests, which is one of the main constraints on serving long contexts; Context Window goes into that trade-off.

Adjust the prompt and output lengths to see how prefill and decode contribute to the delay and the total time.

Return to the on-call developer. After prefill, the assistant must start its answer. The numbers below are invented for illustration; a real model has a vocabulary of tens of thousands of tokens and its own probabilities, which you could only observe through an API that exposes log probabilities.

Step 1: context ends with the assistant marker

Selected: "The". Context is now: …[assistant] The.

Step 2: the model conditions on "The"

Selected: " query". Context: …[assistant] The query.

Step 3: the model conditions on "The query"

Selected: " likely". The reply is heading toward "The query likely performs a sequential scan because…". Each step's distribution depends on every token chosen earlier, which is why a different choice at step 1 ("This" instead of "The") would send the whole answer down a different path. The loop ends only when the model assigns high probability to its end-of-sequence token, or when the token limit cuts it off, which is why a reply can stop mid-sentence if max_tokens is too small.

Weights encode patterns that were frequent or consistent in training data: how SQL works, what a sequential scan is, how lock contention manifests. They do not encode the state of her production database. The model has no way of knowing that the orders table grew to 400 million rows last month or that a migration yesterday dropped an index. That information must reach the model as tokens, either pasted by the user or retrieved by the application.

Two limits follow. Knowledge has a training cutoff: anything that changed after the data was collected is missing, and a newer library version may be described by its older API. And because the training objective rewards plausible continuations rather than verified ones, the model will produce a fluent answer even where its internal evidence is thin; nothing in the loop forces it to say "I don't know". A confident explanation of a table it has never seen is therefore a pattern completion, not a diagnosis. Hallucinations covers how to detect and contain this; RAG Architecture covers supplying current or private facts.

A production assistant is mostly ordinary software. Separating what the model does from what deterministic code does makes the system testable.

The model produces text, including text that requests a tool call such as run_explain(query). Everything else is code you write and can unit test: authenticating the user, deciding which history to include, calling the database tool with the user's own permissions, checking that the output parses, redacting secrets, recording token usage, and deciding what to do on timeout. A rule such as "this user must never see another tenant's data" belongs in that code, not in the prompt. The lesson on System Prompts vs User Prompts explains why.

The same architecture over time, including a tool round trip and streaming:

Suppose the on-call assistant will also read production logs and query snippets that include customer identifiers. A team of six engineers must decide where the model runs. The right decision follows from constraints, not from a general ranking.

A hosted API from a model provider gives the strongest models with no GPU operations, quick iteration, and elastic capacity, but prompts leave your network (so contract terms on data retention and training use matter), cost scales per token, and the provider can change or retire a model version. A self-hosted open-weight model keeps data inside your boundary and allows fine-tuning and stable version pinning, but your team now owns GPU capacity planning, batching, the serving runtime, upgrades, and on-call for it. Open-weight models also differ in license terms; "open" does not automatically mean unrestricted commercial use.

For this team the deciding questions would be: Can logs be redacted before they leave the network? Does the provider offer a data-processing agreement with zero retention? Is the quality of a smaller open model sufficient on our own incident examples, measured, not assumed? Is there anyone who will carry a pager for a GPU cluster? Many teams answer by starting with a hosted model behind an internal gateway (which preserves the option to switch) and moving only the sensitive or high-volume paths to self-hosting once evaluation shows it is worth the operational cost. Latency and throughput are shaped by both choices: time to first token depends on prompt length and provider load, and sustained throughput on batching and GPU memory.

The following provider-neutral pseudocode shows the deterministic wrapper around one model call. It is a sketch, not a runnable library: client, redact, and the exception types stand for whatever your stack provides.

def answer_incident_question(user, question, attachments):
    # 1. Deterministic preparation: permissions and data hygiene first
    context = redact(attachments, rules=PII_RULES)
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT_V3},
        {"role": "user",   "content": question + "\n\n" + context},
    ]

    # 2. Bound the cost before calling
    if estimate_tokens(messages) > INPUT_BUDGET:
        messages = trim_attachments(messages, INPUT_BUDGET)

    # 3. Call with timeout, limited retries, and a recorded trace
    for attempt in range(3):
        try:
            resp = client.generate(model="assistant-large", messages=messages,
                                   max_tokens=600, timeout_s=30)
            break
        except (Timeout, RateLimited) as err:
            log.warn("llm_retry", attempt=attempt, error=str(err))
            sleep(backoff(attempt))
    else:
        return fallback_message("The assistant is unavailable; see runbook DB-12.")

    # 4. Record what was used (usage fields are provider-specific)
    metrics.record(user=user.id, model=resp.model,
                   input_tokens=resp.usage.input_tokens,
                   output_tokens=resp.usage.output_tokens,
                   finish_reason=resp.finish_reason)

    # 5. A truncated reply is not a complete answer
    if resp.finish_reason == "length":
        return resp.text + "\n\n[Reply truncated. Ask me to continue.]"
    return resp.text

Now retrace the opening incident with the mechanics in hand. The developer's question and pasted query were redacted and combined with a system prompt into one message list. The template turned it into a sequence of integers. Prefill processed those integers in one pass and filled the KV cache. The model produced a probability distribution over the vocabulary, a decoding rule chose "The", and the loop repeated, each step conditioning on everything before it, until the end-of-sequence token appeared. The diagnosis it produced reflects patterns from training plus the text of the query she pasted; it does not reflect the actual table statistics unless the application also ran EXPLAIN and supplied the result. So the sensible next engineering step is not a better prompt but a tool that gathers that evidence.

The loop at the center of this chapter, producing a distribution and choosing a token, is worked through with real arithmetic in the next lesson.