Context Window

Watch a context window fill up, overflow, and force tradeoffs between history, retrieved data, and output space.

An LLM context window is the maximum number of tokens a model can process in a single request, covering the instructions, conversation history, retrieved documents, tool outputs and, in many APIs, the reply it is about to generate. It is not persistent memory and it is not training data: the model conditions only on what your application puts in the window for this call. For example, with a shared 32,000-token limit, a prompt of 25,000 tokens leaves 7,000 for the reply; actual limits and how input and output are counted depend on the model and API. Reliable long-running applications therefore treat the window as a budget: decide what goes in, what is summarised, what is retrieved on demand, and how much room to reserve for the answer.

A customer support assistant begins a conversation with a short system prompt. The customer describes a billing problem, the assistant asks clarifying questions, calls an order_lookup tool that returns a 1,500-token JSON record, retrieves three help-center articles, and the conversation goes on for forty messages. On the screen the user sees a clean chat history. The model sees none of that structure. It sees one long sequence of tokens that the application rebuilt from scratch for this turn.

This distinction drives everything that follows. The context is the transient input for one model call. Memory, in the product sense, is durable state your application keeps in a database and decides to put back into context. A model does not remember the previous call; each request starts from whatever your code assembles. If the customer's refund deadline disappeared from the assistant's replies, the question to ask is not "why did the model forget?" but "did our builder still include it?"

Providers describe limits in different ways, so read the documentation of the specific model. Two common patterns exist: a single shared limit where input plus output must fit within it, or separate maximums for input and for generated output. Either way, the reply you want has to be accounted for before you send anything, because a prompt that fills the window leaves no room for an answer.

Count more than the visible text. Chat formats add role markers and separators around each message (a few tokens each, and they add up across forty messages). Tool or function definitions are serialised into the prompt, so a set of ten tool schemas with long descriptions can consume thousands of tokens on every call. Images and files, where supported, are converted to tokens by provider-specific rules. And different tokenizers split the same text differently, so a token budget computed with one model's tokenizer is only approximately right for another model.

Assume an illustrative model with a shared 32,000-token capacity. The numbers below are planning assumptions for this example, not measurements of any product.

system instructions + tool schemas      1,000
conversation (trimmed history)          5,000
retrieved evidence (3-4 help articles) 12,000
tool outputs (order record, etc.)       4,000
reserved for generation                 3,000
                                       ------
planned total                          25,000

headroom = 32,000 - 25,000 = 7,000 tokens

Why keep 7,000 in reserve rather than fill the window? Token counts you compute locally may differ from the provider's count; the next user message is unpredictable in size; a tool might return more than expected; and a longer prompt costs more and slows down the first token. Headroom absorbs those variations. Note also that the 3,000 reserved for generation is a ceiling you set (for example max_tokens), not a promise that the reply will be that long.

Order and serialisation are design decisions. Stable content (policy, tool definitions) goes first, which is also friendly to providers that cache repeated prompt prefixes. Evidence is usually labelled with source IDs so answers can cite it. Dates and units are written unambiguously. When something has to be cut, the builder, not the model, decides what, and it should do so by explicit priority rather than by blindly dropping the oldest text. This is why the visible UI history may not equal the model input: the UI shows all forty messages; the input might contain a structured summary, the last six messages verbatim, and the evidence for the current question.

Within the window, attention is causal: each token can use only earlier ones. In principle every earlier token is reachable, but in practice the model's use of distant information is imperfect. Research on long-context question answering, such as the "Lost in the Middle" study (Liu et al., 2023), found that for the models and tasks tested, accuracy was often highest when the relevant passage was near the beginning or end of the input and lower when it was buried in the middle. That is an empirical tendency, not a universal law: newer models, training methods, and tasks can change it, which is exactly why you should measure it for your own model (see the evaluation section below). Also, a large advertised window does not imply the model uses all of it equally well, and padding the prompt with marginally relevant material can distract rather than help.

Cost, latency, and the KV cache

Processing the prompt (prefill) costs more with more input tokens, so time to first token usually grows with prompt length. During generation, the runtime keeps a key and value tensor for every token of every layer, the KV cache, whose memory grows linearly with context length and with the number of concurrent requests. A rough form is 2 × layers × kv_heads × head_dim × bytes_per_value × tokens per request, so long contexts reduce how many requests a GPU can serve at once. Techniques such as grouped-query attention, quantised caches, paged allocation, and sliding-window attention change those numbers, and architectures differ, so avoid claiming a single complexity or memory figure for "LLMs" in general; consult the model's documentation and measure.

The explorer below evaluates that formula for an illustrative configuration, so you can see how context length and the number of KV heads change the memory one request needs.

Every strategy trades information for space. What matters is knowing what you give up and how it fails.

Sliding window keeps only the most recent N messages. It is simple and predictable, but anything stated earlier (the customer's account type, a constraint such as "do not offer store credit") silently disappears. It is fine for casual chat where only recent turns matter and dangerous for tasks with standing constraints.

Summarisation replaces old turns with a model-written summary. It preserves gist cheaply, but summaries are lossy and can introduce errors, drop numbers and exact wording, and, if repeated, drift (a summary of a summary). A summarisation step can also launder untrusted text into something that looks like trusted instructions, a risk discussed below.

Structured state extraction maintains explicit fields (customer ID, order ID, stated goal, confirmed facts, constraints, open questions) updated after each turn and validated by code where possible. It is compact and auditable and survives truncation of raw history, but it only captures what your schema anticipated, so include a free-text notes field and test with real conversations.

Selective retention keeps specific messages verbatim (decisions, commitments, corrections, anything containing identifiers) and discards chit-chat. It needs a rule for what counts as important, and a wrong rule loses the key message.

Retrieval and external memory store the full history or knowledge base outside the prompt and fetch only the pieces relevant to the current question. This scales far beyond the window but is only as good as retrieval: a missed document is as absent as a truncated one. See RAG Architecture for the pipeline; this lesson does not repeat it.

Drag the slider to see which of the 40 messages a sliding window keeps, and which key facts from the support conversation silently fall out.

By message 40 the raw transcript is about 9,000 tokens, the evidence for the latest question 4,000, and the tool output 1,500. The builder compresses the early history into state, keeping recent messages verbatim.

[1]  user: Hi, I was charged twice for order 88213.
[2]  assistant: Sorry about that. Can you confirm the email on the account?
...  (36 more turns: shipping address change, a coupon, an escalation request) ...
[39] user: Also, I need this resolved before March 14, I'm travelling.
[40] user: So what are my options?

CONVERSATION STATE (maintained by code, last updated turn 40)
  customer_id: C-4471            order_id: 88213
  issue: duplicate charge (2 x $42.10, 2024-02-02)    status: not yet refunded
  confirmed: email verified (turn 3); address updated (turn 11)
  constraints: customer needs resolution before March 14; DO NOT offer store credit (policy POL-7)
  unresolved: refund timeline; whether the coupon applies to the corrected charge
  evidence_refs: KB-212 (duplicate charges), KB-340 (refund timelines), order_record#88213

RECENT MESSAGES (verbatim): turns 35-40

Three safeguards keep this from losing something important. First, designate "never drop" categories (constraints, commitments, identifiers, deadlines) and test that they survive compression. Second, keep the raw transcript in storage so a human or a later retrieval step can recover details. Third, when the model's answer depends on a detail that is absent from the compressed context, the right behaviour is to say so or to retrieve it, which you encourage with instructions and verify with tests.

If the whole knowledge base fits, why not paste it in? Sometimes you should. For a single document that fits comfortably and needs holistic reasoning, long context is simpler and avoids retrieval errors. For larger or changing corpora, retrieval usually wins on cost and latency (you pay for a few thousand relevant tokens, not a million), makes it easier to show citations to exact passages, keeps irrelevant text out, and lets you apply per-user access control before anything enters the prompt. Long context has no retrieval-miss failure but has attention and cost trade-offs described above. Many systems combine them: retrieve broadly, rerank, and then supply a generous but selected set. The Context Injection lesson covers how retrieved chunks are assembled into prompts.

Prompt injection in retrieved content. A help article or customer email can contain text such as "ignore previous instructions". The model reads it as tokens in the same window as your instructions, so wrap evidence in clear delimiters, tell the model it is data, and, more importantly, never rely on that instruction as a security control. Enforce permissions in code. See System Prompts vs User Prompts.

Stale summaries. A stored summary says the refund is "pending", but it was issued yesterday. Store timestamps and source versions with state, and refresh from systems of record rather than trusting a summary.

Cross-user leakage. If summaries, caches, or memory stores are keyed carelessly, one customer's data can appear in another's prompt. Scope every stored item by tenant and user, and filter before retrieval, not after generation.

Redaction and provenance. Remove or tokenise secrets and personal data before they enter the window, and attach source identifiers to each piece so you can later explain why the model said what it said.

Measure rather than guess. For a context strategy, track: token utilisation (input tokens ÷ capacity per request, as a distribution), truncation rate (fraction of requests where something was dropped or the reply hit the length limit), recall by position, answer quality, latency, and cost.

A concrete experiment for position sensitivity: build 200 questions, each answerable from one fact. For each, embed the supporting passage at the start, middle, or end of a 12,000-token context of distractor passages, and record whether the answer is correct. Compare accuracy by position, per model. If middle-position accuracy is notably lower for your model, place critical evidence first and last, or retrieve fewer, better passages. No result is assumed in advance; the experiment tells you.

def build_prompt(state, history, evidence, tools, capacity=32000, reply_reserve=3000, headroom=2000):
    budget = capacity - reply_reserve - headroom
    parts = [("policy", POLICY_AND_SCHEMAS, MUST_KEEP),
             ("state",  render(state),      MUST_KEEP)]

    used = sum(count_tokens(text) for _, text, _ in parts)
    if used > budget:
        raise ContextOverflow("mandatory content exceeds budget")   # fail loudly

    # Add optional content by priority until the budget is reached
    for name, items in [("recent", history[-6:]), ("evidence", rank(evidence)), ("older", history[:-6])]:
        for item in items:
            cost = count_tokens(item.text)
            if used + cost > budget:
                log.info("context_truncated", section=name, dropped=item.id)
                continue
            parts.append((name, item.text, OPTIONAL))
            used += cost

    prompt = serialise(parts)
    metrics.record(input_tokens=used, capacity=capacity, truncated=any_dropped)
    return prompt, reply_reserve

# Fallback if the provider still rejects the request as too long:
#   1. drop the lowest-ranked evidence  2. re-summarise older history  3. ask the user to narrow the question

Everything in the window has a role. The next lesson separates instructions from data and shows what can and cannot be trusted.

1. 1

   Budget the window explicitly: capacity covers instructions, history, evidence, tool output and the reserved output, plus chat-format overhead. Keep a reserve (7,000 of 32,000 tokens in the worked example), because locally computed counts can differ from the provider's.

2. 2

   Put stable content (policy, tool definitions) first and variable content last; order and serialisation are design decisions, and it also suits prefix caching.

3. 3

   Every strategy for a growing conversation trades information for space: a sliding window forgets early statements, a summary is lossy and can go stale, and retrieval can miss. Choose by what you can afford to lose.

4. 4

   Designate never-drop fields (constraints, commitments, identifiers) and keep them verbatim or in structured state, and store timestamps and sources with summaries so stale ones can be detected.

5. 5

   A large advertised window does not mean the model uses all of it equally well. Measure position sensitivity on your own model, for example 200 single-fact questions with the fact placed at different positions.

6. 6

   Scope memory, summaries and caches by tenant and user, treat retrieved and tool text as untrusted, and redact secrets and personal data before they enter the window.