Inference Infrastructure

Watch a single user's prompt travel through tokenization, transformer layers, and token-by-token generation — then see what happens when millions of users ask questions at the same time. ChatGPT is only a familiar entry point: the pipeline shown is a generic, illustrative LLM serving path, not a description of OpenAI's actual systems.

What happens during LLM inference? The engine tokenises the prompt, runs prefill, decodes output tokens one at a time, batches many requests together and streams tokens back as they are produced. AI inference in general means running a trained model to get predictions; for LLMs it is this token-by-token loop. For each request the engine first runs a prefill pass that reads the whole prompt in parallel and builds a per-request memory called the KV cache; it then runs a decode loop that produces one token per iteration, appending to that cache. Efficiency comes from three mechanisms working together: the KV cache, which means earlier tokens are never recomputed; continuous batching, which lets many requests share each decode iteration and lets requests join and leave at token boundaries; and a scheduler that admits only as much work as memory and latency targets allow. What users feel is two numbers, time to first token (dominated by queueing and prefill) and time per output token (dominated by decode).

Everything here concerns a model you run yourself. If your application calls a hosted API, this machinery is hidden behind the provider's endpoint: you still see TTFT, token rate and rate limits, and the same reasoning explains them, but you do not choose batch sizes or buy accelerators. For how inference differs from training as a concept, see Training vs Inference. This chapter covers what happens inside one serving engine; how it is exposed, routed and scaled is the job of Model Serving, and the hardware it runs on is covered in AI Accelerators. All numbers below are illustrative: they are chosen to make arithmetic checkable, not to describe any product.

The enterprise support assistant receives a policy question. After retrieval and prompt assembly (see Context Injection) the prompt is 2,000 tokens: a system prompt, a handful of authorised policy passages and the user's question. The user's interface streams the answer, and the answer turns out to be 300 tokens long. In the self-hosted deployment one worker process on one accelerator handles the request. The timeline, with illustrative timings, looks like this:

The first 215 ms are spent before the user sees anything, and they scale with input length and with how busy the engine is. After that the stream paces itself at one token per 20 ms (50 tokens per second), independent of how long the prompt was, apart from a small slowdown as the cache grows. The last token appears at 0.215 s + 299 × 0.020 s = 6.195 s. Note the 299: the first token is already counted in TTFT, so a 300-token answer has 299 further intervals.

These two phases fail differently, which is why one latency number is not enough. A long retrieved context makes TTFT worse and leaves the streaming speed unchanged. A busy engine sharing each iteration with many requests keeps TTFT acceptable but stretches the per-token interval. A product team that says “the assistant is slow” has to say which of these it means, because the fixes (shorter prompts, prefix reuse and admission limits versus smaller batches, quantisation and more replicas) do not overlap.

Between the network socket and the first streamed character a request passes through a fixed series of stages. Text is tokenised into integer ids by the model's tokenizer (see Tokens Explained); the scheduler places the request in a waiting queue and decides when to admit it; the engine runs prefill, which yields the logits (scores over the vocabulary) for the position after the last prompt token; a sampling rule such as greedy selection or temperature with top-p (see Temperature and Top-K and Top-P) picks the first output token; the decode loop then repeats until the model emits an end-of-sequence token (EOS), a stop string matches, the output limit is reached, or the client cancels. The engine detokenises incrementally and streams text as it goes.

Three iterations with the state written out

The state that matters is the number of positions held in the KV cache, the token fed into the model, and the token that comes out. Token ids below are placeholders.

1. 1

   Prefill (iteration 0)

   Input: prompt tokens p1 … p2000 processed together. The cache gains 2,000 positions. Logits at position 2,000 give token t1 (say id 5821, “Refunds”). Nothing has been fed back yet, so t1 is not in the cache.
2. 2

   Decode step 1

   Input: only t1. Its K and V are computed and appended: cache length 2,001. Its attention reads all 2,001 positions. Logits give t2 (id 318, “for”).
3. 3

   Decode step 2

   Input: t2. Cache length 2,002; logits give t3. Step 3 feeds t3 and makes length 2,003, and so on. If the answer ends with EOS as token 300, the cache stops at 2,000 + 299 = 2,299 positions, because the last token is never fed back.

Two details matter later. Each decode step does the same fixed work for the weights (they are read once per step, whatever the cache length) plus work that grows with the cache (attention over all cached positions). And output tokens are sequentially dependent: token 3 cannot be computed before token 2 is chosen, so one request cannot be parallelised across its own output. The only available parallelism is across different requests, which is the whole motivation for batching.

Prefill feeds N prompt tokens through the network at once, so each weight matrix is multiplied by a matrix with N rows. Reading a weight from memory is then amortised over thousands of multiply-adds, and the accelerator's arithmetic units tend to be the limiting resource. A rule of thumb for dense transformers is that the forward pass costs about 2 × (parameters) FLOPs per token, ignoring attention's extra term, which is minor at these lengths. For an illustrative 8-billion-parameter model and 2,000 tokens:

FLOPs ≈ 2 × parameters × prompt_tokens
      = 2 × 8e9 × 2,000
      = 3.2e13 FLOPs  (32 TFLOP)

Delivered throughput (assumption, NOT the chip's peak): 200 TFLOP/s
Prefill time ≈ 3.2e13 / 2e14 = 0.16 s = 160 ms

Decode processes one new token per sequence. For a single sequence every weight is streamed from accelerator memory to do roughly one multiply-add per weight, so memory bandwidth, not arithmetic, tends to set the speed. If the weights occupy 16 GB (8 billion parameters × 2 bytes, decimal GB) and the memory system delivers an illustrative 2 TB/s, a decode step cannot take less than 16 GB / 2 TB/s = 8 ms, however many FLOP/s the chip has. That is a floor of 125 tokens/s for one sequence. Add the attention reads of the KV cache and the floor rises: a batch of 32 sequences with 2,300 cached tokens each reads another 32 × 287.5 MiB ≈ 9.65 GB per step, or about 4.8 ms, for a combined floor of 12.8 ms per step. The weight read is paid once for the whole batch, so 32 sequences emit 32 tokens per ~12.8 ms floor (about 2,500 tokens/s) whereas one sequence emits 1 per 8 ms (125 tokens/s). That asymmetry is why batching decode is so profitable.

The words “tend to be” are deliberate. Whether a phase is compute- or bandwidth-bound depends on the model, batch size, sequence length, precision, kernel quality and chip. Short prompts make prefill small enough to be inefficient; very large decode batches eventually become compute-bound; a long context makes attention reads dominate. Measure on your own configuration. The 160 ms and 8 ms above are bounds derived from assumed specs, and real engines land somewhat above them: this chapter uses a delivered 20 ms per step at batch 32 (about 1.6× the floor) as its running assumption.

Self-attention lets each new token look at every earlier token. For each layer and each attention head, a token contributes a key vector (used to decide how relevant that token is to a later query) and a value vector (the content that gets mixed into the result). Both depend only on tokens at or before that position, so once computed they never change for the lifetime of the sequence. Recomputing them for 2,000 earlier tokens at every one of 300 decode steps would repeat the prefill work 300 times. The KV cache keeps them in accelerator memory instead, so each decode step computes K and V only for the single new token and reads the rest. The queries are not cached, because each query is used once.

The cost is memory that grows linearly with sequence length and linearly with the number of concurrent sequences. The size of one token's entry is a product of the model's architecture:

bytes_per_token = 2 (K and V) × layers × kv_heads × head_dim × bytes_per_element
cache_bytes     = bytes_per_token × cached_tokens

Take an illustrative model with 32 layers, 8 key-value heads, a head dimension of 128 and 16-bit elements (2 bytes). Models do not all have these dimensions; layer count, head count and head size vary widely, so always read them from the model's configuration file before sizing anything.

bytes_per_token = 2 × 32 × 8 × 128 × 2
                = 131,072 bytes = 128 KiB per token

4,096 tokens    = 131,072 × 4,096
                = 536,870,912 bytes
                = 512 MiB  (2^20 bytes per MiB; ≈ 0.54 GB decimal)

Our request: 2,000 + 299 = 2,299 cached tokens at the end
                = 131,072 × 2,299 ≈ 301.3 MB ≈ 287.4 MiB   (≈ 287.5 MiB if rounded up to 2,300 tokens)

The number of KV heads, not the number of query heads, sets the cache size. Original multi-head attention keeps one K/V pair per query head. Multi-query attention (MQA) shares a single K/V head across all query heads; grouped-query attention (GQA) shares one K/V head among a group of query heads, which is what “8 KV heads” means here. If the same illustrative network had 32 KV heads instead of 8, the cache would be four times larger. Precision is the other lever: storing the cache at 8 bits per element halves it, at some risk to accuracy that must be measured.

Per-sequence cache size at 4,096 tokens for the same 32-layer, 128-dimension illustration with different K/V head counts and element sizes (bars are relative to 2,048 MiB). Whether a shipped model offers any of these options is a fact about that model, not about the serving engine.

Now the pool. Assume an 80 GiB accelerator (illustrative). The 8-billion-parameter weights at 2 bytes occupy 16 × 10⁹ bytes = 14.9 GiB (decimal 16 GB; this chapter converts explicitly because the two units differ by about 7%). An engine typically lets you cap the fraction of memory it may use; assume 90%, or 72 GiB, leaving 8 GiB for activations, kernel workspace and the runtime. That leaves 72 − 14.9 = 57.1 GiB for KV. Divided by 512 MiB per worst-case 4,096-token sequence, the pool holds about 114 sequences; divided by 287.5 MiB it holds about 203 of our actual requests. With 32 active sequences live memory is only 9.0 GiB, so most of the pool is allocated but not utilised. Memory therefore is not what limits this particular fleet; bandwidth and scheduling are. The picture changes with long contexts: at 32,768 tokens one sequence costs 4 GiB, so the same pool holds only 14 of them.

Why engines allocate in blocks

A naive engine reserves a contiguous region for every request's maximum length (here 512 MiB for 4,096 tokens) even though most requests stop earlier, and the reserved-but-unused tail plus fragmentation between regions wastes memory that could host more sequences. Paged designs split the pool into small fixed blocks (16 tokens = 2 MiB in this example) and give each sequence a table that maps logical block numbers to physical ones, so a sequence grows one block at a time and a finished sequence returns its blocks to a free list. The waste is at most one partially filled block per sequence: our 2,300-token request needs 144 blocks (143.75 rounded up) = 288 MiB rather than a 512 MiB reservation. vLLM's README lists PagedAttention among its features for exactly this purpose; its description of the technique is in the paper it cites, and other engines have their own variants.

Block tables also make sharing possible: two sequences whose first blocks hold identical prompt prefix can point at the same physical blocks. That is the mechanism behind prefix reuse, discussed below; the validity rules for reusing cached computation across tenants are a topic for Caching in AI Systems, which deliberately does not repeat this arithmetic.

Because decode is limited by streaming weights, running one sequence at a time wastes the accelerator. If several sequences advance in the same step, the weights are read once and applied to all of them. The question is when a request may join the group and when it must leave. Continuous batching answers it by letting requests join and leave at every decode iteration, so a freed slot is reused immediately instead of idling until the slowest request in the group finishes.

Static batching collects a fixed group, runs it until every member has finished, and only then starts the next. It is simple, but lengths differ: a request that needs 3 output tokens occupies its slot until the 8-token neighbour completes, and a newly arrived request waits for the whole group. Consider three requests A (6 output tokens), B (3) and C (8) started together, and a request D (4 tokens) that arrives at iteration 2. Time is counted in decode iterations.

Continuous batching (also called in-flight or iteration-level batching) makes the scheduling decision at every token boundary instead of every batch boundary. After each iteration the scheduler removes sequences that finished, and admits waiting requests into freed slots, provided memory and token budgets allow it. A new request runs its prefill and then joins the next decode iteration.

D now finishes at iteration 7 instead of 12 and the batch is never left running with an empty slot while work waits. The bookkeeping is more demanding: sequences in the same iteration have different lengths, so attention must handle ragged cache sizes, and every admission needs memory for the newcomer's prompt. That is one reason block-based cache allocators and continuous batching appear together in modern engines. The vLLM README lists continuous batching, chunked prefill and prefix caching as features (README); other engines document their own scheduling policies, and you should read the version you plan to deploy.

Throughput versus latency. A larger batch amortises the weight read across more tokens and so increases aggregate output tokens per second, but each iteration also reads more cache and does more work, so the interval between a given user's tokens grows. In the running example, at batch 1 the floor is 8 ms per token for one stream; at batch 32 it is 12.8 ms per token for each of 32 streams. Aggregate rate rose about 20× (125 to ~2,500 tokens/s at the floor) while per-stream speed fell by 1.6×. Throughput gains flatten as the batch grows, while per-stream latency keeps increasing; operators pick a point on that curve by the p95 latency target, not by the peak throughput number.

Prefill interacts with batching. A long prompt joining the batch occupies the accelerator for one large forward pass; if it is run whole, every decoding sequence pauses for its duration (head-of-line blocking). A common mitigation is chunked prefill: split the prompt into slices of a few hundred tokens and interleave each slice with a decode step, trading slightly higher TTFT for the newcomer for smoother ITL for everyone else. Whether and how an engine supports this is configuration-specific.

An engine that accepts every request will eventually accept more than it can finish. The scheduler needs explicit limits, which are of different kinds. A request-rate or token quota per tenant (enforced at the gateway) protects shared capacity from one noisy customer. A max active sequences limit caps the decode batch size and thus per-token latency. A token budget per iteration bounds how many prompt tokens may be prefilled alongside decode, which bounds the stall prefill can cause. And a memory check ensures there are free KV blocks for the newcomer's prompt plus growth room for everything already running.

A request moves through a small state machine. Waiting requests are admitted when the checks pass, preempted sequences return to the queue, and cancelled, finished or rejected requests release their memory.

An overload example

From the earlier assumptions, one replica spends 0.16 s of exclusive accelerator time on prefill per request and 300 / 1,600 = 0.1875 s on decode (1,600 tokens/s is 32 streams × 50 tokens/s). That is 0.3475 s of accelerator time per request, so a replica sustains 1 / 0.3475 ≈ 2.88 requests/s for this request shape. This is a service capacity, not the chip's capability: shorter outputs or cached prefixes would raise it. As a consistency check, decode occupies 0.1875 / 0.3475 ≈ 54% of the time, so each stream actually advances at 50 × 0.54 ≈ 27 tokens/s under saturation (effective 37 ms per token), an average request lasts 0.215 + 299 × 0.037 ≈ 11.3 s, and by Little's law the number in flight is 2.88 × 11.3 ≈ 32, matching the batch we assumed.

Now traffic rises to 4 requests/s. Unlimited queueing gives a backlog growing by 4 − 2.88 = 1.12 requests per second, 67 requests after one minute, and the 67th waits about 67 / 2.88 ≈ 23 s before even starting. Every user is slow, and by the time old requests are served, many clients have timed out and gone away; the accelerator is then burning cycles on answers nobody receives. With a queue-wait budget of 2 s the queue should hold at most about 2 × 2.88 ≈ 5 requests. Beyond that, new requests are rejected immediately with a retryable status; 1.12 / 4 = 28% of the traffic. The remaining 72% see roughly 2 s of queueing at worst plus normal service time; the rejected 28% learn at once, and the client or gateway can fall back to another replica, a smaller model or a polite message. Fast, explicit failure for some users is better than slow failure for all of them.

Cancellation matters for the same reason. When a user closes the tab, the stream is worthless; continuing to decode wastes batch capacity and holds KV blocks. The gateway must notice the closed connection and tell the scheduler to abort the sequence. Priorities add one more control: interactive requests should be admitted ahead of batch jobs sharing the replica, and quotas should be defined in tokens as well as requests since one request with a 100,000-token prompt costs far more than ten small ones. How requests are spread across replicas is a routing question; see Model Serving.

There are two quite different reasons to add accelerators. To serve more requests, run more copies of the whole model behind a load balancer (replicas, or data parallelism for serving). Each replica handles complete requests independently, so capacity grows almost linearly and a replica failure loses only its in-flight work. Single-request latency does not improve at all.

To serve a model that does not fit on one accelerator, or to cut the per-step weight-reading time, you shard the model. In tensor parallelism each layer's weight matrices are split across devices; every device computes a slice and the slices are combined with a collective operation (all-reduce) a couple of times per layer. Pipeline parallelism instead places different layers on different devices and passes activations along. Parallel-processing mechanics are covered in Parallel Processing; here the question is the effect on serving.

Take an illustrative 70-billion-parameter model with 80 layers at 2 bytes per weight: 140 GB decimal = 130.4 GiB, which cannot fit on one 80 GiB device. Sharded across 2 devices each holds 65.2 GiB, leaving only 72 − 65.2 = 6.8 GiB per device for KV (using the same 90% cap): little room. Across 4 devices each holds 32.6 GiB and 39.4 GiB remains for KV, so more sequences fit. The weight-read floor per step also drops, since each device streams its own shard in parallel: 35 GB / 2 TB/s = 17.5 ms across 4 devices, versus 70 ms on one hypothetical large enough device. But there is a cost: with two all-reduces per layer, 80 layers need about 160 collectives per decode step. At an illustrative 15 µs each on a fast intra-node link, that adds 2.4 ms (floor 19.9 ms); at 60 µs each on a slower interconnect it adds 9.6 ms (floor 27.1 ms). The same arithmetic shows why tensor parallelism is generally kept inside one node with a fast interconnect, and why adding devices stops helping at some point as communication grows to dominate. The hardware side of that trade-off is discussed in AI Accelerators.

Change the sharding degree and the collective cost below to see the memory headroom and the step floor move together.

A practical rule: use the smallest sharding degree that fits the weights with a comfortable KV pool, then scale out by replicas. Four shards are one logical replica; eight devices might therefore be two replicas of 4-way tensor parallelism, not one replica of 8-way. A sharded replica also fails as a unit, since losing one device takes the whole group down, which affects how much spare capacity you keep.

Weight quantisation stores weights at fewer bits (8-bit or 4-bit instead of 16). Since decode is largely bandwidth-bound, reading half as many bytes per step can approach a 2× faster step, and the weights free memory for more KV. In our example, 8-bit weights would occupy 8 GB, a floor of 4 ms instead of 8 ms. Costs are possible quality loss (check on your own evaluation set, particularly for domain terms and non-English text), extra dequantisation work that may reduce the gain, and the need for hardware and kernels that support the format. Large batches that are compute-bound benefit much less. vLLM's README lists many supported quantisation formats; which ones suit your model and chip is something to establish by measurement.

KV-cache precision works on the other term: an 8-bit cache halves the 512 MiB figure to 256 MiB, doubling the sequences that fit and halving attention read traffic, again with a quality risk that grows with context length. It helps when memory or long-context attention reads limit you, and does little when the pool is already mostly empty, as in the 32-sequence example.

Optimised attention kernels fuse the attention steps so large intermediate matrices are not written to memory and re-read; they change how fast the same mathematics runs, not what it computes, and are generally a pure win when available for your hardware and shapes. They show up as engine and library versions, which is why upgrades can change latency without any change to your code.

Prefix reuse. The support assistant's prompts all begin with the same system prompt and policy preamble, say 1,200 of the 2,000 tokens. If the KV entries for that identical prefix are kept and shared, a new request prefills only the remaining 800 tokens: 2 × 8e9 × 800 = 1.28e13 FLOPs, or 64 ms instead of 160 ms, cutting TTFT by about 96 ms. The match must be exact and from the first token: put fixed instructions first and variable user content last, or nothing is shared. A prefix reused across tenants must not leak anything one tenant should not see, and cached state must be invalidated when the prompt template changes; that validity question belongs to Caching in AI Systems.

Speculative decoding addresses the sequential nature of decode. A small, cheap draft model proposes several next tokens; the large model then checks all of them in one forward pass (a verification step looks like a tiny prefill), accepting the longest prefix that agrees with what it would have produced. If three tokens are accepted on average per verification step, three tokens emerge for roughly the price of one large-model step plus the draft's small cost. Output quality is designed to remain that of the large model, but the gain depends on the acceptance rate: on unpredictable text, acceptance is low and the draft work is overhead. It helps most at low batch sizes, where the large model is bandwidth-bound and has spare compute; at high load the spare compute is gone. The vLLM README lists several speculative decoding variants, and the details differ by engine, so treat this as a concept to test rather than a guarantee.

The figure below makes the acceptance-rate dependence concrete under a simple independence assumption.

Vague terms hide problems, so define the metrics. TTFT (time to first token) runs from the moment the request arrives to the moment the first output token is delivered. Inter-token latency (ITL) is the gap between two consecutive tokens, and there is one per token pair. TPOT (time per output token) is the average over a request: (end-to-end time − TTFT) / (output tokens − 1). End-to-end latency is arrival to last token. Output tokens per second for a request is its inverse TPOT; throughput for the service is total output tokens (or requests) per second across all users. Specify whether you count input tokens too, and from which clock (client versus engine), because a figure measured at the engine excludes network and gateway time.

Take the textbook case: TTFT 400 ms, 100 output tokens at a steady 30 tokens/s. The first token arrives at 0.4 s. The remaining 99 tokens take 99 / 30 = 3.3 s, giving 0.4 + 3.3 = 3.7 s to the last token (not 0.4 + 100 / 30 = 3.73 s, which would count the first token twice). TPOT is 3.3 s / 99 = 33.3 ms, i.e. 30 tokens/s. The end-to-end rate including TTFT is 100 / 3.7 ≈ 27 tokens/s, lower than the streaming rate: this blended figure is the one to avoid quoting as “speed”.

This calculation assumes steady state: constant TPOT and no extra queueing. Real runs deviate in both ways. If the request had waited another 300 ms in the queue the same output would finish at 4.0 s with TPOT unchanged, because queueing only shifts TTFT. If the decode speed is uneven, say a large prefill joins the batch at token 50 and stalls everyone for 1.2 s, the mean TPOT rises by only 1.2 s / 99 ≈ 12 ms, yet users see a 1.2 s freeze: the mean hides it, and p99 ITL shows it. Hence the dashboards should track distributions: p50 and p95 TTFT, p95 and p99 ITL, and total completion time by output-length bucket.

Finally, separate peak throughput from latency-constrained throughput. A load test that pushes batch size until tokens/s is maximal reports a number you cannot use in production if p95 TTFT at that load is ten seconds. The number that sizes a fleet is the highest request rate at which all latency targets still hold. Full capacity modelling is the subject of AI Infrastructure Scaling and the monitoring side of AI Observability.

Self-hosting turns every request into accelerator-seconds. Using the running example, with an illustrative rental price of $4.00 per accelerator-hour (a made-up figure; substitute your actual cost), a replica at the 2.88 requests/s limit serves 2.88 × 3,600 ≈ 10,360 requests per hour, so the cost per request at 100% busy is $4.00 / 10,360 ≈ $0.000386, or $0.39 per thousand requests. Nobody runs at 100% around the clock. At 40% average utilisation the same hour yields about 4,144 requests and costs $0.00097 each: $0.97 per thousand. The price of a request is therefore a property of utilisation as much as of the model.

Input and output tokens are not equal. In this request, prefill takes 0.16 s for 2,000 input tokens (0.08 ms per token) while decode takes 0.1875 s for 300 output tokens (0.625 ms per token): an output token consumes about 7.8× the accelerator time of an input token, and the prompt, though 87% of the 2,300 tokens, accounts for 46% of the accelerator time. This is why hosted APIs commonly price input and output tokens differently; the exact ratio is a pricing decision and must be read from the provider's own price list. Prefix reuse removes 96 ms of the prefill in our example, taking 0.3475 s to 0.2515 s per request, which raises the replica limit from 2.88 to 3.98 requests/s; that is a capacity gain of 38% without buying anything.

If you call a hosted API instead, someone else owns this arithmetic and you pay per token, with rate limits replacing your own admission control. The comparison between paying per token and running a fleet depends on utilisation, traffic shape and engineering cost, and is treated as a cost question in Cost Optimization; this chapter only supplies the unit economics.

KV cache exhaustion. The pool holds 57.1 GiB; 16 requests with 32,768-token contexts (4 GiB each) would need 64 GiB. If the engine admitted them on prompt size alone and each then grows during decode, it runs out of blocks mid-generation. Well-designed engines react by preempting: pausing a sequence, freeing its blocks, and later either recomputing its prefill or restoring it from host memory; both cost latency. A poorly configured one fails the batch with an out-of-memory error. The preventative controls are the memory check in admission, a cap on prompt plus output length per request, and headroom for growth. The block pool is sized once at start-up, so a long-context tenant needs its limits set explicitly.

Oversized prompts. A prompt longer than the model's context window or the engine's configured maximum should be rejected immediately and clearly, not silently truncated: truncation removes the system prompt or policy text the model needed, and the answer may be wrong but fluent. A prompt that fits the window but is very long monopolises the accelerator in a single prefill; enforce per-request input limits and chunk the prefill.

Slow decode and head-of-line blocking. If p95 ITL climbs while GPU utilisation looks high, suspect an oversized batch, long contexts increasing attention reads, or large prefills being scheduled whole. If TTFT climbs while ITL is fine, the queue is deep: admission is not shedding early enough. The two signatures point at different parts of the scheduler.

GPU reset or worker crash. A device error or driver reset kills the worker and all KV state with it: every in-flight sequence on that replica is lost. The gateway sees broken streams. Requests that had not yet streamed any token can be retried on another replica transparently. Requests that had streamed some text cannot be invisibly replayed, because sampling is random and the retry would produce different words. The options are to show an error and mark the partial text as incomplete, or to retry with the partial output passed as a forced prefix so the model continues from where it stopped. Streaming APIs should state which semantics they offer, and clients should never treat a stream without a terminal event as a complete answer.

Retries. Client retries after a failure are the usual way a brief overload becomes a long one: each timeout creates a duplicate request while the engine is still working on the original. Retries must use bounded attempts with jittered backoff, carry a request identifier so the gateway can deduplicate or cancel the original, and be limited at the gateway by a retry budget. Because generation has no side effects by itself, retrying the model call is safe; retrying a request that triggered an order-lookup tool or any write is a separate matter, covered in Tools and Tool Calling.

Engines differ in scheduler, cache management, supported model architectures, quantisation formats, parallelism modes and hardware support, and all of these change between releases. Features cited in this chapter are a snapshot, not a specification, because they change between releases; for any engine, read the documentation of the exact version you intend to run, and rerun your evaluation on upgrade. Marketing benchmark numbers measure someone else's prompts on someone else's hardware and do not transfer.

A defensible evaluation builds a replay workload from production-shaped data: the real distribution of prompt lengths (including the long tail, since p99 lengths drive memory and ITL), output lengths, shared-prefix structure and arrival pattern, with the system prompt and retrieved passages your application actually produces. Using “2,000 in, 300 out” for every request would hide the cases that matter. Then, for each candidate engine and configuration:

1. 1

   Fix the quality baseline

   Run your offline evaluation set against the unoptimised configuration, then against every quantised or speculative variant. Reject any whose regression exceeds your tolerance before looking at speed.
2. 2

   Sweep the request rate

   Raise load in steps (open-loop arrivals, not a fixed number of looping clients), record p50/p95 TTFT, p95/p99 ITL and output tokens/s at each, and find the highest rate where all latency targets hold. This is the capacity number.
3. 3

   Measure memory headroom

   Record allocated pool, peak live KV blocks and preemption counts at that rate and with the longest permitted prompts. A configuration that passes at average length but preempts at p99 length is not ready.
4. 4

   Test failure behaviour

   Overload past capacity, kill a worker, disconnect clients mid-stream, send an oversized prompt. Confirm shedding, cancellation and error semantics match what the gateway expects.

The result is a per-replica capacity at a stated latency target for a stated workload, with cost per request from the token-economics arithmetic above. That one number, not a peak tokens/s headline, is what capacity planning consumes.

The following is illustrative TypeScript pseudocode, not any engine's implementation and not runnable as is. It shows where each decision from the previous sections sits in a single loop that executes once per iteration: handle cancellations, admit under budgets, run one forward pass mixing prefill and decode work, sample, stream, and release.

interface Seq {
  id: string; tenant: string;
  promptTokens: number[]; maxNewTokens: number;
  generated: number[]; prefilled: number;      // prompt tokens already in the KV cache
  blocks: number[]; deadlineMs: number;
  state: 'queued' | 'running' | 'done';
}

const MAX_ACTIVE = 32;              // bounds decode batch size, hence per-token latency
const TOKEN_BUDGET = 2048;          // prompt+decode tokens per iteration (bounds prefill stall)
const MAX_QUEUE_WAIT_MS = 2000;
const BLOCK = 16;                   // tokens per KV block

function schedulerLoop(engine: Engine, kv: BlockPool, queue: Seq[], running: Seq[]) {
  while (true) {
    // 1. cancellations and expiry: free memory first so admission sees it
    for (const s of [...running, ...queue]) {
      if (engine.cancelled(s.id) || (s.state === 'queued' && now() - s.arrivedAt > MAX_QUEUE_WAIT_MS)) {
        release(s, kv, running, queue);                   // returns blocks, emits "aborted"/429
      }
    }

    // 2. build this iteration's batch under a token budget
    let budget = TOKEN_BUDGET - running.length;           // each decoding seq needs 1 token
    for (const s of running) {
      if (needsNewBlock(s) && !kv.tryAlloc(s, 1)) {       // grow, or preempt youngest seq
        preemptYoungest(running, queue, kv);
      }
    }
    const work: WorkItem[] = running.map(s => ({ seq: s, tokens: [lastToken(s)] }));

    // 3. admit from the queue while memory, slots and budget allow
    while (queue.length && running.length < MAX_ACTIVE && budget > 0) {
      const s = queue[0];
      const chunk = Math.min(s.promptTokens.length - s.prefilled, budget);   // chunked prefill
      const blocksNeeded = Math.ceil((s.prefilled + chunk) / BLOCK) - s.blocks.length;
      const headroom = kv.free() - blocksNeeded - RESERVE_GROWTH_BLOCKS(running);
      if (headroom < 0) break;                            // backpressure: leave it queued
      kv.alloc(s, blocksNeeded);
      work.push({ seq: s, tokens: s.promptTokens.slice(s.prefilled, s.prefilled + chunk) });
      s.prefilled += chunk; budget -= chunk;
      if (s.prefilled === s.promptTokens.length) { queue.shift(); running.push(s); s.state = 'running'; }
    }

    // 4. one forward pass over the mixed batch, then sample for sequences whose prompt is complete
    const logits = engine.forward(work);
    for (const w of work) {
      if (w.seq.prefilled < w.seq.promptTokens.length) continue;
      const tok = engine.sample(logits.get(w.seq.id), w.seq);
      w.seq.generated.push(tok);
      engine.stream(w.seq.id, tok);
      if (tok === EOS || w.seq.generated.length >= w.seq.maxNewTokens) {
        release(w.seq, kv, running, queue);               // blocks back to the free list
      }
    }
  }
}

Read the loop against the failure analysis. Step 1 is why disconnects do not waste capacity. The memory check in step 3 (leaving growth blocks for running sequences) is what turns KV exhaustion into queueing instead of an out-of-memory crash, and breaking out of the admission loop is the backpressure that the overload example needs; the queue-expiry rule plus a bounded queue length at the gateway realise the 2 s wait budget. The token budget limits how much prefill rides along with a decode step, which is the lever for ITL versus TTFT. And a finished sequence releases its blocks in the same iteration, so freed capacity is available at the next token boundary, not the end of a batch.

An engine that produces tokens efficiently is still a process on a machine. Applications need a stable endpoint with authentication, a way to reach healthy replicas and avoid unhealthy ones, model versions that can be rolled out and rolled back, and capacity that follows traffic. None of those depend on how a KV block is allocated, and none of the engine's internals can supply them. The quantities defined here are what that layer takes as inputs: the per-replica request rate at a latency target, the TTFT and ITL distributions it must watch, the overload signals (queue depth, free KV blocks, preemptions) that should trigger routing away or scaling out, and the cancellation and partial-output semantics it must pass through to clients.

Retrieval-heavy workloads such as the support assistant also feed this machinery: the more passages the retrieval step injects, the longer the prefill and the larger the cache, a trade-off the context window and reranking lessons help you manage upstream, since sending 5 well-chosen passages instead of 20 is often the cheapest inference optimisation available.

1. 1

   TTFT is queue plus prefill (and network and gateway work) and grows with input length and load; TPOT or ITL is the per-token decode interval. With TTFT 400 ms and 100 tokens at 30 tokens/s, the last token arrives at 0.4 + 99/30 = 3.7 s, and the blended rate is about 27 tokens/s.

2. 2

   KV cache per token = 2 × layers × KV heads × head dimension × bytes per element. The illustrated model (32 layers, 8 KV heads, head dimension 128, 16-bit) needs 128 KiB per token, 512 MiB at 4,096 tokens; read the real dimensions from the model config.

3. 3

   Decode is mostly bandwidth-bound and prefill mostly compute-bound, and both share one accelerator; continuous batching shares each weight read among sequences, trading per-request latency for aggregate throughput.

4. 4

   Bound the engine explicitly: admission limits, a queue with a deadline, cancellation when the client disconnects, and preemption rules, because unlimited queueing turns a small rate excess into a growing backlog.

5. 5

   Use the smallest sharding degree that fits the weights with a comfortable KV pool and scale out by replicas; quantisation, optimised kernels, prefix reuse and speculative decoding each make a step cheaper only under their own conditions.

6. 6

   Report latency-constrained throughput rather than peak throughput, check the engine's documentation for your exact version, and treat vendor benchmark numbers as measurements of someone else's workload.