Model Serving

Watch a single request travel from a browser to a load-balanced fleet of model servers — then see what happens when traffic spikes, and when a server fails outright. ChatGPT is only a familiar entry point: the fleet shown is a generic, illustrative serving architecture, not a description of OpenAI's actual systems.

Model serving, and LLM serving in particular, is the layer that turns a trained artifact into a network endpoint other software can depend on. The inference engine is the program that runs the model on an accelerator; the production endpoint is everything around it that makes it deployable and reliable. A serving system authenticates and validates each request, routes it to the right model version and region, admits it into a bounded queue, runs it on a replica that has the weights loaded and warmed, streams the result back, and reports what happened. Around that data path sits a control plane that registers artifacts, rolls out versions gradually, scales replicas on signals that reflect real work, and replaces replicas that fail. Operating an endpoint is mostly the discipline of keeping that machinery honest: never sending traffic to a replica that cannot serve it, never letting overload turn into unbounded latency, and always being able to say which version answered which request.

1. 1

   Package

   Version the weights, tokenizer and runtime together as one deployable artifact.
2. 2

   Deploy and warm

   Start replicas, load weights and pass a readiness check before they receive traffic.
3. 3

   Route

   Authenticate, then send each request to the right model version, pool and replica.
4. 4

   Scale

   Add or remove replicas on signals that reflect real work, such as queue depth and token load.
5. 5

   Roll out

   Shift traffic to a new version gradually with a canary, and keep the ability to roll back.
6. 6

   Observe and recover

   Tag every request with the serving version, watch latency and errors, and replace failing replicas.

How the model runs inside one replica (prefill, decode, batching, KV memory) is the subject of Inference Infrastructure. This chapter starts where that one stops: with a working engine and the question of how to put it behind an endpoint that thousands of users, several tenants and a release process can share.

Imagine the enterprise support assistant from earlier chapters: it answers policy questions, retrieves documents the caller is allowed to see, and can call an order-lookup tool. The team has a self-hosted model and a script that loads the weights into one GPU process and answers a prompt in two seconds. Demo day goes well. Then the first real week arrives, and these are the things the script has no answer for.

• Start-up is slow and stateful. Loading weights, initialising the runtime and compiling or capturing kernels can take minutes (illustrative: 3 minutes). A process that is "up" for the load balancer while still loading serves errors to every user routed to it.
• Requests are long and uneven. One caller asks for a 40-token answer, another for 900. A single in-flight request can occupy accelerator memory for tens of seconds, so "requests per second" is not a stable unit of load.
• Overload has no graceful form. A script accepts connections until memory runs out. With no admission limit, the 200th concurrent request does not wait politely; it pushes the process into an out-of-memory crash that kills the 199 requests already in progress.
• Updates are risky. Replacing the model means either downtime or a window where nobody knows which version answered, and no way back when the new weights turn out to hallucinate more.
• Nobody else can tell it is sick. A GPU that has silently stopped making decode progress still holds a TCP socket open.
• Every tenant shares the same blast radius. One customer's batch job starves everyone, and the script has no notion of who is calling.

Loading weights into memory is therefore the smallest part of operating an endpoint. Everything else (how requests are admitted, where they go, how replicas join and leave, how versions change) is the serving system. The same holds when the model is hosted by a provider: you do not run GPUs, but you still run a gateway, timeouts, quotas, retries, version pinning and a rollback story around an API you do not control. A large share of this chapter applies to both; where it applies only to a self-hosted fleet, the text says so.

Follow one request in the support assistant. The application sends the user's question, retrieved policy passages and, if needed, an order-lookup result to the model endpoint (the application side of that is covered in Context Injection and Tools and Tool Calling; here the model endpoint is treated as a black box that accepts a prompt). Figure 1 shows the components the request touches and, below them, the control-plane parts that never see a user prompt but decide what the data path looks like.

Each box exists because of a specific failure it prevents.

Gateway: identity and the edge of trust

The API gateway terminates TLS, authenticates the caller (an API key or a signed token mapped to a tenant), checks the request against a schema, and enforces coarse limits. Rejecting a malformed request here, at a cost of microseconds, is what keeps it from consuming GPU time. It also assigns the request ID and trace context that every later component attaches to its logs.

Router: which model, which version, which pool

The caller asks for a logical model such as support-assistant. The router resolves that to a concrete deployed version and a pool of replicas, using the tenant, region, rollout weights and current capacity. This indirection is what lets you change versions without changing any client. The decisions it makes are covered in the routing section below.

Admission queue: where overload becomes a decision

Between the router and the replicas sits a bounded queue with deadlines. Its job is to turn "too much traffic" from a crash into an explicit policy: wait up to a limit, then fail fast with a retryable status. The queue can live in the router, in a per-replica scheduler, or both; what matters is that it is bounded and that requests whose deadline has already passed are dropped instead of executed. A request that finishes after its caller gave up is pure waste.

Replicas: warm, stateful workers

A replica holds weights on its accelerator, an inference engine with its own scheduler, and the KV cache for sequences in flight. It is stateful in exactly the way a stateless web service is not: killing it loses the in-flight decode state of every active request, and bringing a new one up takes minutes, not milliseconds. Almost every operational rule below follows from those two facts.

Streaming transport

Responses are produced token by token, so the path back to the client is a long-lived connection, typically server-sent events or a similar framing, held open through the gateway for seconds. That makes the gateway part of the failure surface: its idle timeouts, buffering and connection limits apply to the stream. The streaming section covers this in detail.

Control plane

The registry stores versioned artifacts with manifests. The deployment controller (a Kubernetes Deployment and Horizontal Pod Autoscaler, a managed endpoint service, or your own scripts) creates, replaces and scales replicas. Metrics and traces flow back from the data path to it. The separation matters because the two planes fail differently: if the control plane is down, existing replicas keep serving at their current size; if the data plane is down, no control-plane cleverness helps.

An endpoint does not serve "a model"; it serves a specific combination of things that must agree. Weights trained against one tokenizer produce nonsense with another, and nothing crashes when they disagree: the engine happily maps the wrong integers to the wrong embeddings and returns fluent garbage. The packaging discipline is to make that combination one immutable, versioned, verifiable unit.

{
  "name": "support-assistant",
  "version": "2.0.0",
  "weights": { "uri": "registry://models/support-assistant/2.0.0/weights", "sha256": "<64 hex chars>", "format": "safetensors", "dtype": "bf16", "size_gb": 16 },
  "tokenizer": { "sha256": "<64 hex chars>", "vocab_size": 128000, "chat_template_sha256": "<64 hex chars>" },
  "limits": { "max_context_tokens": 8192, "max_output_tokens": 1024 },
  "runtime": { "image": "registry.example.com/llm-runtime@sha256:<digest>", "engine_args": ["--max-concurrent-sequences=24"] },
  "device": { "accelerators": 1, "min_memory_gb": 40 },
  "warmup": { "prompts": ["short", "medium", "near-max-context"], "golden_prompt_id": "g-017" },
  "eval": { "suite": "policy-qa-v7", "passed_at": "2026-10-20T09:14:00Z" }
}

Several fields deserve a sentence of justification, because each encodes a real failure.

• Tokenizer and chat template. The tokenizer defines the integer vocabulary; the chat template defines how roles and special tokens wrap a conversation. Both must travel with the weights. A subtle version of the bug is a silently changed template: the model still answers but follows system-prompt instructions less reliably, and only an evaluation set notices. Check at load time that the tokenizer's vocabulary size matches the model's embedding rows.
• Runtime image and engine arguments. The same weights can behave differently, or not fit in memory, under a different engine build, precision setting or concurrency limit. Treat the image digest and the arguments as part of the version.
• Device placement. The manifest states how many accelerators a replica needs and how much memory the weights plus overhead plus KV cache require. The scheduler uses that to decide where a replica can run; if a model needs two GPUs for tensor parallelism, placement must put them in the same fast domain (see AI Accelerators).
• Integrity. Weights should be fetched by content hash and verified before loading; the security section returns to this.

From scheduled to ready: the start-up sequence

A replica becomes useful only after a fixed sequence, and each step has its own failure signature. Pull the runtime image and the weights; load weights into accelerator memory; allocate the KV cache pool; run warmup requests; pass a self-test; only then declare readiness. Illustrative timing for the 16 GB artifact from fast local storage at an effective 1 GB/s (decimal): 16 GB ÷ 1 GB/s = 16 s of weight transfer. In the same illustrative run, scheduling and image pull take 60 s, and warmup (kernel compilation, capturing execution graphs where the engine does that, exercising short, medium and near-limit prompt lengths) takes 74 s, so a cold replica needs about 150 s from the scaling decision to its first useful request. Warmup matters because the first request of each shape can be much slower than the steady state, and you want real users to pay for none of it.

The self-test is cheap insurance: send one fixed "golden" prompt with deterministic decoding and compare a checksum or a few expected tokens. It catches a class of bugs, such as a wrong tokenizer, corrupted weights or a broken precision setting, that load without complaint and fail only in content.

Graceful drain

Leaving is as important as arriving. When a replica is told to stop (a scale-down, a rollout or a node maintenance event), it should first stop receiving new requests, then finish the sequences it holds, then exit. The longest legitimate sequence sets the drain budget. With an illustrative maximum of 1,024 output tokens at 30 tokens/s, a streamed answer can take 1,024 ÷ 30 ≈ 34 s of decode plus queueing and prefill, so a termination grace period of 90 s leaves room. Setting it to the Kubernetes default of 30 s would cut off long answers mid-sentence on every rollout. A running generation cannot practically be handed to another replica mid-decode, because its KV state lives in this replica's accelerator memory; "migrating" an in-flight request means finishing it where it started, or failing it and retrying from the start under the retry rules covered later.

A web load balancer can send a request to any healthy instance, because instances are interchangeable. LLM replicas are not interchangeable, in four ways, and routing is where those differences are handled.

Identity of the model. Requests name a logical model; replicas serve a specific version. The router holds the mapping, so promoting a new version is a router change and not a client release. The routing key must also respect compatibility: a request that uses a feature only v2 supports (say, a longer context limit) must not be sent to v1.

Tenant and policy. Some tenants need dedicated replicas (contractual isolation, residency in a specific region), others share a pool. The router enforces that mapping, and it is the right place to apply per-tenant concurrency caps so that one customer's batch job cannot occupy every sequence slot. Isolation here is a scheduling guarantee, not a security boundary by itself; see the security section.

Capacity. Round-robin is a poor policy for requests whose cost differs by an order of magnitude. A replica holding twenty long generations is not equivalent to one holding two. Good routers pick by reported load (active sequences, queue depth, free KV capacity), using the metrics replicas expose, with a small amount of randomness ("choose the less loaded of two random candidates") to avoid every router stampeding to the same momentarily idle replica.

Locality of cached state. This is the difference that surprises people. If replicas keep a prefix cache of the KV state for shared prompt prefixes (see Inference Infrastructure for how it works and Caching in AI Systems for when it is valid), then a hit is only possible on the replica that holds it. Our assistant has a roughly 3,000-token shared prefix (system prompt plus tenant policy preamble). The router can improve the hit rate by hashing the prefix key, composed of the model version, the tenant and a hash of the shared prefix, to a preferred replica.

A small calculation shows when that is worth it. Suppose a tenant sends 3 requests per minute (Poisson arrivals, illustrative), and a replica keeps that prefix resident for about 60 s before eviction pressure removes it. With 6 replicas and load-blind routing, a given replica sees this tenant at 3 ÷ 6 = 0.5 requests per minute, so the probability that the replica received a request within the last 60 s is 1 − e^−0.5 ≈ 0.39. With affinity, all 3 requests per minute reach one replica: 1 − e⁻³ ≈ 0.95. A hit saves the prefill compute for those 3,000 tokens; whether that is worth chasing depends on how much of your latency is prefill. Affinity has a price: it concentrates load. A popular tenant can saturate its preferred replica while others idle, so affinity must be a preference bounded by load: use the preferred replica unless its queue exceeds a threshold, then spill to the next candidate and accept a cold prefill. A policy that maximises cache hits and ignores load will lose to round-robin the first time one tenant is popular.

Affinity also interacts with versions and failures. Cache entries are valid only for the exact model version and tokenizer; the key must include them, and a canary should not reuse v1's cache. When the preferred replica dies or drains, its cache dies with it, so the first requests after a rollout are slower than steady state; that is a reason to roll slowly rather than to avoid rollouts.

Failover and isolation. A standby pool in another region can take traffic when the primary is unhealthy or full, but failover is only legitimate if it respects the constraints that motivated the primary choice: a tenant whose data must stay in region A must receive an error (or queue), not a silent cross-region reroute. Make those constraints explicit fields in the routing table, not implicit properties of where the capacity happens to be.

route: support-assistant
  versions:
    - id: v1   weight: 95   pool: gpu-pool-a-v1
    - id: v2   weight: 5    pool: gpu-pool-a-v2     # canary
  bucket_key: [tenant_id, conversation_id]          # stable per conversation
  affinity:
    key: [model_version, tenant_id, prefix_hash]
    spill_when_queue_over: 6                        # requests waiting on preferred replica
  constraints:
    - tenant: globex   region: eu-only   failover: none
    - tenant: default  failover: pool gpu-pool-b-v1
  limits:
    per_tenant_max_active_sequences: 8

A health check answers a question for a specific consumer, and different consumers ask different questions. Orchestrators distinguish at least three, and for model servers confusing them has concrete costs. In Kubernetes terms, a liveness probe decides whether to restart the container, a readiness probe decides whether the pod receives traffic, and a startup probe holds off the other two until a slow start finishes (the startup probe exists precisely for slow-starting containers like ours).

What each check should and should not look at

Liveness answers "is this process wedged beyond self-repair?" The cost of a false positive is a restart, and a restart for us means another 150 s of cold start and the loss of every in-flight request. So liveness should be conservative and depend only on the process itself: is the engine's main loop turning and making progress? It must not call the database, the retrieval service or the model itself on every probe. A liveness probe that fails because a downstream dependency is slow converts a dependency incident into a fleet-wide restart storm, the opposite of what you want.

Readiness answers "should the router send new work here now?" For a model server this is true only when the weights are loaded, warmup is finished and the golden self-test passed. This is the check that prevents the most common outage in the lesson: routing real users to a replica that is "running" but still loading. Because readiness is evaluated repeatedly, it is also the check that removes a replica from rotation when something goes wrong, so be careful what you let flip it.

Overload should usually not fail readiness. It is tempting to report "not ready" when the queue is full. Consider what happens: the replica leaves the endpoint list, its share of traffic moves to the others, which become overloaded and also report not ready, and the whole pool empties out at the moment demand is highest. A better design keeps the replica ready, rejects requests above its queue limit quickly with a retryable status and a Retry-After hint, and publishes its load so the router can steer around it. The router, not the orchestrator's health check, makes load-aware decisions.

GPU health is not covered by an HTTP 200. Accelerators fail in ways that leave the web server happy: a driver fault, uncorrectable memory errors, a thermal throttle that cuts throughput by half, or a kernel that hangs. Two practical signals are worth wiring in. First, a progress watchdog inside the engine: if sequences are active but no decode step has completed for, say, 10 s, mark the replica unhealthy. Second, the accelerator's own health telemetry (error counters from the vendor's management library) exported as a metric and used by the node-level repair process. A hung GPU is usually not fixed by restarting the container; the node itself may need to be drained and replaced, which is why you want the signal distinguished from ordinary liveness failures.

Startup probe for the long tail. With a 150 s typical start and occasional 5-minute starts when storage is slow, the liveness probe would kill a replica that is merely slow to load. A startup probe with generous limits gives the load phase a separate budget; Kubernetes holds the other probes off until it succeeds. Illustrative: period 10 s × failure threshold 60 = 600 s allowed before the replica is considered failed.

Autoscaling adds replicas when demand exceeds capacity and removes them when it does not. The hard parts for a model endpoint are that capacity is measured in concurrent sequences and memory rather than CPU cycles, and that a new replica takes minutes to help. Choosing the signal and accounting for that delay is most of the work.

Why CPU% and GPU utilisation mislead

CPU utilisation on a GPU server mostly reflects the Python and networking work around the engine; it can sit at 20% while the accelerator is saturated, and rises with request parsing and streaming overhead rather than generation work. GPU utilisation as typically reported (commonly a time-based measure: the fraction of a sampling period in which at least one kernel was running; check the definition your monitoring tool uses) is also a poor guide: a replica can show 95% because decode kernels are always executing while it is serving 3 sequences or 24, and the number does not tell you how close the replica is to its sequence or memory limit. It is a measure of busy time, not of remaining capacity. A replica can be "100% utilised" with room for twice the load, or at 60% with its KV cache full and about to refuse requests.

Signals that track capacity directly include the following.

• Waiting plus active sequences per replica. This is demand relative to the admission limit. Active sequences alone saturate at the limit, so they cannot tell you how far above capacity you are. Adding the queue restores the information.
• Queue depth and queue wait time. Direct evidence that arrivals exceed service. Wait time is the better of the two because it relates to user experience; depth is easier to act on quickly.
• KV cache memory pressure. When the pool is nearly full, new sequences wait or running ones are preempted regardless of what compute shows (details in Inference Infrastructure).
• Token throughput against the replica's measured ceiling. Useful for capacity planning; as a scaling trigger it lags because it plateaus once saturated.
• TTFT and p95 latency. The symptoms users feel. They are good alarms and safety nets, but as the primary scaling signal they react after the damage has begun, and they are noisy at low volume (a p95 over a handful of requests is a poor statistic).

A worked autoscaling calculation

All figures illustrative. Each replica runs with an admission limit of 24 concurrent sequences, and we target 17 (about 70%) for headroom. Mean request duration, queue plus generation, is 6 s. Steady traffic is 12 requests/s. By Little's Law, concurrency L = λ × W = 12 × 6 = 72 sequences, so we need ⌈72 ÷ 17⌉ = ⌈4.24⌉ = 5 replicas. Their hard capacity is 5 × 24 = 120 concurrent, or 120 ÷ 6 = 20 requests/s at the same duration.

Now traffic jumps to 30 requests/s (a product announcement, say). Demand is 30 × 6 = 180 sequences, which needs ⌈180 ÷ 17⌉ = ⌈10.59⌉ = 11 replicas. The Kubernetes Horizontal Pod Autoscaler's documented rule is desiredReplicas = ceil(currentReplicas × currentMetricValue ÷ desiredMetricValue); with a metric of (active + waiting) sequences per replica, the current value is 180 ÷ 5 = 36 against a target of 17, giving ⌈5 × 36 ÷ 17⌉ = ⌈10.59⌉ = 11. Had we fed it active sequences only, each replica would report at most 24, and the result would be ⌈5 × 24 ÷ 17⌉ = ⌈7.06⌉ = 8, an undershoot caused by the signal hiding the queue.

Move the traffic slider below to watch the two metrics diverge once the queue forms.

Read the figure for what it says about design. For 195 s the system receives 10 req/s more than it can serve. If nothing limits the queue, 1,950 requests pile up and the newest wait about 1,950 ÷ 20 ≈ 97 s even at the old capacity, so every one of them has probably timed out client-side and will still consume GPU time. With a queue limit of 100 requests (a 5 s wait at 20 req/s) and expired-deadline dropping, roughly 1,850 of the 5,850 arrivals in those 195 s are rejected quickly (about 32%). That is a bad few minutes, but a bounded, visible and recoverable one, and users who are served are served at normal latency. Autoscaling cannot remove this window, because it is made of start-up time; only headroom, predictive or scheduled scaling, a pool of pre-warmed standby replicas, or a smaller, faster-starting model can shorten it, and each costs money.

Scale-down is the opposite hazard. Removing replicas as soon as the metric dips and re-adding them minutes later pays the 150 s cold start each time and drops KV and prefix caches. The Kubernetes HPA addresses this with a stabilisation window in its behavior field: for scale-down it uses the highest desired value over the window (the documentation example uses stabilizationWindowSeconds: 300), which behaves like a rolling maximum. For slow-starting replicas, long scale-down windows are cheap insurance. Scale-down also has to follow the drain rules above, or each reduction cuts off answers in progress.

When the metric the autoscaler needs is not a built-in resource metric, it comes through the Kubernetes custom or external metrics APIs, which require an adapter that serves them; a managed endpoint service may offer an equivalent hook. The blueprint section shows how this is written.

The team has fine-tuned the assistant for better policy answers. Version 2 passed the offline evaluation suite and was packaged with a new manifest. Offline tests are necessary and insufficient: they use a fixed set of questions, while production brings the real prompt-length mix, retrieval results, tenant quirks and load. So v2 meets production traffic gradually, with a rule for turning back. All numbers below are illustrative.

Tag every request with the version that served it

A rollout you cannot measure is a coin flip. The router stamps each request with the version it chose; the replica echoes the actual served version in the response and in every log line, metric label and trace span. Comparing v1 and v2 then becomes a group-by, and a user complaint ("the answer was wrong yesterday at 14:02") can be traced to a version.

→ POST /v1/generate
  X-Request-Id: req_8f3a21
  traceparent: 00-<trace-id>-<span-id>-01

← 200 OK
  X-Request-Id: req_8f3a21
  X-Model-Version: support-assistant@2.0.0
  X-Rollout-Arm: canary

{"ts":"2026-10-28T09:41:12Z","request_id":"req_8f3a21","tenant":"acme","model":"support-assistant",
 "version":"2.0.0","arm":"canary","input_tokens":1840,"output_tokens":212,"ttft_ms":410,"status":"ok"}

Choosing who goes to the canary

Random per-request assignment is simple but splits a conversation across versions, so turn 2 might be answered by a different model than turn 1, with different style and sometimes contradicting earlier turns. Assign by a stable hash of the conversation (and tenant), so a conversation stays on one arm, and raising the percentage only adds conversations to v2 without ever moving a v2 conversation back to v1. The following code is runnable Python (standard library only); the assertions check determinism and monotonicity.

import hashlib
from collections import Counter

def bucket(tenant_id: str, conversation_id: str, salt: str = "rollout-2026-10") -> float:
    """Stable value in [0, 1) for a conversation, so retries and later turns agree."""
    digest = hashlib.sha256(f"{salt}:{tenant_id}:{conversation_id}".encode()).digest()
    return int.from_bytes(digest[:8], "big") / 2**64

def pick_version(tenant_id, conversation_id, canary_pct, allow_tenants=None):
    if allow_tenants is not None and tenant_id not in allow_tenants:
        return "v1"
    return "v2" if bucket(tenant_id, conversation_id) < canary_pct / 100 else "v1"

counts = Counter(pick_version("acme", f"conv-{i}", 5) for i in range(100_000))
print(counts["v2"] / 100_000)          # close to 0.05 (about 0.049 on one run)
assert pick_version("acme", "conv-1", 5) == pick_version("acme", "conv-1", 5)
assert all(
    pick_version("acme", f"c{i}", 5) == "v1" or pick_version("acme", f"c{i}", 25) == "v2"
    for i in range(5000)
)  # raising the percentage never moves a conversation from v2 back to v1

What the gate compares, and how long it needs

The gate compares v2 with v1 over the same period, on the same traffic mix, on four families of signal: errors (non-2xx rate, timeouts, cancelled-by-server), latency (TTFT and p95 end-to-end; compare p95 with p95, not with the mean), cost (output tokens per answer; a new model that is twice as chatty doubles decode cost), and quality. Quality is the hard one, since there is no HTTP status for "the answer was subtly wrong". Practical proxies are a sampled LLM or human judgement against a rubric, citation-validity checks for answers that cite retrieved policies, tool-call success for the order-lookup path, and user signals such as thumbs-down and escalation to a human agent.

Statistics set the minimum hold time. At an illustrative 40 requests/s total, 5% is 2 requests/s: 1,200 canary requests in 10 minutes. If v1's error rate is 1%, we expect 12 failures from v1-like behaviour (standard deviation √12 ≈ 3.5); if v2 truly fails at 3% we expect 36, which is obvious, so a gross regression shows up in minutes. Quality is slower. Suppose 20% of canary responses are judged (240 in 10 minutes) and the true quality rate is about 0.90. The standard error on one arm is √(0.9 × 0.1 ÷ 240) ≈ 0.019. For the difference between two arms of 240 judged responses each (v1 sampled equally), the 95% interval is about ±5.9 percentage points, far too wide to detect a real 5-point drop. To bring the interval half-width down to 5 points needs about 1.96² × (0.9 × 0.1 + 0.85 × 0.15) ÷ 0.05² ≈ 335 judged responses per arm, which at 0.4 judged canary requests/s is about 14 minutes. Hence the 20-minute hold in Figure 5. It is also why a canary at 0.5% would be nearly useless for quality at this traffic. The sample size, not the calendar, is the real gate. Statistical significance is not enough by itself, either: a tenant with unusual documents might see a regression the pooled number hides, so also slice by tenant tier and by the retrieval path.

1. 1

   Gate passes

   Error rate within +0.5 points of v1, p95 latency within +10%, judged quality not lower than v1 by more than the pre-agreed margin at the required sample size, cost per answer within +15%. Advance to the next weight.
2. 2

   Gate fails

   Set the v2 weight to 0 in the routing table. New requests go to v1 within the router's config propagation time. In-flight v2 requests finish on v2 (see below). Keep the v2 replicas for debugging, then scale them down.
3. 3

   Automated vs human decision

   Hard safety conditions (error spike, p95 over the SLO) can trigger automatic rollback. Quality and cost judgements usually deserve a human reading the evidence, because the metrics are noisy and the cost of a false rollback is small.

In-flight requests and multi-turn conversations during the change

A routing change affects only requests that have not started. A request already decoding on a v2 replica finishes on v2: the version was chosen at admission and the KV state lives in that replica. Weight changes, scale-downs and rollbacks therefore need the drain behaviour from the packaging section, or a rollback would cut off exactly the requests you were protecting. For conversations, decide explicitly whether a conversation is pinned to the version that started it (consistent style, but requires keeping v1 around longer and complicates retiring it) or may move at a turn boundary (simpler operations; the new version simply re-reads the conversation history as input, and the prefix cache is cold). Either is defensible; what is not defensible is leaving it undefined. Two version-specific hazards deserve a check: stored conversation state (summaries or tool-call formats written by v2 that v1 cannot parse after a rollback) and the prefix cache, whose key must contain the version.

Availability is the promise that capacity survives failure. Each failure scale needs its own mechanism, and each mechanism costs spare capacity that sits mostly idle by design.

A replica fails. It crashes, its GPU faults, or its node is reclaimed. In-flight sequences on it are lost and must be retried by the caller (rules below). The router stops sending it new work as soon as it sees connection failures or a failed readiness check; the orchestrator starts a replacement, which then needs about 150 s of cold start. Until then, the remaining replicas carry its load, which requires headroom: with 10 replicas at 70% target utilisation, losing one leaves 9 serving the same demand at 78% (0.7 × 10 ÷ 9), still below the hard limit. If you run at 95%, the loss of one replica tips the pool into overload.

A zone fails. Spread replicas across failure domains (in Kubernetes, topology spread constraints on the zone label) so that one zone is never all of your capacity. Suppose steady state needs 10 replicas' worth of capacity and you want to survive losing a whole zone out of three. The surviving two zones must hold 10, so each zone hosts 5 and the total is 15: spare capacity is 50%. With two zones the same requirement needs 10 in each (total 20, 100% spare). More zones cost less per unit of resilience; this is arithmetic, not opinion. At an illustrative $2.50 per GPU-hour, the 5 spare GPUs cost 5 × 2.5 × 24 = $300 per day; whether that is worth paying depends on what an hour of outage costs the business. Many teams decide that some tenants get zone-failure capacity and others get best-effort. Spare capacity can be made useful in the meantime by running lower-priority batch work (evaluation, offline summarisation) that is preempted when the zone event occurs, if your scheduler supports priority preemption.

The figure below varies the number of zones, rounding up to whole replicas.

A region fails. Failover to another region needs capacity there, which is reserved, warm or both. Cold regional failover means a 150 s start per replica plus the time to notice, so a real regional failover plan keeps a minimum warm fleet in the second region and capacity reserved to scale up quickly, because in a regional outage everyone else is also trying to acquire scarce accelerators. Failover also has to respect residency constraints from the routing table. The deeper multi-region capacity model belongs to AI Infrastructure Scaling.

Maintenance and rollouts. Replacing replicas is a self-inflicted failure. A rolling update withmaxUnavailable: 0 and maxSurge: 1 adds a new ready replica before removing an old one, at the cost of one extra GPU during the rollout; a PodDisruptionBudget prevents voluntary node drains from taking too many replicas at once.

Overload shedding. Sometimes demand simply exceeds all capacity, including spare. The goal is to fail cheaply and deliberately. Reject at the edge when the queue is full or the estimated wait exceeds the caller's deadline, with a 429 or 503 and a Retry-After value; prioritise (interactive chat above background summarisation, paying tiers above free); and reduce work per request by capping output tokens or switching to a smaller fallback model when permitted. Retries are the sharp edge here: clients that retry immediately turn a 20% overload into a 60% overload. Retries need exponential backoff with jitter, a budget (for example at most 10% extra traffic from retries), and should not be performed at every layer: a gateway retrying, behind an SDK retrying, behind a user pressing Refresh, multiplies.

A non-streaming call has two outcomes: a complete response or an error. A streamed call has many more, because the failure can occur after the user has already seen half an answer. Streaming exists because decode is slow (interactive latency is dominated by when the first token appears, not when the last does, as Inference Infrastructure explains), so the endpoint must define what a half-finished response means.

Framing and the completion contract

Server-sent events (SSE) is a common transport: an HTTP response with content type text/event-stream, in which each event is one or more data: lines followed by a blank line, optionally with anevent: name. The essential design rule is that a stream is complete only if it says so. If the last event received is a token delta, the client cannot know whether the answer is finished or the connection died, and a truncated policy answer ("refunds are allowed within") is worse than an error. Define a terminal event that carries the finish reason and usage, and have clients treat a stream that ends without it as incomplete.

event: delta
data: {"index":0,"text":"Refunds"}

event: delta
data: {"index":1,"text":" for duplicate charges"}

event: done
data: {"finish_reason":"stop","model_version":"support-assistant@2.0.0","usage":{"input_tokens":1840,"output_tokens":212}}

# If the replica fails after some deltas, a well-behaved gateway ends with:
event: error
data: {"code":"replica_failed","retryable":true,"partial":true}

Timeouts that mean different things

A single request timeout is the wrong abstraction for a stream. Three deadlines are needed. A time-to-first-token timeout (illustrative 10 s) covers queueing and prefill: if no token appears, the request is probably stuck or the replica is overloaded, and retrying elsewhere is still safe because the client has seen nothing. An inter-token idle timeout (for example 15 s) detects a stalled replica mid-answer. An overall deadline (60 s) bounds total resources. Proxies and load balancers add their own idle timeouts, often tuned for ordinary web responses, and they may buffer responses unless told not to; test that the stream actually arrives incrementally through every hop, and keep-alive comments or small heartbeat events can prevent idle cut-offs during a long silent prefill.

Cancellation and what a retry is allowed to do

Cancellation has to propagate through the layers: the client closes the connection, the gateway notices and calls cancel on the replica (or the replica notices the closed socket), and the engine aborts the sequence and releases its KV blocks. A user who closes a tab 2 s into a 200-token answer should free that sequence slot within a second, not after another 6 s of decode. The same applies to a timeout: the gateway must cancel what it abandons.

Retries follow the retry boundary: the point at which output has been delivered to the client. Before the first token is sent, the gateway can retry on another replica transparently; the user sees only extra latency. After the first token, the gateway cannot silently restart, because the replacement would generate a different answer from the one already on screen (sampling is not deterministic across replicas, and even greedy decoding may differ with batch composition). The honest options are to end the stream with a retryable error and let the client decide (discard and regenerate, or continue from the partial text via a new request that includes it), and to record the outcome as a partial completion. Whichever you choose, document it; the client-visible contract is that a stream ends with done, or with error, or it is incomplete.

Tool use adds a stricter boundary. If the model decided mid-stream to call the order-lookup tool, retrying the generation might call it twice. Read-only lookups are harmless; a refund-issuing call must carry an idempotency key so a replay does not execute twice (the agents track covers this in Tools and Tool Calling). Serving's part is to preserve the request ID through retries so the key remains stable.

A complete worked latency: with an illustrative TTFT of 400 ms and 100 output tokens at 30 tokens/s, the first token arrives at 0.4 s and the last at about 0.4 + 99 ÷ 30 = 3.7 s. That is the simplified steady-state figure; queueing before prefill and uneven decode speed under batching both stretch it, which is why the stream deadline should be set from measured p95, not from this arithmetic.

The support assistant soon needs more than one model: the main generator, a small model for classification and routing of questions, an embedding model for retrieval, perhaps a reranker. The question is how to place them on accelerators, and the choices trade utilisation against isolation.

Dedicated replicas give each model its own accelerators. Performance is predictable, a noisy model cannot steal memory from another, and each model scales on its own signals. The cost is stranded capacity: a model at 10% load still occupies a whole GPU.

Shared replicas put several models on one device, either co-resident in memory or swapped in and out on demand. Utilisation improves for lightly used models. The costs are real. Co-resident models compete for memory bandwidth and for the same KV budget, so a spike on one inflates latency on the other. Swapping weights in and out brings back the load time (16 GB at 1 GB/s is 16 s) as a latency spike for the first request after a swap, and it is only reasonable for rarely used models whose users tolerate a cold start. Shared placement also weakens isolation: an out-of-memory failure in one process can take down the others on the device unless memory limits are strictly enforced.

Fragmentation is the subtler problem. Memory freed on several devices does not combine into memory usable on one. Figure 7 shows three 80 GB GPUs (illustrative) with a 6 GB runtime overhead each, hosting models with 45, 30 and 50 GB of weights. The free blocks are 29, 44 and 24 GB: 97 GB in total. A fourth model with 38 GB of weights needs 38 + 6 overhead + 10 for a minimal KV pool = 54 GB on a single device. It does not fit, although 97 GB is free in aggregate. Spreading it across GPUs with tensor parallelism needs fast interconnect and co-scheduled placement, and consumes interconnect and reserved memory on all of them; it is a design decision, not a free workaround.

The remedies are the usual ones for bin packing: place large models first, keep a few model sizes standard so blocks are interchangeable, reserve whole devices for models above a size threshold, and defragment by draining and rescheduling during low traffic. The router carries the other half: it must know which models each replica holds, route accordingly, and fail a request for an unloaded model quickly instead of letting it queue behind a loading operation. Admit loads and unloads deliberately (rate limited, never during peak), because a loading replica is a warming replica and should not be in rotation.

Small models deserve a note: embedding and reranking models are often light enough to run on CPU or on a small share of a device, and the right place for them may be a separate pool with its own autoscaling (their latency profile is a single forward pass, not a decode loop), which is closer to ordinary stateless serving than the generator is.

The endpoint processes confidential prompts, spends expensive compute and loads executable-adjacent artifacts. Security belongs in the serving design from the start; the full threat model, including prompt injection and tool abuse, is in AI Security. These are the serving-layer obligations.

Authentication and authorization. The gateway authenticates the caller, maps it to a tenant, and then checks authorization: may this tenant use this model, this version (a canary is not for everyone), this context length and this feature? The tenant identity is established once, at the edge, and propagated as a verified claim; replicas never trust a tenant ID in the request body. Service-to-service calls inside the cluster use mutual TLS or equivalent workload identity, and replicas accept traffic only from the router.

Quotas. Request counts are a poor unit when one request may cost 50 or 5,000 tokens. Quotas should be in tokens (input and output tracked separately, since output costs more compute per token), with a burst allowance. The practical wrinkle is that output length is unknown at admission. The common approach is to reserve against the maximum (input tokens plus max_output_tokens), then refund the unused part when the request finishes or is cancelled. The runnable Python below shows the mechanism and the arithmetic.

import time

class TokenBucket:
    def __init__(self, rate_per_s, burst, clock=time.monotonic):
        self.rate, self.burst, self.clock = rate_per_s, burst, clock
        self.level, self.last = burst, clock()

    def _refill(self):
        now = self.clock()
        self.level = min(self.burst, self.level + (now - self.last) * self.rate)
        self.last = now

    def reserve(self, tokens):
        self._refill()
        if self.level < tokens:
            return False
        self.level -= tokens
        return True

    def refund(self, tokens):
        self._refill()
        self.level = min(self.burst, self.level + tokens)

now = [0.0]                                    # fake clock for a deterministic example
bucket = TokenBucket(rate_per_s=20000 / 60, burst=4000, clock=lambda: now[0])  # 20,000 tokens/min
print(bucket.reserve(1500 + 1024))             # True: 1,500 input + 1,024 max output = 2,524 reserved
print(bucket.reserve(1500 + 1024))             # False: only 1,476 left
bucket.refund(1024 - 120)                      # request ended after 120 output tokens: give back 904
print(bucket.level)                            # 2380.0

Prompt and response handling. Prompts contain customer data and retrieved confidential documents. Decide explicitly what is logged: usually metadata (IDs, token counts, latency, version, status) always, content only when a tenant has agreed, for a bounded retention period, with access control and a redaction step. The failure to avoid is "debug logging" of full prompts that quietly becomes an unmanaged copy of confidential data. Responses stream through proxies and caches: make sure no shared cache stores them under keys that ignore the tenant (see Caching in AI Systems).

Transport and secrets. TLS on every hop that crosses a trust boundary; provider API keys, signing keys and registry credentials in a secret manager, injected at runtime, scoped to the service that needs them, and rotated. The browser never holds a provider key, which is the reason a hosted-API deployment still needs a gateway.

Artifact integrity. Weights and runtime images are code-adjacent supply-chain inputs. Fetch by content digest, verify before loading, and restrict who can publish to the registry. Be careful with serialization formats: Python's pickle-based formats can execute arbitrary code when loaded, as the Python documentation warns for pickle, so prefer tensor-only formats such as safetensors for third-party weights and treat any pickle checkpoint as executable.

import hashlib

def verify_artifact(path: str, expected_sha256: str, chunk: int = 1 << 20) -> None:
    h = hashlib.sha256()
    with open(path, "rb") as f:
        while block := f.read(chunk):
            h.update(block)
    if h.hexdigest() != expected_sha256:
        raise RuntimeError(f"digest mismatch for {path}; refusing to load")

Tenant isolation deserves honesty. Routing tenants to separate replicas isolates performance and fault domains. It does not by itself isolate data on a shared replica: batched sequences from different tenants share a process, and a prefix cache keyed without the tenant could serve one customer's cached state to another. Keys must include tenant scope, and contracts that demand stronger isolation should get dedicated replicas.

A serving system that cannot be observed cannot be rolled out or scaled with confidence, because the gates and the autoscaler both run on its metrics. The central idea is to trace a request through every stage so that "the assistant is slow" decomposes into which stage. Figure 8 shows one illustrative streamed request.

Instrument with a trace that starts at the gateway and propagates through the router and replica (trace context passed in headers), and give each span the attributes you will later want to slice by: model, version, rollout arm, tenant tier, region, replica, input and output token counts, finish reason. The metric families that matter are the following.

• Request level. Rate, error rate by code (429 versus 5xx versus client cancel are different stories), queue wait, TTFT, end-to-end latency, tokens in and out, finish reasons.
• Replica level. Active and waiting sequences, KV cache occupancy, tokens/s, step time, memory used versus allocated (allocated memory is what the engine reserved; used memory is what live sequences hold, and alarming on the wrong one hides pressure), accelerator error counters.
• Fleet level. Ready versus desired replicas, replicas in Loading or Warming, start-up duration, restarts by cause.

Be careful with label cardinality: per-tenant labels on every histogram bucket can overwhelm a metrics system when there are thousands of tenants. Use tenant tier or the top-N tenants as labels, and put the full tenant ID in traces and logs, where high cardinality is cheaper. Percentiles must be computed from histograms or sketches that merge correctly across replicas; averaging each replica's p95 does not give the fleet p95. The wider treatment of diagnosis, quality monitoring and SLOs is in AI Observability.

Pulling the earlier sections together gives a concrete contract and a deployment description. The contract is what clients depend on; the deployment description encodes readiness, drain, limits and rollout behaviour.

Endpoint contract (illustrative TypeScript)

type Role = "system" | "user" | "assistant" | "tool";

interface GenerateRequest {
  model: string;                 // logical alias, e.g. "support-assistant"
  conversation_id: string;       // stable key for canary bucketing and affinity
  messages: { role: Role; content: string }[];
  max_output_tokens: number;     // required: bounds cost, drain time and quota reservation
  stream: boolean;
  deadline_ms?: number;          // caller's overall budget; server may lower it
}

type FinishReason = "stop" | "length" | "cancelled" | "error";

interface GenerateResponse {
  id: string;                    // request id, also in X-Request-Id
  model_version: string;         // concrete version that served it, e.g. "support-assistant@2.0.0"
  output: string;
  finish_reason: FinishReason;   // "length" means truncated, not finished
  usage: { input_tokens: number; output_tokens: number };
}

interface ApiError {
  code:
    | "invalid_request"          // 400: schema error, or prompt over context limit
    | "unauthenticated"          // 401
    | "forbidden"                // 403: tenant not entitled to model/version
    | "rate_limited"             // 429 + Retry-After: tenant quota
    | "overloaded"               // 503 + Retry-After: queue full / shedding
    | "deadline_exceeded";       // 504: TTFT or total deadline
  retryable: boolean;
  request_id: string;
}

Deployment description (Kubernetes, illustrative values)

The field names below (strategy.rollingUpdate.maxSurge, startupProbe,readinessProbe, livenessProbe, terminationGracePeriodSeconds,lifecycle.preStop, topologySpreadConstraints, the autoscaling/v2 behavior.scaleDown.stabilizationWindowSeconds, and PodDisruptionBudget maxUnavailable) were checked against the Kubernetes documentation. The image, paths, ports, metric name and every number are placeholders chosen to be consistent with this chapter. The manifest is a sketch to adapt, not a tested production configuration: GPU scheduling also needs a device plugin and node labels for your environment, and the Pods metric requires a metrics adapter that serves it.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: support-assistant-v1
spec:
  replicas: 6
  minReadySeconds: 30
  strategy:
    type: RollingUpdate
    rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }   # add a ready replica before removing one
  selector:
    matchLabels: { app: support-assistant, version: v1 }
  template:
    metadata:
      labels: { app: support-assistant, version: v1 }
    spec:
      terminationGracePeriodSeconds: 90                  # > 1,024 tokens / 30 tok/s + queue + prefill
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels: { app: support-assistant, version: v1 }
      containers:
        - name: server
          image: registry.example.com/llm-runtime@sha256:<digest>
          ports: [{ containerPort: 8080 }]
          resources:
            limits: { nvidia.com/gpu: 1 }
          startupProbe:                                   # allows up to 60 x 10 s = 600 s to load
            httpGet: { path: /health/startup, port: 8080 }
            periodSeconds: 10
            failureThreshold: 60
          readinessProbe:                                 # true only after warmup + golden prompt
            httpGet: { path: /health/ready, port: 8080 }
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:                                  # engine loop progressing; no downstream calls
            httpGet: { path: /health/live, port: 8080 }
            periodSeconds: 10
            failureThreshold: 6
          lifecycle:
            preStop:
              exec: { command: ["sleep", "5"] }           # let the router see not-ready first
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: support-assistant-v1 }
spec:
  maxUnavailable: 1
  selector:
    matchLabels: { app: support-assistant, version: v1 }
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: support-assistant-v1 }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: support-assistant-v1 }
  minReplicas: 6
  maxReplicas: 18
  metrics:
    - type: Pods
      pods:
        metric: { name: sequences_active_plus_waiting }   # exported by the engine, via a metrics adapter
        target: { type: AverageValue, averageValue: "17" }
  behavior:
    scaleDown: { stabilizationWindowSeconds: 600 }

Note what the YAML cannot express: the version-weighted routing, the queue limit, the quota policy and the gate thresholds live in the router and rollout tooling (the routing sketch earlier), not in the Deployment. A reasonable division of labour is Kubernetes for replica lifecycle and the router for per-request decisions; many teams use a managed endpoint service or a service mesh for the router role. Whichever you choose, keep each decision in exactly one place so that a rollout does not require editing three systems in the right order.

No universal answer exists, because the options differ in what you are responsible for. With a hosted API the provider owns accelerators, engines, scaling, regional capacity and most of the replica-level reliability; you own the application, gateway, retries, quotas, evaluation, and the consequences of provider changes. With your own fleet you own all of the above plus the pager for GPU faults.

• Operational skill. A fleet needs people who can debug accelerator memory, driver and engine behaviour, and capacity. Hosted APIs need people who understand rate limits, versioning and evaluation, a smaller surface but not zero.
• Privacy and residency. Some data cannot leave an environment, or only under a contract the provider cannot offer in the required region. That can decide the question before cost is discussed. Check the provider's current data-handling terms rather than assuming.
• Model control and lock-in. Own serving lets you pin an exact version indefinitely, fine-tune, and move between hardware. A hosted API can deprecate a model on its schedule, change behaviour behind an alias, or change prices; mitigate with explicit version pins, an evaluation suite you can run against alternatives, and a thin abstraction at your gateway (but not so abstract that you lose model-specific features you rely on).
• Workload shape. Spiky, low-volume or experimental traffic suits pay-per-token: you pay nothing when idle. Steady, high-volume traffic with predictable shape can make dedicated capacity cheaper per token, if you keep it busy.

An illustrative break-even shows how the shapes interact. Suppose a fleet of 6 GPUs costs 6 × $2.50 = $15 per hour, or $360 per day (fictional price), and a hosted API costs a blended $1.00 per million tokens (also fictional). The fleet matches the API bill at 360 ÷ $1.00 = 360 million tokens/day, an average of 360,000,000 ÷ 86,400 ≈ 4,167 tokens/s. If your traffic averages 40% of that, the fleet's effective cost per token is 2.5× the nominal figure, and that is before the engineers' salaries and the spare capacity for availability from the previous section. Conversely at 100% steady utilisation, the fleet wins on marginal cost. Real prices, real throughput per GPU and real quality differences between models will move the line in either direction; the structure of the calculation, not these numbers, is the takeaway. Cost trade-offs beyond break-even are the subject of Cost Optimization.

Many production systems end up hybrid: a hosted frontier model for the hardest questions, a self-hosted small model for high-volume classification, and the router deciding between them per request. That is one more reason to treat the model alias in your router, not a vendor name in your application code, as the stable interface.

With a working serving layer you can answer the operational questions: which version served this request, whether a replica is ready, what happens when one dies, and whether a new model is safe to promote. You cannot yet answer the planning questions: how many replicas does next quarter need, how does the answer change when the mean output length doubles, what does queueing do to p95 as utilisation approaches the limit, and where does a second region pay for itself? Those require capacity models built on arrival rates, service times and concurrency, the material of the next chapter. A good habit before moving on is to write down, for your own system, the three numbers from this chapter that you do not yet have: per-replica concurrent-sequence limit, cold-start time, and the p95 request duration under load. They are the inputs to everything that follows.

1. 1

   Package a model as a versioned manifest: weights verified by content hash, tokenizer and chat template, runtime image and engine arguments, and device placement. Run a golden-prompt self-test with deterministic decoding before a replica takes traffic.

2. 2

   Use three probes for three questions: liveness decides restart, readiness decides whether to route traffic, and a startup probe covers slow, variable model loads (about 150 s typical, occasionally several minutes). Do not fail readiness merely because the replica is busy, and do not rely on an HTTP 200 to detect a GPU that has stopped making decode progress.

3. 3

   Route as a cache decision: honour model version and tenant policy, balance on cost rather than round-robin, and use affinity only when prefix-cache reuse repays the lost balance.

4. 4

   Autoscale on signals that track work (waiting plus active sequences per replica, queue wait, KV cache pressure) rather than CPU, expect a cold start of minutes, and scale down slowly.

5. 5

   Gate a rollout on measured signals: a small slice of traffic with sticky assignment per conversation, a minimum hold time set by the request rate, comparison of v2 with v1 on the same mix, and shadowing to test latency and errors first.

6. 6

   Availability costs spare capacity: spread replicas across zones, keep warm capacity for a region failure, set maxUnavailable and a PodDisruptionBudget for rollouts, and shed load deliberately when demand exceeds everything.

7. 7

   Treat streaming as its own contract: separate deadlines for first token, inter-token gaps and total time, cancellation propagated to the engine, and no automatic retry after output or a tool call has been delivered.