AI Infrastructure Scaling

Watch a single-server system grow from 100 users to millions — hitting the limits of vertical scaling, discovering horizontal scaling, and absorbing a sudden traffic spike without downtime. The user counts and growth are hypothetical numbers chosen for illustration, not reported figures for ChatGPT or any other service.

How do you scale LLM inference infrastructure? Measure token throughput (tokens per second), estimate the concurrent KV-cache demand, fix a latency SLO, divide demand by derated per-replica capacity, and keep headroom for bursts and failures. Capacity planning for an AI system means converting a forecast of user behaviour into the quantity that actually runs out first, and then buying that quantity with headroom. For a conventional web service the quantity is usually requests per second. For a language-model service it is token work: prompt tokens that must be processed before the first word appears, output tokens that must be generated one step at a time, and the memory that every in-flight sequence occupies while it waits. The working method has five steps: characterise the workload as distributions (not averages); use Little's law to estimate how many requests are in flight; measure what one replica sustains on that workload while meeting a latency target; divide demand by derated per-replica capacity; then add headroom for bursts, for failures, and for the time it takes new capacity to become ready. Every step is checked by load tests that resemble production.

in flight (Little's law)  L = λ × W = 20 req/s × 5 s                = 100 requests
demand                    8 req/s × 100 output tokens               = 800 tokens/s
usable per replica        100 tokens/s benchmark × 0.70 utilisation = 70 tokens/s
replicas, expected load   ceil(800 / 70)                            = 12  (then add burst and zonal headroom)

Two cautions shape the rest of the chapter. Peak throughput is not the capacity you can promise: p95 latency degrades nonlinearly as a server approaches saturation, so usable capacity is a fraction of the maximum. And hardware capacity is not service capacity: the gateway, the retrieval cluster, tool quotas and the model fleet form a chain, and the slowest link sets the end-to-end limit. If you use a hosted model API you plan quotas and concurrency rather than GPUs, but the same arithmetic applies to the quantities you do own. Everything numeric here is illustrative; the method transfers, the constants do not.

Our running scenario is an enterprise support assistant. It answers policy questions from retrieved, authorised documents, and for a minority of conversations it calls an order-lookup tool through the orchestrator. It runs on a self-hosted model: a small fleet of replicas, each holding the weights and serving streams through a continuous-batching runtime (see Inference Infrastructure). Traffic has been a steady 8 requests per second at business peak. At 09:40 on a Tuesday the dashboard still says 8 requests per second, and p95 time-to-first-token (TTFT) has just tripled.

Nothing in the request-rate graph moved, because the workload changed underneath it. A new tenant was onboarded and its agents upload long contracts and ask for summaries. Requests are not interchangeable units, and the reason is that a request carries a prompt that must be prefilled, an answer that must be decoded, and sometimes several model calls. Take three classes and a simple cost model. A replica ingests prompt tokens at 4,000 tokens/s and, at its benchmarked batch size, generates 160 output tokens/s in aggregate (both illustrative). The replica-seconds one request consumes are then

policy question   1,300 in /  80 out   1300/4000 + 80/160   = 0.325 + 0.500  = 0.825 s
order lookup      3,200 in / 150 out   3200/4000 + 150/160  = 0.800 + 0.9375 = 1.7375 s
                  (two model calls: a planning call, then an answer after the tool result)
long summary      9,000 in / 400 out   9000/4000 + 400/160  = 2.250 + 2.500  = 4.75 s

The prototype mix was 85% policy questions, 14% order lookups and 1% long summaries. Its mean cost is 0.85×0.825 + 0.14×1.7375 + 0.01×4.75 = 0.992 replica-seconds, so one replica sustains about 1/0.992 = 1.01 requests per second at 100% busy, and 12 replicas sustain about 12.1. At 8 requests per second the fleet is 8×0.992/12 = 66% utilised. After the new tenant the mix is 80/12/8. The mean cost becomes 0.80×0.825 + 0.12×1.7375 + 0.08×4.75 = 1.2485 s, which is 26% more work per request. Utilisation climbs to 8×1.2485/12 = 83% with the request rate untouched. The simulated p95 request latency (the pool simulator from the load-testing section, run at 8 requests/s) rises from 1.74 s to 4.75 s with the 80/12/8 mix for this change alone.

The figure lets you slide between the two mixes (the intermediate points are a straight-line blend of the two stated mixes) and compare the request rate, the output-token rate and the replica load.

The same blind spot exists on the retrieval and tool sides: a question that triggers three retrieval rounds, a reranking pass and an order lookup is one request on the load balancer and a dozen units of work downstream. That is the problem the next sections give vocabulary and arithmetic for.

A capacity model is only as good as the workload description feeding it, and for language models a single average hides the part that causes outages. Prompt lengths are heavy-tailed: most questions carry a modest context, a few carry a contract. Output lengths are heavy-tailed too, because an open-ended "explain" question can run to hundreds of tokens while a yes/no answer ends in ten. Suppose the production logs give a median prompt of 1,100 tokens and a mean of 1,500. If you fit a lognormal (a common first model for such data), the mean-to-median ratio fixes the spread: σ = √(2·ln(1500/1100)) = 0.79, which implies a p95 of about 4,000 tokens and a p99 of about 6,900. A test that sends 1,500 tokens every time is therefore running a different, easier experiment than production. Output tokens behave the same way: with a median of 70 and a mean of 100, σ = 0.84, a p95 near 280 and a p99 near 500.

Distributions matter because the scheduler experiences the tail. A 9,000-token prompt monopolises prefill compute for a noticeable slice of time and holds a large KV allocation for the whole decode; a 500-token answer keeps its sequence slot, and its KV memory, alive through five hundred decode steps. Beyond the length distributions, a workload profile has to say how the load arrives and who sends it:

• Concurrency and streaming duration. With streamed responses a request stays open for the whole generation, so connections, sequence slots and KV memory are held for seconds, not milliseconds. Concurrency, not rate, is what fills a replica.
• Burstiness. Requests arrive in clumps (a morning shift starting, a campaign email). Poisson arrivals are the gentlest assumption; real traffic has a higher variance and produces longer queues at the same average rate.
• Region and residency. A tenant that must keep data in one geography cannot be served from a convenient idle pool elsewhere, so capacity is planned per region.
• Model mix. A small model for classification and routing, a larger one for answers, an embedding model for retrieval: each has its own capacity curve and queue.
• Cache behaviour. Hit rates shrink effective demand (an exact-answer hit costs no model work; a prefix-cache hit removes prefill work). They are measured per tenant and per request type, and they change when prompts change; see Caching in AI Systems.
• Tenant fairness. One enterprise tenant can generate more tokens than all the others together. Whether that tenant can degrade everyone else is an architectural decision, not an emergent property.

{
  "window": "business-peak hour, 5-minute buckets",
  "arrivals": { "mean_rps": 8, "p99_5min_rps": 11, "pattern": "bursty" },
  "prompt_tokens": { "mean": 1500, "p50": 1100, "p95": 4000, "p99": 6900 },
  "output_tokens": { "mean": 100, "p50": 70, "p95": 280, "p99": 500 },
  "classes": { "policy": 0.85, "order_lookup": 0.14, "long_summary": 0.01 },
  "streaming": true,
  "prefix_cache_hit_rate": 0.35,
  "tenants": { "top1_share_of_tokens": 0.42 },
  "regions": { "eu": 0.30, "us": 0.70, "eu_residency_required": 0.22 }
}

A caution on the cache field: a 35% prefix-cache hit rate does not make demand 35% smaller. It removes the prefill work for the shared leading tokens of those requests and nothing else, so you apply it to the prefill term of the cost model only.

Little's law says that in a stable system, the average number of items inside it equals their arrival rate multiplied by the average time each spends there:

L = λ × W
L: average number in the system (requests)
λ: average arrival rate = throughput in steady state (requests/second)
W: average residence time, arrival to departure (seconds)

Suppose peak growth brings the assistant to λ = 20 requests/second, and the measured mean end-to-end time, from the gateway receiving the request to the last streamed token leaving, is W = 5 s. Then L = 20 × 5 = 100 requests are in flight on average. If W drifts to 8 s because the fleet is getting loaded, L = 160 at the same request rate: the in-flight count grows by 60% and nothing about the arrival rate announced it. This is why in-flight requests are a better health signal than rate.

The law applies to any boundary you draw, so apply it stage by stage. Suppose the 5 s decomposes into gateway and auth work (0.1 s), retrieval (0.4 s), the order-lookup tool (2 s for the 15% of requests that call it, which averages 0.3 s over all requests), and model time of 4.2 s covering queueing, prefill and streamed decode. The parts add to 5.0 s. Then the in-flight population at each stage is 20×0.1 = 2, 20×0.4 = 8, 20×0.3 = 6 and 20×4.2 = 84, which sums to 100.

This also shows what Little's law does not do. It does not give GPU capacity. It tells you how much work is simultaneously present, which sizes connection pools, worker counts, queue depth limits and the number of concurrent sequences the fleet must hold; whether the accelerators can progress those sequences fast enough is a separate question (the next section). The law also assumes a stable system (arrivals equal departures over the window, so it is not valid while the queue is growing), uses means (it says nothing about the tail), and depends on a consistent definition of "in the system". For a streamed answer the residence time includes the full generation, not only the time to first token; a W measured to TTFT would undercount occupancy by the length of the stream.

Per replica the same arithmetic constrains memory. With 12 replicas and 84 requests in the model stage, each holds about 7 sequences. If a sequence's KV cache is 400 MB at its length, 7 sequences need 2.8 GB of the replica's KV pool whether or not the compute has room. The KV figure in this paragraph is a placeholder to show the shape of the check; Inference Infrastructure derives KV size from model dimensions.

Dividing a replica's capacity into a single "tokens per second" number conflates two different activities. Prefill processes the whole prompt in parallel; it is compute-heavy, its cost scales with prompt length, and it determines TTFT. Decode generates one token per sequence per step; each step has to read the model weights and the sequence's KV cache from memory, so it is usually limited by memory bandwidth and batching rather than arithmetic, and it determines the pace of the stream (see Inference Infrastructure for the mechanics). Input and output tokens are therefore not interchangeable units: a 3,000-token prompt with a 30-token answer and a 30-token prompt with a 3,000-token answer present entirely different loads.

They compete because they share the device. In a continuous-batching runtime every iteration has a token budget: a new request's prefill either preempts decode steps for the running sequences (their streams stutter and inter-token latency spikes) or is split into chunks that are interleaved with decode (the streams stay smooth, and the newcomer's TTFT lengthens). Neither choice removes the work. Disaggregating prefill and decode onto separate pools is a further option that trades the contention for transfer of KV between pools; it belongs in the specialised-serving phase later, not in a first deployment.

For planning, a linear approximation around the benchmarked operating point is useful: the replica-time a request consumes is prompt/prefill_rate plus output/decode_rate, which is the model we used for the traffic spike. It is an approximation: real decode throughput rises with batch size until memory bandwidth or KV capacity binds, and prefill throughput depends on chunking and kernel efficiency. That is exactly why it must be calibrated by a benchmark of your model, runtime, hardware and prompt distribution, not taken from a data sheet; theoretical FLOPS are an upper bound on prefill, not a promise.

If you call a hosted model API, the provider has already done this calibration, and the constraints surface as quotas, usually stated as requests and tokens per minute, and as concurrency limits. The arithmetic you own is converting your demand into those units. At 8 requests per second with 1,600 total tokens per request (1,500 in, 100 out), the consumption is 8×1,600×60 = 768,000 tokens per minute; at the 20 requests/second peak, it is 1,920,000. Whether your provider counts input and output tokens in one limit or two, and how cached or batch traffic is treated, differs by provider and plan, so check the current documentation of the one you use. Quota exhaustion looks like a capacity incident even though you own no GPUs.

Vertical scaling means each replica gets more: a larger accelerator, more memory, or more devices per replica through tensor or model parallelism (see Parallel Processing). Horizontal scaling adds more replicas behind the router. They address different limits, and in a model-serving fleet the first decision is usually forced by whether the model fits.

Take a hypothetical model whose weights occupy 140 GB (decimal; for example 70 billion parameters at 2 bytes each) and hypothetical GPUs with 80 GB of memory each (not a claim about any product). One GPU cannot hold it. Split across two with tensor parallelism and each holds 70 GB; assume 6 GB per GPU is reserved for the runtime and activations, which leaves 80 − 70 − 6 = 4 GB per GPU, or 8 GB of KV pool, for the replica. Across four GPUs each holds 35 GB and 80 − 35 − 6 = 39 GB is free: 156 GB of KV pool for twice the hardware, nearly 20 times the room. At an assumed 0.25 MiB of KV per token, 8 GB holds about 30,500 tokens of live context while 156 GB holds about 595,000. The 2-GPU replica cannot hold many long conversations at once; the 4-GPU replica can, which raises the achievable batch size and so the per-replica throughput.

The advantages of a larger replica are that it fits the model at all, may lower single-request latency (more devices cooperate on each token), and provides the memory that makes batching efficient. Its costs are the interconnect traffic between devices (an all-reduce per layer, so the devices need a fast link, and it is a poor idea to span slow links), a bigger blast radius per failure, coarser scaling granularity (adding capacity means adding four GPUs at a time), longer model-load times, and harder scheduling: four free devices on one host are scarcer than one. Horizontal scaling is the reverse: it adds throughput and fault tolerance in small steps, but does nothing for a single request's latency, and it cannot help if one replica cannot hold the model or if requests are individually too long.

A sound default is to scale vertically only until the model fits with a useful KV pool, and horizontally after that. If a replica is too small to batch efficiently, move up; if it is large enough and demand still exceeds capacity, add replicas. The operational limits matter as much as the technical ones: some tiers of accelerator are hard to obtain on demand, and a replica that spans a whole host or rack constrains which zones you can use.

Now the fleet-sizing arithmetic. Suppose a benchmark of one replica, using the workload profile above and with the latency target that p95 TTFT must stay under 1.5 s, finds that the replica sustains 100 output tokens per second before the target is violated (illustrative). With mean 100 output tokens per request that is about 1 request per second per replica. Planned demand is 8 requests/second × 100 = 800 output tokens/s. You do not plan to run at the benchmark knee, because queueing delay climbs steeply there (next section), so apply a target effective utilisation of 70%, giving 100 × 0.7 = 70 tokens/s of planning capacity per replica:

demand            = 8 req/s × 100 output tokens   = 800 tokens/s
usable / replica  = 100 tokens/s × 0.70           = 70 tokens/s
replicas (base)   = ceil(800 / 70) = ceil(11.43)  = 12

+ burst: plan for 1.5 × peak = 1,200 tokens/s
replicas (burst)  = ceil(1200 / 70) = ceil(17.14) = 18

+ zonal loss (3 zones, any one may fail → 2/3 must carry the burst)
replicas (zonal)  = ceil(1200 / (70 × 2/3)) = ceil(25.71) = 26 → 27 (9 per zone)

Read the three lines as three different promises. Twelve replicas meet the expected peak with a healthy margin. Eighteen also absorb a 50% burst without breaching the latency target, if you are willing to be at 70% utilisation at that burst. Twenty-seven keep that promise even after the loss of an availability zone, rounded up so each zone holds the same number. The jump from 12 to 27 is not waste; it is the price of the reliability objective, and discussing it as a price is what turns a sizing exercise into a decision. Many teams choose a lighter promise for the second and third lines (a degraded-quality mode during a zone loss rather than full capacity) and write that choice down.

import math

def replicas_needed(demand_tok_s, per_replica_tok_s, target_util=0.7, burst=1.0, zones=1):
    effective = per_replica_tok_s * target_util
    base = math.ceil(demand_tok_s * burst / effective)
    if zones <= 1:
        return base
    survive = (zones - 1) / zones
    need = math.ceil(demand_tok_s * burst / (effective * survive))
    return zones * math.ceil(need / zones)  # equal replicas per zone

print(replicas_needed(800, 100))                      # 12
print(replicas_needed(800, 100, burst=1.5))           # 18
print(replicas_needed(800, 100, burst=1.5, zones=3))  # 27

Why must the 100 tokens/s come from a benchmark? Because it depends on everything the data sheet cannot know: the model and its quantisation, the runtime and its scheduler settings, the prompt and output distribution, the share of cache hits, the number of concurrent sequences that fit, and the latency objective that defines "before the target is violated". The same hardware sustains very different numbers at a p95 TTFT target of 0.8 s and 5 s. A replica that looks twice as fast on a short-prompt benchmark can be slower on the real mix. Treat the benchmark as a measurement of the service, not of the chip, and repeat it whenever model version, runtime version or workload profile changes.

A server that is on average 90% busy is not 10% idle in a useful way: arrivals are random, so sometimes a burst arrives while all slots are busy, and the surplus waits. Queueing theory gives the shape. For a single server with random arrivals and service times (the M/M/1 model), mean response time is S/(1−ρ), where S is the uncontended service time and ρ the utilisation. At ρ = 0.5 that is 2 S; at 0.7 it is 3.3 S; at 0.8 it is 5 S; at 0.9 it is 10 S; at 0.95 it is 20 S. Equal increments of load cost ever more time. Beyond ρ = 1 the queue has no steady state at all: it grows for as long as the overload lasts.

A pool of servers fed from one queue behaves better. With 12 servers (the M/M/c model, using the Erlang C formula) the mean response is 1.05 S at ρ = 0.7, 1.15 S at 0.8, 1.53 S at 0.9, 2.35 S at 0.95 and 4.8 S at 0.98. The knee moves right as the pool grows, which is a real argument for fewer, larger fleets over many tiny ones. But three caveats stop this from becoming comfort. First, those are means; users feel p95 and p99, which grow faster. Second, LLM service times are far more variable than exponential (a 4.75 s long summary beside 0.8 s policy answers), and higher variance makes queues longer. Third, a replica is not a single server: it serves many sequences at once, and its per-sequence speed drops as the batch grows, so utilisation and latency are coupled inside the replica as well as between replicas. The shape of the curve survives; the exact numbers do not, which is why we use 70% as a planning target and why you measure the knee.

Head-of-line blocking and fairness

In a first-come-first-served queue, a request is delayed by whatever is ahead of it. Behind one 4.75 s long summary (or a prefill that occupies the device for 2.25 s), twenty short policy questions each pay a wait that has nothing to do with their own size. Chunked prefill limits how long one prompt can monopolise an iteration inside a replica; at the fleet level you can route classes to separate pools or at least separate queues, schedule by estimated cost rather than arrival order, and enforce per-tenant limits so that one tenant's burst consumes its own budget rather than the shared queue. Shortest-job-first scheduling reduces mean latency but starves long requests, so any such policy needs an ageing rule.

Shedding load deliberately

Because a queue past saturation only grows, the only choices under sustained overload are to add capacity, to reduce work, or to refuse some requests. Refusing nothing is itself a choice: every request waits, almost all of them exceed their client timeout, and the replicas spend their capacity on answers nobody is waiting for. Admission control bounds the damage: limit the queue by expected wait (not just length: 40 queued long summaries are not 40 queued policy questions), return a fast 429 or 503 with Retry-After for what cannot be served in time, and drop requests whose clients have already gone. A refused request costs a few milliseconds; an abandoned one costs full prefill and decode.

Until now we have scaled the model fleet. A support assistant, however, is a pipeline, and its end-to-end capacity is the minimum of the capacity of its stages, each expressed in the same unit: user requests per second. Suppose, illustratively, that the gateway sustains 60 requests/s, the retrieval cluster (embedding, vector search and reranking) 45, the order system's API a quota of 10 calls/s which, since 15% of requests use it, supports 10/0.15 = 66.7 requests/s, and the 12-replica fleet 12 × 0.7 = 8.4 at the planning utilisation. The model fleet is the bottleneck by a factor of five, so adding gateway capacity would change nothing. Scale the fleet to 29 replicas for the 20 requests/s peak (29×0.7 = 20.3) and the next bottleneck is retrieval at 45, then the tool quota at 66.7. A scaling plan lists the order in which these limits will be reached, with the measurement that would reveal each.

The dimensions you can turn, with the signal that tells you which one is hot:

• Replica count raises token throughput linearly when the router balances well; the signal is queue wait and in-flight per replica.
• Tensor or model parallelism changes what one replica can hold or how fast it answers; it is a design change, not a runtime knob.
• Batch size and sequence limits (maximum concurrent sequences, per-iteration token budget) trade throughput against per-stream latency; they are bounded by KV memory.
• KV memory sets how many tokens of context can be live. Its signal is cache utilisation and preemptions; its levers are context limits, quantised KV where quality allows, bigger or more devices, or fewer sequences per replica.
• Retrieval scales by index replicas, sharding or approximate-search settings, and it has its own tail latency that feeds into every model call.
• Gateways and queues scale on connections and open streams, which long generations hold for tens of seconds.
• Regions add capacity close to users and under residency rules, at the price of fragmentation (below).
• External tool and provider quotas are capacity you do not control; they are negotiated, cached around or protected with per-tenant budgets.

An autoscaler is a control loop: observe a metric, compare it with a target, compute a desired replica count, change the deployment, wait, and repeat. The loop is easy to write. It is hard to make it effective for model servers because the actuator is slow. A new replica has to be scheduled onto a node with free accelerators, which may need a new machine; the container image (often many gigabytes) must be pulled; the weights must be read into GPU memory; the runtime has to initialise and warm up kernels. Even with generous assumptions this takes minutes, while a spike hurts within seconds.

Choose a signal that leads the damage

CPU utilisation is nearly useless for GPU serving. GPU utilisation as typically reported says only that something ran on the device in the sampling interval, not how close it is to saturation. Better signals are in-flight (running plus queued) requests per replica, queue wait time, and KV cache utilisation, which sit directly on the bottleneck. With in-flight per replica as the metric, the Kubernetes Horizontal Pod Autoscaler computes desiredReplicas = ceil(currentReplicas × currentMetric / targetMetric). From the planning model, the target in-flight per replica is 70% of 5 = 3.5. If the 12 replicas observe 8 in flight each, the desired count is ceil(12 × 8 / 3.5) = ceil(27.4) = 28, close to the 29 replicas that the token arithmetic gives for 20 requests/second. Two independent routes to the same number is a useful consistency check.

Spike versus readiness

Now trace a spike from 8 to 20 requests/second at t = 0 against a fleet of 12 replicas that can complete 12 requests/second at saturation. Assume (illustratively) 30 s for metrics scraping and the controller's decision, 60 s to provision a node, 40 s to pull the image, 30 s to load weights and 10 s to warm up. New replicas serve traffic from t = 170 s.

The arithmetic: the deficit is 20 − 12 = 8 requests/s for 170 s, so 1,360 requests accumulate. The last of them waits 1,360 / 12 ≈ 113 s, far beyond any chat timeout. After readiness the fleet clears 29 req/s against 20 arriving, a net 9/s, so the backlog takes 1,360 / 9 ≈ 151 s, until t ≈ 321 s. Autoscaling alone, in other words, protects the fifth minute, not the first two. If the queue is capped so that no request waits more than about 10 s (a bound of roughly 120 requests at 12 req/s), the remaining ≈1,240 requests in that window are shed rather than answered late.

There are four ways to shrink the window. Pre-provision headroom: a standing 18 replicas turns the deficit into 2 req/s (340 requests, a ~19 s wait for the last). Predict: scale on schedule or on leading indicators (business hours begin, a campaign was sent, an upstream queue is growing) so that readiness precedes load. Shorten readiness: pre-pulled images on warm nodes, a weights cache on local disk or a shared fast store, and keeping a small pool of already-warmed spare replicas. Shed and degrade in the meantime (see failure capacity below). Each of these trades money or complexity for time.

Hysteresis and cooldown

Scale-down is the dangerous direction: removing a replica that took four minutes to start because traffic dipped for thirty seconds guarantees a repeat of the incident. Use asymmetric rules, fast up and slow down, a stabilisation window on scale-down so the controller acts on the recent maximum, and a tolerance band around the target so that noise does not cause constant changes. Kubernetes' HPA documents all three: the behavior field with per-direction stabilisation windows and rate-limiting policies, and a default tolerance of 10%. Its defaults (no scale-up stabilisation, a 300 s scale-down window) already follow this asymmetry. The sketch below is illustrative; the custom metric name is invented and requires a metrics adapter you must provide, and a replica that serves traffic only after it passes a real readiness check is assumed.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: assistant-llm }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: assistant-llm }
  minReplicas: 12            # standing floor from the sizing exercise
  maxReplicas: 40            # bounded by accelerator quota and budget
  metrics:
    - type: Pods
      pods:
        metric: { name: llm_inflight_requests }   # running + queued, per pod (adapter required)
        target: { type: AverageValue, averageValue: "3500m" }   # 3.5 per replica
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies: [ { type: Percent, value: 100, periodSeconds: 60 } ]
    scaleDown:
      stabilizationWindowSeconds: 600
      policies: [ { type: Pods, value: 1, periodSeconds: 300 } ]

Past a certain size, one pool in one region stops being the right shape. Users far from the region pay round-trip latency on every streamed token batch; a cloud region can lose a dependency; and some tenants may legally require that prompts, retrieved documents and logs stay within a geography. The architecture that follows has a global routing layer that consults a tenant policy table (residency, quotas, tier), regional stacks that each contain a gateway, retrieval index and model pool, and an explicit rule about which traffic may spill across regions.

Regions change the capacity arithmetic in three ways. First, fragmentation. Suppose a 20 requests/s global peak splits 30% EU, 70% US. A single pool needs ceil(20/0.7) = 29 replicas at one request/s each and a 70% target. Two regions need ceil(6/0.7) = 9 for the EU and ceil(14/0.7) = 20 for the US, still 29, but each needs one spare for N+1 (10 + 21 = 31), and a small third region carrying 3% of traffic (0.6 req/s) still needs two replicas to be available at all, a utilisation of 0.6/2 = 30%. Total 33 replicas against 29: each region pays a floor. Second, cross-region spill turns idle capacity elsewhere into usable capacity for tenants who are allowed to use it, at an added round trip (an illustrative 90 ms against a TTFT budget of 1,500 ms is affordable for TTFT, but cross-region retrieval round trips per question add up). For residency tenants it is forbidden, so their regional capacity must stand alone. Third, cost: idle capacity in every region is the premium paid for latency and compliance, so keep the regional footprint to what the measured user and legal geography justify.

Inside a region, tenants compete for the same replicas, and the defence is isolation by budget. Give each tenant a budget measured in estimated cost, not requests, because a request-count limit lets a tenant spend its allowance on 9,000-token prompts. A token-bucket limiter charged by estimated replica-seconds is a small mechanism with a large effect:

import time

class TokenBudget:
    """Per-tenant budget of estimated replica-seconds, refilled continuously."""
    def __init__(self, rate_per_s, burst, clock=time.monotonic):
        self.rate, self.burst, self.clock = rate_per_s, burst, clock
        self.level, self.stamp = burst, clock()

    def try_take(self, cost):
        now = self.clock()
        self.level = min(self.burst, self.level + (now - self.stamp) * self.rate)
        self.stamp = now
        if self.level >= cost:
            self.level -= cost
            return True
        return False

def estimate_cost(prompt_tokens, max_output_tokens, prefill_tps=4000, decode_tps=160):
    return prompt_tokens / prefill_tps + max_output_tokens / decode_tps

t = [0.0]
budget = TokenBudget(rate_per_s=2.0, burst=6.0, clock=lambda: t[0])
cost = estimate_cost(9000, 400)                   # 4.75 replica-seconds
print(budget.try_take(cost), budget.try_take(cost))  # True False  (6 -> 1.25 left)
t[0] += 2.0
print(budget.try_take(cost))                      # True  (1.25 + 2.0*2.0 = 5.25 >= 4.75)

The budgets should add up to more than the fleet (otherwise it is a static partition and wastes idle capacity) but each should be small enough that no single tenant's limit equals the fleet. Premium tenants can additionally get a reserved slice, or a dedicated pool, a mechanism with its own fragmentation cost.

The sizing arithmetic above depends on one measured number, so the credibility of the whole plan rests on how that measurement is taken. A good LLM load test differs from a web-service benchmark in four ways: the load generator must follow the workload distributions, not a single canned prompt; it must be open-loop; it must control the state of the system (cold or warm, cache populated or not); and it must report percentiles at the stages that matter.

Open-loop versus closed-loop

A closed-loop test uses a fixed number of virtual users, each sending its next request only after the previous reply. If the server slows down, the generator automatically sends fewer requests, so the test hides the very overload it should reveal (a form of coordinated omission). An open-loop test sends requests on a schedule (for example Poisson arrivals at a given rate) regardless of completions, as real users do. The program below simulates a 12-replica pool serving the 85/14/1 mix (costs from earlier) and compares the two. It is a queueing simulation, not a benchmark of real hardware.

import heapq, random

CLASSES = [("policy", 0.85, 0.825), ("order", 0.14, 1.7375), ("long", 0.01, 4.75)]  # share, replica-seconds
REPLICAS = 12

def sample_service(rng):
    r, acc = rng.random(), 0.0
    for _, share, secs in CLASSES:
        acc += share
        if r <= acc:
            return secs
    return CLASSES[-1][2]

def percentile(xs, q):
    xs = sorted(xs)
    return xs[min(len(xs) - 1, int(q * len(xs)))]

def open_loop(rate, n, seed):
    """Poisson arrivals independent of completions; FIFO over REPLICAS servers."""
    rng = random.Random(seed)
    free = [0.0] * REPLICAS
    heapq.heapify(free)
    t, lat = 0.0, []
    for _ in range(n):
        t += rng.expovariate(rate)
        start = max(t, heapq.heappop(free))
        done = start + sample_service(rng)
        heapq.heappush(free, done)
        lat.append(done - t)
    return lat, n / t

def closed_loop(workers, n, seed):
    """Each worker sends its next request only after the previous one completes."""
    rng = random.Random(seed)
    free = [0.0] * REPLICAS
    heapq.heapify(free)
    ready = [(0.0, w) for w in range(workers)]
    heapq.heapify(ready)
    lat, last = [], 0.0
    for _ in range(n):
        t, w = heapq.heappop(ready)
        start = max(t, heapq.heappop(free))
        done = start + sample_service(rng)
        heapq.heappush(free, done)
        heapq.heappush(ready, (done, w))
        lat.append(done - t)
        last = max(last, done)
    return lat, n / last

open  8.0 req/s     thr= 8.05/s  p50=  0.82  p95=  1.74  p99=  2.31
open 10.5 req/s     thr=10.57/s  p50=  0.97  p95=  2.29  p99=  3.90
open 11.5 req/s     thr=11.57/s  p50=  1.62  p95=  5.38  p99=  9.42
open 12.5 req/s     thr=12.58/s  p50= 95.64  p95=169.24  p99=179.33
closed 12 workers   thr=12.12/s  p50=  0.82  p95=  1.74  p99=  1.74
closed 24 workers   thr=12.12/s  p50=  1.82  p95=  2.79  p99=  3.19

Read the table for what it teaches about the metrics. The median is nearly useless near the knee; p95 and p99 carry the signal. Throughput equals offered load until capacity and then stops, which is why peak throughput is a poor capacity statement: the highest throughput a closed-loop test reports comes with latency a user would refuse. Capacity is the highest open-loop rate at which p95 (or p99) still meets the SLO, here between 10.5 and 11.5 req/s, then derated for variability, which is the same logic as the 70% target.

A load-test plan

objective: find open-loop capacity of one replica and of the 12-replica pool at p95 TTFT <= 1.5 s
workload:
  source: sampled from production logs (PII removed), stratified by class and tenant
  prompts: replay real prompt lengths; synthetic text allowed if tokenised length matches
  outputs: max_tokens set high; let generation stop naturally so output lengths follow the model
  mix: { policy: 0.85, order_lookup: 0.14, long_summary: 0.01 }
  cache: run twice - cold (empty prefix cache) and warm (35% hit rate as in production)
arrivals: open-loop Poisson, stepped 4 -> 14 req/s in 1 req/s steps, 10 min per step, + one 3x burst
state:
  - cold start: measure time from scale-out to first successful request (readiness SLO)
  - soak: 2 h at 70% of measured capacity (memory growth, KV fragmentation, tail drift)
measure: [TTFT, inter-token latency, end-to-end] x [p50, p95, p99]; queue wait; KV utilisation;
         preemptions; error and timeout rate; in-flight per replica; tokens/s (in, out)
dependencies: retrieval and order tool run against realistic stubs with recorded latency distributions
pass: p95 TTFT <= 1.5 s and error rate < 0.1% at planned peak; degrade gracefully at 1.5x

Three details deserve emphasis. Cold and warm states answer different questions: cold tells you the worst case after a deploy or failover, warm tells you steady state, and the hit-rate distribution should be that of production (replaying the same prompt makes every request a cache hit and makes capacity look wonderful). Percentiles should be computed per class as well as overall, since an overall p95 can be satisfied by short questions while long ones suffer. And test the dependencies you own at the same load, since a retrieval cluster that adds 300 ms at 8 requests/s may add 2 s at 20.

A fleet sized for expected peak has no tolerance for the loss of anything. Failure capacity is the extra provisioned so that a defined failure still leaves the service within its SLO, and it is written down as a requirement: which failures, which SLO. Two replicas are the minimum for any availability (N+1). A zone-level requirement, as in the sizing example, means that with three zones the survivors carry 3/2 of their normal load, so each zone must be planned at two-thirds of the target utilisation. The same logic applies to a rack or a host that holds a multi-GPU replica.

For fleets built on a hosted API the failure is different: the provider can run out of capacity for you (a 429), a regional endpoint can be degraded, or your account quota may be exhausted by one runaway tenant. Mitigations are a second provider or region as a fallback (with an evaluated quality gap and, for regulated data, a compliant arrangement), per-tenant limits inside your quota, and retry budgets, so that retries do not multiply a brownout into an outage. For self-hosted fleets accelerator scarcity is a capacity failure of its own: the autoscaler asks for a node and none is available. Reserved or committed capacity covers the baseline, and the burst plan needs a fallback that does not depend on getting more devices.

Shedding should be SLO-aware: decide in advance which work is cheap to refuse. Background and batch jobs, free-tier tenants, and the optional enhancements (a reranking pass, an extra retrieval round) go first. A degraded-model fallback has to be a measured quality trade, evaluated on your questions beforehand, with the degraded state visible to the user or the calling system. An interactive answer that is shorter or less thorough is better than none; a wrong answer from an unevaluated fallback is worse than a refusal. And the ladder needs the hysteresis shown in the figure; otherwise a fleet that flaps between states spends its time on transitions.

The architecture above is where a successful service ends up, not where it should start. Each step has a cost in engineering and money, and a metric that justifies it.

1. 1

   Prototype on a hosted API

   There is no fleet; you manage quotas, timeouts and retry budgets. Instrument tokens per request, per tenant and per class from day one, because the workload profile you will need later is built from this data. Move on when quota limits, cost per task or data-handling requirements change what is acceptable.
2. 2

   One or two self-hosted replicas

   You own a benchmark, a deployment and a readiness check. Capacity is one replica’s measured throughput; two replicas are availability, not performance. Add replicas when planning utilisation (not peak utilisation) is routinely above target.
3. 3

   Small fleet behind a gateway

   Introduce the admission queue, per-tenant budgets, cost-aware routing and the sizing sheet with burst margin. Autoscaling here is often manual or scheduled: with five replicas and daytime-shaped traffic, a calendar beats a metrics pipeline.
4. 4

   Autoscaled, multi-zone fleet

   Add the metrics adapter, scale-up/scale-down asymmetry, zonal headroom and a warm spare. Justified when traffic is spiky enough that a static fleet would be either idle or overloaded, and when a zone loss would otherwise breach the SLO.
5. 5

   Multi-region

   Add regions only for measured reasons: users whose latency cost is visible, residency obligations, or a regional-dependency failure that has already hurt. Each region pays the floor described above, so avoid adding regions for a few percent of traffic that a neighbouring region could serve.
6. 6

   Specialised serving

   Prefill/decode disaggregation, model routing across sizes, and KV offload belong here, adopted when profiling shows prefill-decode interference or memory pressure as the bottleneck that simpler knobs cannot fix.

A retrieval cluster follows its own, usually later, curve: one index replica is fine until query latency or availability needs another, and its scaling depends on index size and filters more than on the model. In the small phases most of your capacity risk sits in prompt length and tenant behaviour, which cost nothing to constrain.

Months later, a Friday review: the on-call has pasted a week of dashboards for the self-hosted EU pool of 12 replicas. Request rate is flat at 8 requests/s. Starting Wednesday, the queue depth rose from near 0 to 45, p95 TTFT went from 0.9 s to 4.2 s, KV cache utilisation went from 17% to 98%, and the runtime reports preemptions (sequences evicted and recomputed). Decode throughput per replica is unchanged. Which knob first?

Start by locating the bottleneck with the formulas already built. Request rate is unchanged and decode speed is unchanged, so it is not an output-token story. Look at the prompt-token histogram: the mean prompt grew from 1,500 to 2,700 tokens. Deploy history shows why: on Wednesday retrieval depth went from top-4 to top-8 chunks of about 300 tokens to improve answer recall, and 4×300 = 1,200 plus 300 tokens of other context grew to 8×300 + 300 = 2,700.

cost before = 1500/4000 + 100/160 = 0.375 + 0.625 = 1.000 replica-s   -> 1.00 req/s per replica
cost after  = 2700/4000 + 100/160 = 0.675 + 0.625 = 1.300 replica-s   -> 0.77 req/s per replica

utilisation before = 8 / (12 x 1.00)  = 0.67
utilisation after  = 8 / (12 x 0.769) = 0.87   (above the 0.70 target; queueing grows)

Little's law per replica (lambda = 8/12 = 0.667 req/s):
  before: W =  5 s  -> L = 3.3 in flight, x 1,600 tokens = 5,300 tokens  -> 17% of a 30,500-token KV pool
  after:  W = 16 s  -> L = 10.7 in flight, x 2,800 tokens = 29,900 tokens -> 98% of the pool
  (16 s is the observed mean residence on the dashboard)

fleet needed at 70% target with the new prompts: 8 x 1.3 / 0.7 = 14.9 -> 15 replicas (+3)

The explanation fits all four symptoms. Prefill work per request rose 80% (0.375 to 0.675 s), pushing utilisation past the planning target; queueing delay lengthened TTFT; and a longer wait means more requests in flight, each now holding more KV, until the pool filled and the runtime began evicting sequences, which makes everything slower still, since an evicted sequence repeats its prefill.

The ordering of remedies comes from how fast each works, how reversible it is and what it costs.

1. 1

   First: cut the prompt, not the traffic (minutes, reversible, free)

   Return retrieval to top-4 for tenants where the evaluation does not show a recall loss, or retrieve 8 and rerank to the best 4 (see Reranking) so quality is kept while the prompt returns toward 1,500 tokens. That restores 0.67 utilisation and relieves the KV pool at once, with no hardware.
2. 2

   Second: stop the bleeding while that rolls out (seconds)

   Tighten admission to a maximum expected wait and reduce max concurrent sequences enough that the KV pool stops thrashing: fewer sequences running at full speed beat many evicted ones. Shed the lowest-priority tenant’s batch traffic.
3. 3

   Third: add replicas (4 to 5 minutes, costs money)

   If the evaluation says top-8 is worth keeping, plan 15 replicas (three more) and move the standing floor, because readiness takes minutes and the cost is permanent. Scaling out first would have treated a symptom of a prompt-size change with hardware and hidden the quality-vs-cost decision.
4. 4

   Last: change the replica shape (days)

   More KV per replica (the 4-GPU shape from earlier) is a redesign justified only if long prompts are a permanent part of the product.

The durable fix is in the process: prompt-token percentiles per tenant and per route belong on the capacity dashboard (see AI Observability), and a change to retrieval depth, system prompt or tool schemas is a capacity change that needs the load test to be rerun before rollout. The model fleet was never too small in the abstract; the work per request moved.

The sizing chain so far treats every request as new work. Reuse changes the inputs: an exact-answer cache removes arrivals from the model entirely (λ becomes λ×(1−h) for the cached class); a prefix cache removes the shared leading tokens from the prefill term but leaves decode untouched; a retrieval cache lowers the retrieval stage's load but does not change what the model sees. Each reuse layer therefore enters the capacity model in a different term, and each is valid only under conditions: the same tenant, the same document version and permissions, the same prompt template and model. A hit rate measured on a benchmark that replays the same prompts is the most common way plans end up too optimistic, because it vanishes the day the system prompt or retrieval layout changes, which is when you can least afford a surprise.

Capacity planning also hands its numbers to the finance side: replica-hours per accelerator tier, committed versus on-demand capacity, and idle headroom are the cost model that Cost Optimization then optimises, while this chapter's job was to make sure the numbers can be defended. Next is the layer that turns repeated work into saved capacity, along with the correctness rules that keep it safe.

1. 1

   Request rate is not capacity. A change in workload mix (for example a tenant sending longer prompts) can raise token demand while requests per second stays flat, so plan and alert on tokens, not just requests.

2. 2

   Characterise the workload as distributions (prompt and output lengths, concurrency, burstiness, cache behaviour, tenant skew); the scheduler experiences the tail, not the mean.

3. 3

   Little's law gives requests in flight (arrival rate × time in system), applied stage by stage, but it does not give GPU capacity; replica memory and tokens per second still come from a benchmark of your own workload.

4. 4

   Prefill and decode compete for the same replica, and a benchmarked tokens-per-second figure is the planning input. A 35% prefix-cache hit rate does not make demand 35% smaller because it removes only part of the work.

5. 5

   Near saturation latency explodes: a server that is 90% busy is not usefully 10% idle. Plan headroom, and under sustained overload choose between adding capacity and shedding load.

6. 6

   Autoscale on a signal that leads the damage (in-flight requests per replica, queue wait, KV cache utilisation), not CPU or reported GPU utilisation, and expect new replicas to take minutes: pre-provision headroom, scale up fast and down slowly.

7. 7

   Size for failure and for reuse: leave capacity to survive the loss of a replica or region, and put prompt-token percentiles per tenant and route on the capacity dashboard so drift is caught before the queue grows.