AI Gateway System Design: From Proxy to Production

One tenant's traffic, quota exhaustion, or misconfiguration must never degrade another tenant's service. This is the #1 correctness invariant for this design — the equivalent of 'no double-selling' in a booking system.

Because output length isn't known until generation finishes, the gateway can never promise zero possible overshoot against a budget — only a documented, bounded worst case, enforced by an atomic reservation made before admission.

Once a response has started streaming to a client, a retry or fallback can never silently replace what's already been shown with content from a different generation. Partial failure must be surfaced explicitly, never papered over.

A shared request/response shape across providers is a convenience, not a guarantee that every provider supports every feature. An unsupported capability is rejected explicitly, never silently dropped.

Authentication, authorization, and quota/budget reservation sit on every single request's critical path — they need to add single-digit milliseconds of overhead, not a noticeable fraction of total latency, which is comfortably true for an atomic cache operation and comfortably false for a synchronous call into a slow control plane.

Operational observability (tenant, model, tokens, cost, latency, routing decision, finish reason) must be fully traceable without ever requiring the prompt or completion text. Payload capture is opt-in, purpose-limited, and redacted.

For reasoning purposes only: 120 req/s average, 1,200 req/s peak for ~5 minutes, ~1,500 average input tokens, up to 500 requested output tokens, ~10% of requests are long streams. These are planning numbers, not measured production traffic — treat every number in this module the same way.

Each tenant gets a key the gateway itself issues and controls — not the provider's real key. The gateway maps that virtual key to a tenant, a team, and a policy. The real provider credential is injected server-side, immediately before the call upstream, and never returned to a client or logged in plaintext.

Identity alone isn't enough — the gateway also needs to know what each tenant may use. Team A's policy might allow a fast, cheap model class but not an expensive reasoning model. A disallowed model request is rejected with 403 *before* any network call to a provider — no tokens spent, no upstream latency, no leaked capacity.

A request with no valid credential at all gets 401 (unauthenticated — 'I don't know who you are'). A request with a valid credential naming a disallowed model gets 403 (unauthorized — 'I know exactly who you are, and the answer is no'). A well-built client treats these very differently: 401 might mean 'refresh my credential,' 403 should never be blindly retried.

Replace local counters with a shared, atomic store (a Redis-backed token bucket is the common choice) that every replica calls before admitting a request. The check-and-decrement happens as one indivisible operation, so only one of two simultaneous requests for the last unit can win; the other is denied with honest retry guidance (the provider's real Retry-After, not an invented number).

A request-count budget and a token budget don't move together — one large request can exhaust token budget while barely touching request-count budget. Both need their own atomic reservation, checked independently.

At admission time, output length is unknown — only the client's requested cap is known. Reserve pessimistically: estimated input tokens (using the right tokenizer for this model) plus the full requested output cap. When the request finishes using less, the unused part is refunded to the budget at settlement. The number shown mid-request is a reservation, not a bill.

The identical race exists for spend: forty concurrent requests, each individually fine against a daily cap, can collectively blow past it if spend is only checked *after* each request completes. The fix is the same shape — reserve a conservative dollar estimate atomically at admission, record the real amount in the ledger on completion and refund the rest of the reservation. This bounds exposure; it does not make overshoot impossible, because tokenizer differences and delayed provider usage reporting leave a small, honestly-documented gap.

Clients only ever ask for a stable name like 'support-fast.' The gateway's model registry maps that name to one or more approved deployments, each with documented capabilities: streaming, tool calls, context length, JSON mode. This is a stable contract for the client and a flexible routing decision for the gateway.

Each provider deployment has its own adapter that translates the gateway's standardized request into that provider's native shape. When a target deployment can't honor a requested capability — say, structured JSON output — the adapter rejects the request explicitly rather than silently dropping the requirement. A caller that thinks it got structured output but actually got free text can crash downstream, or worse, silently misparse.

The router eliminates any candidate that fails a hard constraint — required capability, data residency, policy — *before* comparing price or latency. A cheaper, disqualified deployment never enters scoring. Getting this ordering backwards is how compliance incidents happen: scoring first and discovering the disqualification afterward.

If the first-choice provider rejects a request (429, 5xx) *before any bytes have been sent to the client*, a bounded retry against a capability-compatible, healthy alternative is safe. The client never sees the first provider's bad moment. This safety condition disappears the instant a response starts streaming — see Chapter 08.

Six consecutive failures from a deployment opens its circuit — new requests skip it entirely for a cooldown window, favoring a healthy alternative. After cooldown, a single half-open probe decides whether to close the breaker (deployment is trusted again) or restart the cooldown. See Circuit Breaker for the general pattern this specializes.

Time-to-first-token and total completion time fail independently — a system can have excellent TTFT and still take ten seconds to finish. Track both; a single flat latency number stops meaning much once ~10% of requests are long streams.

If a client reads slowly (a flaky connection, a backgrounded tab) and the gateway keeps buffering provider chunks without limit, enough concurrent slow clients becomes an unbounded memory problem. Apply the same flow control any producer-faster-than-consumer system needs — see Backpressure — slow reading from upstream, or cap the buffer and drop the connection past a limit.

Seventeen chunks into a response, the provider connection drops. The client has already rendered half an answer — there's no safe way to seamlessly continue that exact sentence on a different model or a different generation; claiming it's the same completion would be a lie. The gateway terminates the stream with an explicit error event and marks the response incomplete. It never pretends success.

If a caller disconnects mid-stream, the gateway propagates the cancellation upstream where supported and frees the concurrency slot immediately — but the provider may already have generated (and billed) tokens the client never received. The reservation stays charged until the provider's usage report confirms what was actually generated, and only then is the unused part refunded.

A client that times out and retries with the same idempotency key gets its ledger entry deduplicated to one — but the original request may still be executing upstream when the retry goes out, so the *provider* can end up running inference twice for what the client experiences as one logical request. A local ledger (Idempotency) dedupes your own bookkeeping; it cannot force a remote provider to have exactly-once execution semantics. State that gap honestly rather than promising it away.

A guardrail that detects and redacts a pasted secret before it reaches a provider or a log line is useful, and imperfect — false positives and false negatives are both normal. But a prompt that tries to talk its way into an disallowed tool call is an authorization problem, not a content-filtering one. No amount of pattern matching on text substitutes for the tenant's actual model/tool permissions enforced back in Chapter 05.

A single correlation ID follows a request through every hop — client, gateway, chosen deployment, back — carrying tenant, logical model, deployment chosen and why, TTFT, estimated/actual tokens, estimated/actual cost, any policy action taken, and the finish reason. None of that requires the prompt body; payload capture stays off by default.

Spread traffic across multiple gateway replicas behind a load balancer; all of them share the same atomic quota store and the same durable ledger, which is what makes 'stateless' safe — any replica can handle any request because none hold state the others lack. Replicas scale request-handling throughput and failure isolation; they do not increase the upstream provider's own rate limit, which remains the real ceiling.

Tenant policy, the model registry, pricing, and provider credentials change far less often than requests arrive, so they live in their own control plane. Changes produce a new versioned policy snapshot, pushed asynchronously to every gateway replica — never a live admin-database query inside a per-request hot path, which would put the slowest, flakiest part of the infrastructure on the critical path of every chat response.

Because gateways hold the last-known-good snapshot, they keep serving normally during a control-plane outage. The documented cost: a permission revoked moments before the outage might not take effect until the next snapshot arrives, bounded by the policy's freshness window. A design that insists on instant revocation would need a synchronous control-plane call on every request — exactly the hot-path dependency just avoided.

If an EU-restricted tenant's required deployment goes down, a healthy US deployment of the same logical model is *not* an eligible fallback — compliance constraints filter the candidate set before the router ever scores anything, exactly as in normal-path routing, just now under failure.