AI Gateway Architecture: Routing, Quotas & Streaming

Virtual keys are hashed at rest exactly like password credentials (a keyed hash, e.g. HMAC-SHA256 with a server-side pepper) — a leaked database dump should not hand out usable keys. Only the hash and a short, non-secret prefix (for display/debugging, e.g. 'vk_live_8f2c…') are stored; the plaintext key is shown to the tenant exactly once, at creation.

A tenant can hold two active virtual keys simultaneously during a rotation window — the old key keeps working until the tenant confirms the new one is deployed, then the old key is revoked. Revocation takes effect on the next policy snapshot, bounded by the freshness window discussed in Chapter 10.

Real provider API keys live in a secrets manager the control plane reads from, never in the policy snapshot pushed to gateways in plaintext — gateways receive a reference/alias and fetch the actual secret through a short-lived, audited path, or the control plane injects it into the adapter layer directly at call time. Either way, a compromised gateway replica's memory dump should expose at most the credentials for requests currently in flight, not the entire credential vault.

A Redis Lua script (or equivalent single-round-trip atomic operation) checks the current bucket level and decrements it in one indivisible step: if bucket[key] >= cost then bucket[key] -= cost; return OK else return DENY. Because Redis executes Lua scripts atomically, two replicas racing for the last unit cannot both succeed — exactly the fix for the naive-local-counter race.

A fixed window (reset every 60s on the clock) is simple but allows a burst of up to 2x the limit straddling a window boundary. A sliding window (weighted average of the current and previous window, or a sorted-set of timestamps) bounds the burst more tightly at the cost of slightly more storage and computation per check. See Rate Limiting for the general algorithm comparison — this gateway applies the identical choice independently to RPM and TPM.

A single reservation call checks and decrements two counters in the same atomic script: request count and token count. Both must succeed for the reservation to succeed; if either is insufficient, the whole reservation is denied and neither counter is touched (no partial decrement).

If the shared counter store itself is down, the gateway has three honest choices, and must pick one explicitly rather than default accidentally: fail closed (reject all requests — safest for budget, worst for availability), fail open with a conservative local fallback limit (bounded risk, preserves availability), or fail open with no limit (never acceptable for a financial resource). Most production designs fail closed for budget/TPM and fail open with a strict local fallback for RPM, reflecting that a request-count overshoot is cheaper than a budget overshoot.

A reservation is a Redis hash stored next to the tenant's counters, keyed by the request ID: tenant, estimated input tokens, estimated output tokens (= requested cap), estimated USD, price version, status (RESERVED, SETTLED, RELEASED, PENDING_RECONCILIATION), a list of upstream attempts, created_at, and an expiry. A reservation that never settles, for example because the gateway process crashed mid-request, must not stay RESERVED forever, but it also must not simply be released: if an upstream call was started, the provider may have run it. The reconciler described below decides what to do.

On completion the gateway does two things in a fixed order. First it inserts the usage record into the ledger, keyed uniquely by the reservation ID, so repeating it creates nothing new. Second it asks the quota store to settle: a single atomic script that checks the status is still RESERVED, marks it SETTLED, and refunds reserved minus actual to the tenant's counters. If the status is anything else, the script does nothing and returns the existing result, so a retried settle can never refund twice. These two steps are not one atomic operation, because the ledger and the quota store are different systems. The next sections explain what keeps them consistent.

The estimate uses the tokenizer and price list in effect when the request was admitted. If pricing changes between reservation and finalization (a control-plane snapshot update mid-flight), the finalize step uses the price that was in effect at admission time, not the new one — otherwise a request's cost could change retroactively after the tenant already saw the estimate. This is a deliberate, documented choice, not an oversight.

A request sets max_tokens=500. Reservation assumes the worst case — the full output cap — priced at realistic asymmetric per-token rates: input at $0.00003/token, output at $0.00012/token (completion tokens typically cost several times more than prompt tokens with most providers). Reserved = 1,500 input × $0.00003 + 500 output × $0.00012 = $0.045 + $0.06 = $0.105. The call actually completes after only 310 output tokens: actual = 1,500 × $0.00003 + 310 × $0.00012 = $0.045 + $0.0372 = $0.0822. The ledger finalizes at $0.0822 and releases $0.0228 of reserved budget — exactly the value of the 190 unused output tokens at the output rate — back to the tenant's quota.

Atomicity here is per store, never across stores. Inside the quota store, one Lua script on one shard (counters and reservation record share a hash tag so they live on the same shard) performs each of these as a single indivisible step: reserve (check RPM, TPM and budget, decrement all, create the record), settle (check status, mark SETTLED, refund the unused part), and release (check status, mark RELEASED, refund everything). Inside the ledger, one idempotent insert keyed by reservation ID is atomic. No transaction spans the ledger and the quota store. Consistency between them comes from three things instead: a fixed order of operations (ledger first, then quota store), status checks that make every quota-store step safe to repeat, and a reconciler that repairs the gap if the gateway dies between the two. The ledger is the system of record for money; the quota counters are fast enforcement state that can be corrected from it.

Consider each point where a gateway replica can die. Before reserve completes: nothing was held, nothing to repair. After reserve, before any upstream call: the reservation expires, and because no attempt was ever recorded the reconciler releases it in full. After an attempt was recorded and before usage is known: the provider may have run it, so the reconciler does not release; it looks up the provider's usage report by the correlation ID and settles from that, and if no report arrives before a hard ceiling it escalates, leaving the worst case charged rather than quietly refunding it. After the ledger insert and before the quota-store settle: the reservation is still RESERVED while the ledger already holds the usage, so the reconciler finds the ledger row and runs the same settle script. Double charge is prevented because the ledger insert is unique on the reservation ID; double release is prevented because settle and release only act on a RESERVED reservation, so a second attempt finds a terminal status and changes nothing.

A timeout does not mean the provider did nothing. It means the gateway stopped waiting. So the attempt is recorded as unknown usage, and a second attempt is allowed only if the quota store can extend the same reservation to cover a possible second execution; otherwise the gateway stops and returns an error instead of risking uncovered spend. Where the provider supports an idempotency key, the retry sends the same key so a duplicate execution is not started; where it does not, a retry can produce two generations and two charges, and both are accounted against the reservation. Only one response is ever delivered to the client. An outright refusal (429, 5xx, connection refused) is different: the provider reports that it did not run the request, so the retry costs nothing extra and uses the existing reservation as is. Even so, providers' billing behavior differs, so the reconciler compares final provider usage with the gateway's records and flags a mismatch.

{ name: 'support-fast', deployments: [{provider:'A', model:'a-chat-v3', region:'us-east-1', capabilities:{streaming:true, tools:true, jsonMode:true, maxContext:128000}, priceVersion:'2026-10'}, {provider:'B', model:'b-chat-2', region:'eu-west-1', capabilities:{streaming:true, tools:true, jsonMode:false, maxContext:96000}, priceVersion:'2026-10'}] }

'support-fast supports streaming' is only true if every deployment behind it supports streaming — the registry must document capability intersection, not just union, or callers will intermittently get capability-mismatch errors depending purely on which deployment the router happened to pick that time.

Deprecating a specific provider model (not the logical name) is transparent to clients: remove or down-weight that deployment from the logical model's deployment list in a new control-plane snapshot. Clients calling 'support-fast' never change their code. This is the entire point of the indirection introduced in Complete Design Chapter 07.

Different providers emit different SSE event shapes (delta-text vs. full-message-so-far, different field names for the finish reason, different tool-call streaming protocols). The adapter's translateChunk is where this gets normalized into one gateway-wide chunk shape, so the client-facing stream format never changes when a provider changes behind the scenes.

Actual token usage always comes from the provider's own usage report in its response (or final streaming event), never computed by re-tokenizing the prompt and completion on the gateway side — different providers use different tokenizers, and a locally recomputed number would silently diverge from what the tenant is actually billed for (Invariant I7).

If scoring ran first and filtering second, a disqualified deployment could theoretically win the score and only get caught afterward — exactly the ordering bug that causes a residency violation to reach a provider before anyone notices. Filtering first makes the disqualification a pre-condition of entering scoring at all, not a post-hoc check.

These are two different signals operating at two different granularities. Circuit state is binary-ish (closed/open/half-open) and gates eligibility entirely. Health score is continuous (0..1, based on recent error rate and latency percentile) and only affects ranking among still-eligible deployments — a deployment can have a mediocre health score and still be selected if every alternative is worse.

A single request gets a bounded number of upstream attempts (commonly one primary plus one or two alternates, with jittered backoff — see Retry & Backoff) and a total deadline that covers every attempt, so retries can never stretch the client's wait without limit. Failover is allowed only while no bytes have been committed to the client. A provider's Retry-After header is honored verbatim when present; otherwise backoff starts at a small base delay with full jitter. All attempts run under the one reservation made at admission; the next sections explain what that means for accounting when an attempt times out.

CLOSED (normal — requests flow, failures counted in a rolling window) → OPEN (≥N consecutive or ≥X% failures — all requests to this deployment are skipped immediately, no network call attempted, for a cooldown period) → HALF-OPEN (cooldown elapsed — exactly one probe request is allowed through) → back to CLOSED on probe success, or OPEN again (with the cooldown possibly extended) on probe failure. See Circuit Breaker for the general pattern.

Circuit state is tracked per deployment (a specific model + region + provider combination), not per provider company — Provider A's us-east-1 deployment can be open while Provider A's eu-west-1 deployment (if the tenant's policy even allows it) stays closed. Collapsing these would incorrectly take healthy capacity offline.

The gateway reads from the upstream provider connection only as fast as it can write to the client connection (or an internal bounded buffer with a hard cap). When the client is slow, this naturally slows the read from upstream too — standard backpressure, the same mechanism as Backpressure in any producer/consumer system, applied specifically to SSE/chunked relaying.

A streaming request holds a concurrency slot (and its token/budget reservation) for the entire stream duration, not just the initial handshake. Slow streams — by far the ~10% long-tail minority — disproportionately hold slots, which is exactly why admission control must reason in terms of concurrent in-flight load (Little's Law), not just requests-per-second.

A mid-stream failure emits a distinct stream_error event type to the client (not a reused 'chunk' event with an error-shaped payload) carrying: chunks delivered so far, a reason code, and incomplete: true. This lets client UIs render 'response was cut off' affordances instead of silently truncating or retrying automatically in a way that could duplicate tool-call side effects.

On client disconnect, the gateway closes its own upstream connection where the provider's protocol supports cooperative cancellation, and always releases the concurrency slot immediately regardless of whether the upstream cancel succeeds cleanly. The budget reservation is not released: it is marked PENDING_RECONCILIATION and stays charged at its worst case until the provider's usage report (which may arrive after the cancel) confirms the actual generated amount, at which point the normal ledger-then-settle sequence runs. A failure after the first byte follows the same path with whatever usage the provider reported.

Each policy snapshot carries a monotonically increasing version number. A gateway logs which version it's operating on with every request's telemetry — so an incident review can answer 'was this request affected by the policy bug introduced in v42' precisely, instead of guessing from timestamps.

Gateways poll for (or receive a push of) the latest snapshot on an interval short enough to be operationally acceptable — commonly tens of seconds. A permission revoked between snapshots is honored only once the next snapshot lands; this is a deliberate, bounded staleness traded for removing the control plane from the hot path entirely. The bound must be documented and short enough for the actual threat model, not assumed to be zero.

If the control plane is unreachable, gateways continue serving on their last successfully loaded snapshot indefinitely (or until an operator-configured max-staleness kill-switch, if one exists for the deployment). New tenant onboarding and policy changes are blocked until the control plane recovers; existing traffic is unaffected.