AI Gateway System Design Interview Guide

Name the race precisely (local vs. shared state across stateless replicas), then the fix: an atomic check-and-decrement against a shared counter store (Redis Lua script or equivalent single-round-trip atomic op), so only one of two simultaneous reservations for the same unit can succeed.

• What if the quota store itself is down? — Discuss explicit fail-open vs. fail-closed policy — and that failing closed for budget/TPM and a conservative fail-open for RPM is a defensible, commonly-used split.
• Why not just use a database transaction for this? — Latency — a transactional DB write on every admission doesn't meet the sub-10ms overhead target at hundreds of req/s; an in-memory atomic store does.

Explain why this differs from a pre-first-byte failure: content has already been shown to the user, so silently failing over would splice in a different generation — dishonest. The correct behavior is an explicit stream_error event marking the response incomplete, with billing settled against actual tokens generated.

• Does the client get billed for the partial response? — Yes — those tokens were real provider work. Billing reflects what was generated, not what was successfully delivered.

Distinguish after-the-fact metering (checks spend once a request completes — allows concurrent overshoot) from atomic reservation (reserves a conservative worst-case estimate before admission, debits the real amount after, releases the rest). Be explicit that even reservation doesn't guarantee zero overshoot — only a small, bounded, documented one.

• Can you guarantee zero overshoot? — No, and say so directly — tokenizer differences and delayed provider usage reporting leave a real, if small, gap. A hard cap can get arbitrarily close at the cost of more false rejections near the budget edge.

That's architecturally valid for a single team — walk through why it stops working at N teams: duplicated integration logic, N held credentials, no shared spend visibility, and critically no way to enforce a per-tenant model allowlist or atomic cross-tenant quota.

Two separate answers: the gateway's own request-handling scales horizontally (stateless replicas behind a load balancer, sharing one quota store and ledger). The upstream provider's own rate limit does not scale just by adding gateway replicas — that's a separate capacity negotiation, and the design should surface it rather than hide it.

• What becomes the bottleneck first? — Usually the shared quota store's op rate for a single hot tenant, or the upstream provider's own RPM/TPM ceiling — not the gateway's own compute, which is cheap to replicate.

It doesn't look up a provider model directly — it resolves a logical model name (e.g. 'support-fast') to a list of candidate deployments the registry maintains, each with its own documented capability set (streaming, function calling, context window, vendor). The registry must publish the *intersection* of capabilities across all deployments behind one logical name, not the union — otherwise a client's request can succeed or fail purely depending on which deployment the router happened to pick that time.

• What happens when a provider silently changes a model's behavior? — The adapter contract is the firewall — an adapter raises an explicit CapabilityUnsupported error rather than silently dropping a field the new model version no longer honors. Routing never masks a real capability regression as a transient failure.

No — there are at least two different 429s, and conflating them is a common interview mistake. A 429 from the gateway itself means this tenant's own RPM/TPM/budget reservation failed at admission; the client should honor Retry-After and back off per-tenant. A 429 relayed from the provider means the *provider's* aggregate limit was hit — which can happen even when the tenant's own admission succeeded, if the gateway's total traffic across all tenants exceeds what the provider allots the account. The two call for different operator responses: one is a per-tenant policy signal, the other is a capacity/procurement signal.

• Why not just treat every 429 the same and retry? — A blind retry on a gateway-issued 429 ignores a deliberate admission decision and can make a hot tenant's problem worse; a blind retry on a provider 429 without routing to a different deployment just re-hits the same exhausted capacity.

RPM counts requests; TPM counts a quantity that isn't known exactly until the response finishes. The gateway reserves an estimated token count at admission — input tokens from a real tokenizer pass over the prompt, output tokens from the request's max_tokens cap (the worst case) — against a sliding or fixed window counter in the same atomic store used for RPM. The estimate, not the eventual actual, is what's checked against the limit at admission time, because the actual isn't known yet.

• What if the real output comes in far under the estimate? — The reservation settles to the real token count at completion and releases the unused portion back to the window — the same reserve-then-settle pattern used for budget.

Reservation happens at step 4 of admission — after authentication and policy authorization, but strictly before the router picks a deployment or any upstream call is made. Reserving any later (say, after the provider accepts the call) reopens the exact race two concurrent requests can exploit: both could pass a non-atomic check before either one's usage is recorded. Reserving earlier (before authorization) would needlessly charge quota against requests that policy was always going to reject.

You mostly don't count them yourself — the adapter's extractUsage step reads the provider's own usage accounting, typically delivered in a final SSE event or trailing metadata after the last content chunk, and that provider-reported count is the only source of truth for actual billing, never the gateway's own re-tokenization of the streamed text. If a stream is canceled before that final usage event arrives, settlement is marked pending reconciliation until the provider's usage report (which can arrive asynchronously) confirms the real amount.

Before any bytes are sent, a retry is invisible to the client, so failing over to a healthy deployment is the right move, within a bounded number of attempts and a total deadline, all under the one reservation. A timeout deserves care: the provider may have run the request anyway, so the retry has to be covered by the reservation and the first attempt's usage reconciled later. Once content has started streaming, the client has already begun rendering one specific generation; silently retrying and splicing in a different provider's continuation would mean the user sees text from two unrelated generations stitched together, which is actively dishonest, not just a minor inconsistency. The only correct behavior past that point is an explicit stream_error event marking the response incomplete, settled for whatever was actually generated.

Residency is a hard filter, applied before scoring, not a preference weighed against price or latency — a deployment that doesn't satisfy the tenant's required region is removed from the candidate set entirely and can never be selected, even if every other eligible deployment is down. This holds during failure too: if the tenant's only compliant deployment is unhealthy, a healthy out-of-region deployment of the same logical model is still not an eligible fallback. The honest answer here is a 503, not a silent compliance violation.

• Isn't that worse for availability? — Yes, deliberately — for a regulated tenant, violating a residency constraint to preserve uptime is a worse outcome than a bounded, visible 503. That trade-off should be named explicitly, not hidden.

A pure-price router optimizes exactly one axis and ignores two others that matter just as much: latency (the cheapest deployment might also be the slowest, degrading the product) and health (a deployment can be cheap because it's currently struggling, about to trip its circuit breaker). The routing score is deliberately a weighted combination of price, latency, and health precisely so a single cheap-but-unhealthy deployment can't dominate every routing decision.

• Who sets those weights? — Policy, per tenant or per logical model — a cost-sensitive internal tool and a latency-sensitive customer-facing chat product should not share the same routing weights even if they call the same logical model.

A tenant (or the platform, for shared credentials) can hold two active virtual keys simultaneously during a rotation window — the old key keeps working while the new one is deployed and verified, and only gets revoked once the tenant confirms the cutover. Real provider API keys never leave server-side custody: they live in a secrets manager, and gateways either fetch them through a short-lived audited path or never see the raw secret at all, so a compromised gateway replica's memory exposes at most in-flight request credentials, not the whole vault.

Freshness is a deliberate, bounded staleness window — commonly tens of seconds — traded for keeping the control plane completely off the request hot path. A revocation that lands between two snapshot pulls is honored only once the next snapshot arrives; that bound must be documented and sized to the actual threat model, not silently assumed to be zero. If the control plane is unreachable entirely, gateways keep serving on their last good snapshot indefinitely, trading currency for availability.

• When would that staleness window be unacceptable? — A high-security environment needing instant revocation would add a dedicated low-latency revocation broadcast channel for that specific case, rather than making every policy change synchronous for everyone.

A fallback is a different, capability-compatible deployment the router tries after the first choice rejects a request — but only automatically, and only before any bytes have reached the client. The fallback must pass the same hard filters (residency, capability, policy) as the original candidate; a fallback is never a compliance escape hatch. After streaming has started, there is no automatic fallback at all — the honest behavior is an explicit stream error, not a silent swap to a different model mid-response.

A single correlation ID is attached at ingress and carried through every hop — gateway, router, adapter, provider call, and back — tagged with tenant, logical model, the deployment actually chosen and why, time-to-first-token, estimated and actual tokens, estimated and actual cost, any policy action taken, and the finish reason. Every one of those is metadata, not payload — full tracing and incident review is possible without the prompt or completion body ever being captured by default.

• When would you ever capture the actual prompt? — Only opt-in, purpose-limited capture — e.g. a tenant explicitly enabling debug logging for their own traffic — never as a default, and ideally with the captured payload itself access-controlled and separately audited from routine telemetry.

Concretely: every request is authenticated against a verified virtual key regardless of network origin — there's no implicit trust granted just because a call arrived from inside the VPC. Every hop re-checks authorization (tenant, model, capability) rather than trusting an upstream component's decision. Provider credentials are never exposed to the caller, even on introspection endpoints. And the control plane, not caller-supplied headers, is the only source of truth for what a tenant is allowed to do — the gateway never trusts a client to self-report its own entitlements.

Apply Little's Law — concurrency in flight equals arrival rate times average time in system (L = λW) — but treat it as a planning number, not a guarantee, because LLM request durations are not uniform: a long-tail minority of slow streams disproportionately holds concurrency slots far longer than the average suggests. Admission control has to reason about concurrent in-flight load directly, not just requests-per-second, or a burst of long streams can exhaust capacity while the RPS graph still looks calm.

• What if 10% of requests are 10x longer than average? — That tail alone can account for roughly half of total concurrency held at steady state — which is exactly why a single flat latency number stops being a useful planning input once a meaningful long tail exists.