System Design · 9 chapters · 12 decisions · ~26 min read
AI Gateway Design Trade-offs & Decisions
Twelve concrete decisions, each with a real failure or cost that motivates it, the realistic alternatives, and why one was chosen for this design — and when a different scale or risk tolerance would legitimately choose differently.
Decisions 1-2
Integration Shape: SDK, Proxy, or Gateway
Three points on a spectrum, not a strict progression — a single internal tool genuinely doesn't need a gateway, and knowing when to stop adding architecture is as important as knowing when to add it.
Decision 1 — Direct SDK integration vs. a gateway
| Option | Pros | Cons |
|---|---|---|
| Direct SDK call | Zero added latency hop; zero new infrastructure; trivial to reason about | No shared governance across teams; N copies of retry/error handling; N held credentials |
| Gateway | One enforcement point for identity, quota, routing, and cost; one place to see total spend | Added hop (tens of ms); new operational dependency; must stay highly available |
Decision 2 — Generic HTTP reverse proxy vs. a model-aware gateway
| Option | Pros | Cons |
|---|---|---|
| Generic reverse proxy | Reuses commodity infrastructure (nginx, Envoy); handles TLS, basic routing, simple rate limiting well | No concept of tenant, model capability, or token economics — can't prevent the noisy-neighbor failure |
| Model-aware gateway | Understands logical models, token/cost accounting, streaming semantics, provider capability differences | Purpose-built component to build and operate; more moving parts than a commodity proxy |
Decision 3
Local vs. Shared vs. Durable Quota
Three coordination tiers exist for a reason — using the wrong one for a given check is either unsafe (too weak) or needlessly slow (too strong).
Swipe to pan
| Tier | Coordination cost | Correct use |
|---|---|---|
| Local in-memory counter | ~0 (no network hop) | Single-replica deployments only, or as a cheap pre-filter before the real check |
| Shared atomic counter (Redis-like) | One network round trip, sub-millisecond | The actual RPM/TPM/budget admission decision — every multi-replica deployment |
| Durable SQL ledger | A transaction, several ms, more under contention | Authoritative settlement and audit trail — never the per-request hot-path check |
Decision 4
Hard Cap vs. Bounded Exposure
Both approaches reserve atomically before admission. They differ in what happens at the edge of the budget, and that difference has real product consequences.
Swipe to pan
| Option | Pros | Cons |
|---|---|---|
| Hard cap (deny once reserved ≥ budget) | Mathematically guarantees the budget is never exceeded, by construction | Rejects legitimate requests the moment reservations alone (not actual spend) approach the cap — can feel broken to a tenant who 'should' have budget left |
| Bounded exposure (allow a small, documented worst-case gap) | Fewer false rejections; smoother experience near the budget edge | A real, if small and bounded, chance of overshoot — must be explicitly disclosed to the tenant, never silently absorbed |
Decision 5
Rate-Limiting Algorithm Choice
Fixed window, sliding window, and token bucket all solve 'limit requests per unit time' — they differ in burst tolerance and implementation cost.
| Algorithm | Pros | Cons |
|---|---|---|
| Fixed window | Trivial to implement and reason about | Allows up to 2× burst across a window boundary |
| Sliding window (weighted or log-based) | Tightly bounds burst, smooth enforcement | More storage/compute per check; slightly more complex |
| Token bucket | Naturally allows controlled bursts up to the bucket size, smooths sustained rate | Needs a refill-rate tuning decision separate from the capacity decision |
Decision 6
Timeout-Based Health vs. Circuit Breakers
Detecting an unhealthy provider deployment can be passive (count real request failures) or active (periodically probe independent of traffic) — and the two are complementary, not mutually exclusive.
| Option | Pros | Cons |
|---|---|---|
| Passive (circuit breaker on real traffic failures) | No extra load on the provider; reacts to real, current failure modes | Needs live traffic to detect a problem — slow to notice issues on a rarely-used deployment |
| Active health probing (synthetic periodic requests) | Detects problems even on idle deployments; can probe before real traffic arrives | Costs real tokens/money on every probe; probe success doesn't guarantee real-traffic success (different payload shape) |
Decision 7
Streaming Honesty vs. Synchronous Fallback
Once a stream starts, should the gateway ever try to recover transparently by re-running the request on another deployment and re-sending output from the start?
| Option | Pros | Cons |
|---|---|---|
| Transparent mid-stream recovery (retry + re-stream from scratch) | User never sees a visible failure | Potentially double-bills the first partial generation; can duplicate tool-call side effects if any occurred before the failure; replaces shown content with a different generation — violates streaming honesty |
| Explicit partial-failure signal (chosen) | Honest about what happened; no duplicated side effects; client controls the retry UX | User sees a visibly interrupted response and must explicitly retry if they want a fresh attempt |
Decision 8
Caching LLM Responses: Savings vs. Safety
Caching can meaningfully cut cost and latency for repeated or near-duplicate prompts — but a cache sitting in front of a non-deterministic, tenant-isolated system has sharper failure modes than a typical HTTP cache.
| Option | Pros | Cons |
|---|---|---|
| No caching | Simplest; no staleness or cross-tenant leak risk | Pays full token cost for genuinely repeated prompts (e.g. identical system-prompt-heavy requests) |
| Exact-match caching (hash of normalized request) | Safe, deterministic hit criteria; easy to reason about and invalidate | Low hit rate — only catches byte-identical requests, which is rare outside specific workloads |
| Semantic caching (embedding-similarity match) | Higher hit rate — catches paraphrased, near-duplicate prompts | Risk of returning a subtly wrong cached answer for a similarity-matched-but-not-identical prompt; must never be shared across tenants regardless of similarity score |
Decisions 9-10
Policy Distribution & Deployment Topology
Decision 9 — Control-plane push vs. synchronous policy lookup
| Option | Pros | Cons |
|---|---|---|
| Synchronous lookup per request | Policy is always maximally fresh — zero staleness window | Puts the control plane's own availability and latency directly on every request's critical path |
| Async versioned snapshot push (chosen) | Control-plane outages don't affect live traffic; hot path has no external dependency | Bounded staleness window for policy changes, including revocations |
Decision 10 — Deployment topology: centralized gateway vs. sidecar vs. service mesh
| Option | Pros | Cons |
|---|---|---|
| Centralized gateway cluster | One place to operate, scale, and observe; simplest mental model | A network hop for every calling service; a single (if replicated) blast radius |
| Sidecar per calling service | No extra network hop for the calling service; failure isolated per-service | N operational instances to upgrade and patch; harder to get a single company-wide spend view |
| Service-mesh-integrated (gateway logic as a mesh policy) | Reuses existing mesh observability/mTLS investment | Couples AI-specific policy evolution to the mesh's own release cadence; fewer teams have deep mesh expertise |
Decisions 11-12
Guardrail Placement & Idempotency Key Scope
Decision 11 — Inline (blocking) vs. async (out-of-band) guardrails
| Option | Pros | Cons |
|---|---|---|
| Inline, blocking (scan before forwarding/returning) | Can actually prevent a violation from reaching the provider or the client | Adds latency to every request proportional to the scan; a slow scanner becomes the gateway's new latency ceiling |
| Async, out-of-band (scan a sampled or full copy after the fact) | Zero added latency on the request path | Cannot prevent anything — only detects and alerts after the content already left the building |
Decision 12 — Client-supplied vs. gateway-generated idempotency keys
| Option | Pros | Cons |
|---|---|---|
| Client-supplied idempotency key | Client controls retry semantics precisely — it alone knows whether a retry is 'the same logical request' | A buggy client can reuse a key across genuinely different requests, or never set one at all |
| Gateway-generated key (derived from request hash) | Works even for clients that don't participate explicitly | A hash-derived key can't distinguish 'identical by coincidence' from 'intentional retry' — risk of false-positive deduplication |