AI Gateway Failures: 429s, Streams & Recovery

A consecutive-failure counter per deployment, exported as a metric, crossing the breaker's trip threshold (six in this design) is the detection signal — paired with a per-deployment error-rate dashboard so an operator sees the circuit open in real time rather than inferring it from a drop in Provider A's traffic share after the fact. Prevention is largely structural here: the breaker is the prevention mechanism, not a separate control — the real operational task is tuning the trip threshold and cooldown window against real provider failure patterns, and alerting if a breaker stays open longer than its cooldown would predict (a sign the provider outage is longer than transient).

The fix is a gateway-wide retry budget — a rolling count of retries as a percentage of total requests (a common target is capping retries at 10% of request volume). Once the budget is exhausted, new failures return 503 immediately instead of attempting a retry, converting an amplifying failure into a flat, bounded one. See Retry & Backoff for the general retry-budget pattern.

Detection: total outbound request rate to both providers combined exceeding the admitted inbound request rate by a wide margin is the tell-tale sign of retry amplification — this ratio, not raw error count, is what should page someone. Prevention: capacity planning that assumes providers can correlate (a shared upstream region, a shared underlying model host) rather than treating 'two independent providers' as a guarantee of independent failure, and load-testing the retry budget's behavior specifically under simultaneous dual-provider degradation before it happens in production.

Replace local counters with a shared atomic counter store (Architecture Deep Dive, Chapter 03) — the admission check and the decrement happen as one indivisible operation, so only one of two simultaneous requests for the last unit can succeed.

Detection: a tenant's measured TPM usage (summed from the ledger) exceeding its configured TPM limit is a direct, unambiguous signal that admission allowed more than it should have — this metric should be alerted on independently of any customer complaint. Prevention: a pre-production load test that deliberately races concurrent requests against the last unit of a tiny test quota is the standard way to catch a local-counter regression before it reaches production, since the bug is invisible at low concurrency.

Atomic budget reservation at admission (Architecture Deep Dive, Chapter 04) bounds this to the per-request reservation-vs-actual gap (typically single-digit percent, from estimate conservatism) rather than an unbounded number of concurrent requests all slipping through the same unenforced check.

Detection: a real-time 'reserved vs. hard cap' meter (not a post-hoc daily rollup) is what catches this while it's still happening — a dashboard that only updates once an hour will always show the overshoot after the fact. Prevention: reject new admissions once reserved-but-unsettled spend alone would exceed the cap, even before any of those reservations finalize — the reservation itself, not just the eventual settled total, counts against the limit.

An explicit stream_error event, carrying chunks-delivered and incomplete: true (Architecture Deep Dive, Chapter 09). The client can offer an explicit 'regenerate' action; the partial usage is still billed for what was actually generated, since those tokens were real provider work.

Detection: a spike in the stream_error event rate, segmented by provider and deployment, pinpoints whether disconnects are concentrated on one upstream (pointing at the provider) or spread evenly (pointing at the gateway's own network path). Prevention: this failure mode can't be prevented outright — a mid-stream TCP reset is an external event — so the real prevention target is making sure every streaming code path actually emits stream_error on disconnect rather than silently dropping the connection, which is a testable invariant in itself.

Bounded per-connection buffers with upstream read throttling matched to client consumption rate (Architecture Deep Dive, Chapter 09) — the same flow-control discipline as Backpressure generally, applied to SSE relaying specifically.

Detection: gateway process memory climbing in correlation with concurrent streaming-connection count (not with request rate) is the specific signature of this failure, distinguishable from a generic memory leak by that correlation. Prevention: load-testing with deliberately slow/throttled readers — not just fast synthetic clients — is the only way to catch an unbounded per-connection buffer before it reaches production, since fast test clients never exercise the backpressure path at all.

Mark the reservation PENDING_RECONCILIATION on cancel rather than immediately releasing it; finalize once the provider's usage report (which may arrive asynchronously after the cancel) confirms the real generated amount.

Detection: a growing count of reservations stuck in PENDING_RECONCILIATION past a reasonable provider-reporting SLA (e.g. a few minutes) signals either a provider usage-reporting outage or a bug in the reconciliation consumer — either way, it should page before those reservations silently age out unresolved. Prevention: a scheduled sweep that force-reconciles or escalates any reservation still pending past a hard ceiling, so a provider-side reporting delay can never turn into a permanently stuck ledger entry.

The gateway can hold a short-lived in-flight marker per idempotency key, so a retry arriving while the original call is still outstanding attaches to the same in-flight request instead of starting a new one (this closes the gap for retries arriving within the same gateway's processing window). What it cannot fix: a retry arriving at a *different* gateway replica after the in-flight marker has expired, or a provider whose own API doesn't expose a way to check an in-flight request's status. See Idempotency for why idempotency keys dedupe your own system's bookkeeping but can't force a remote system's execution semantics.

Detection: a per-tenant ratio of 'idempotency keys seen' to 'distinct provider calls made' drifting above 1.0 indicates duplicate upstream execution is actually happening, not just a theoretical risk. Prevention: keeping the in-flight marker's TTL deliberately longer than the gateway's own upstream timeout (so a retry can never outlive the window where it would find no marker) closes the main version of this gap, even though it can't close all of them.

The tool-call permission check from identity/ACL enforcement (Complete Design, Chapter 05) rejects the disallowed tool call regardless of how the request was phrased — because it's enforced structurally against the tenant's policy, not by inspecting the prompt's intent. Guardrails remain valuable for what they're actually good at (known-pattern secrets, known-category toxic content) and are not relied upon as the access-control boundary.

Detection: redaction-triggered events and tool-call authorization rejections are both logged as metadata-only security signals — a spike in either, per tenant, is worth an automated alert regardless of whether the underlying request was ultimately blocked successfully. Prevention: periodic red-team testing of the tool-call authorization boundary specifically (not just the content guardrail) is what catches a policy-enforcement regression, since a guardrail passing its own tests says nothing about whether authorization is still correctly wired to every tool-call code path.

A dedicated, independent low-latency revocation channel (separate from the normal policy-snapshot cadence) for exactly this class of event — confirmed credential compromise — bypasses the control plane's normal distribution path and propagates within seconds via a lightweight broadcast gateways always listen for, even during a broader control-plane outage. This adds operational surface specifically to shrink the one freshness-window case that matters most; it is not used for routine policy changes.

Detection: any successful request authenticated with a key flagged as revoked-but-not-yet-propagated is a direct, loggable security event — the gateway can know after the fact exactly which requests fell in the exposure window, from telemetry alone, without needing the prompt. Prevention: paging on 'revocation requested' immediately, independent of whether the normal snapshot or the emergency channel delivers it, so an operator is watching the window close rather than assuming propagation succeeded silently.

The reservation record locks the price version at admission time (Architecture Deep Dive, Chapter 04); finalization always uses the locked version, never the live one. The result: during a price change, in-flight requests briefly finalize at the old price while new requests reserve at the new one — a short, intentional, and auditable transition window.

Detection: the ledger's per-request price-version field makes this trivially auditable after the fact — a finance reconciliation can always answer 'what price was this specific request billed at and why' without ambiguity. Prevention: price changes are themselves versioned control-plane snapshots, so this isn't a special case to remember — it's the same locked-at-admission mechanism every other policy field already uses, which is the point of routing all policy through one snapshot mechanism instead of a one-off price-update path.

Detection: a residency-tagged 503 NoEligibleDeployment rate greater than zero for a sustained period is a direct signal that a compliance-constrained tenant has no healthy in-region capacity — this should alert operations and account management simultaneously, since it's both a reliability and a customer-relationship event. Prevention: provisioning at least two independently-failing deployments within the compliant region for any tenant with a hard residency requirement, so a single in-region deployment failure doesn't become a total compliance-bounded outage for that tenant.

Shard the quota store by tenant-hash across multiple Redis instances/clusters, so one tenant's admission-check volume is physically confined to one shard rather than contending with every tenant on a single shared instance — the same partitioning principle used for hot-key problems in any shared cache.

Detection: per-shard p99 admission latency diverging from the fleet-wide average is the earliest signal of a hot shard, well before it's severe enough to show up in overall error rates. Prevention: tracking each tenant's admission-check rate against shard capacity at onboarding time, and re-sharding or giving a dedicated shard to any tenant whose traffic alone would saturate a shared one — treating this as a capacity-planning input, not an incident to react to after the fact.

A naive expiry job releases every expired reservation in full. For R-7731 that refunds $0.105 on top of a ledger charge of $0.0822, which double-counts money back to the tenant. For the timeout case, releasing the first attempt's reservation would hide a generation the provider billed. A naive retry that charges a second reservation would double-charge the tenant for one logical request.

Settlement runs ledger insert first, then the quota store's compare-and-set settle script. The reconciler treats an expired RESERVED reservation by checking the ledger: if a usage row exists it runs the same settle script with that usage; if no attempt was ever recorded it releases in full; if an attempt may have executed and no usage is known, it marks the reservation PENDING_RECONCILIATION and waits for the provider's usage report. Retries run under the original reservation, extended only when the quota store can cover the extra possible execution.

Detection: the count and age of reservations that are expired but unsettled, and any mismatch between ledger totals and quota-store spend per tenant per day, which a nightly comparison reports. Prevention: every quota-store transition checks the current status before acting, so repeating settle or release is harmless, and the ledger's unique key on reservation ID makes a replayed insert a no-op. Residual risk: a provider that never reports usage for a timed-out call leaves a reservation pending at its worst case until a hard ceiling escalates it to a person, which is a deliberate bias toward over-charging the budget counter, not under-charging it.