API Gateway Production Engineering

A route table looks trivial until it has a thousand rules and every request pays for matching them.

Naively, routing is a loop over rules calling a regex on each. That is O(n) per request and collapses under a large route table. API gateways use implementation-specific route-matching strategies. Some optimize path matching using prefix trees or other indexing structures, while others evaluate configured rules and priorities. Matching complexity and route precedence depend on the gateway and its configuration. Where a gateway ranks rules by specificity, /users/me can be handled before the /users/* catch-all; where it uses first-match, order the rules accordingly.

Whether the gateway verifies a token locally or asks an auth service is one of the most consequential design choices you'll make.

Local verification (stateless JWT)

With a signed JWT, the gateway verifies the signature using the auth provider's public key (fetched from a JWKS endpoint and cached) and checks exp, iss, and aud. No network hop per request — fast and scalable. The cost is revocation lag: a token stays valid until it expires, so you need short lifetimes plus refresh tokens, or a revocation list, to invalidate early.

Token introspection (opaque tokens)

With opaque tokens the gateway calls the auth service's introspection endpoint to check validity. This improves revocation speed, but adds a network hop to the request path. In practice you often cache positive introspection results for a few seconds to protect the auth service, accepting a small revocation window as the trade-off.

The algorithm you pick and where you store its state decide whether limits are accurate — or a lie clients route around.

Choosing an algorithm

Token bucket allows bursts up to a capacity and caps the sustained rate — the default for user-facing APIs. Fixed window counts per calendar minute; it is trivial but lets a client send 2× the limit around the window boundary. Sliding window weights the previous window to smooth that boundary burst at the cost of a little more state and math.

State in a cluster

The subtle production bug: with N gateway instances and per-instance counters, a client keyed by API key can get up to N× its intended limit because each instance counts independently. The fix is shared state — typically a Redis counter or bucket updated atomically (e.g. INCR with TTL, or a Lua script for token-bucket refill) so all instances agree. The trade-off is either a shared-store round trip on the hot path for strict global limits, or approximate local token leasing/sync for lower latency with a small overshoot window.

Aggregation trades client simplicity for gateway complexity — and it must fail gracefully, because now one call depends on many services.

A production aggregator fans out to backends in parallel, not sequentially, so total latency is the slowest call, not the sum. Each downstream call gets its own timeout and circuit breaker. When one backend is slow or down, the aggregator returns a partial response with the fields it could gather rather than failing the whole payload — the home screen still renders even if the recommendations service is having a bad day.

Discovery decides how the gateway finds live instances — and the client-side vs server-side choice shapes your whole topology.

Client-side vs server-side discovery

In server-side discovery, the gateway (or a load balancer) queries the registry and picks an instance; clients stay dumb. In client-side discovery, the caller itself queries the registry and load-balances — common inside a mesh. For a gateway, server-side is the norm: the gateway is already the smart edge, so it owns instance selection.

Health-checked registries

Registries like Consul, etcd, or Eureka store instance addresses plus liveness. Instances register on startup and send heartbeats; a missed heartbeat window evicts them. The gateway caches the healthy set and refreshes on a short interval or via a watch/subscription, so a newly registered instance starts receiving traffic within seconds and a dead one is dropped without a redeploy.

The gateway is the front door, so it's where a request's identity and story begin — get tracing right here and the rest of the system inherits it.

The gateway assigns a trace id (and a root span) to each request, injects it into downstream headers (W3C traceparent or X-Request-ID), and every service propagates it. Correlated logs let you grep one request across a dozen services; traces show the latency breakdown span by span. The gateway also emits the golden signals per route — request rate, error rate, and latency percentiles (p50/p95/p99) — which are usually your first alerting layer.

A gateway cluster is only as good as the mechanism that keeps every instance's configuration in sync.

Split the gateway into a data plane (the instances handling traffic) and a control plane (where routes, keys, and limits are defined). The control plane pushes config to every data-plane instance — via a shared database, a config service, or a watch on etcd/Consul. Instances are stateless in the architectural sense: they keep only ephemeral caches, open connections, and worker-local state, so the load balancer can add or remove them freely.

The gateway sits on every request, so when it's unhappy, everything looks unhappy. Start monitoring with three signals — the gateway's own latency, the error-code mix, and upstream health — because knowing where to look first is the skill.

What to watch

Common issues