Distributed Rate Limiter System Design: Step-by-Step Architecture

A distributed rate limiter is a shared admission decision: before a request reaches expensive work, a fast check consults a policy and an authoritative counter, and answers allow or deny. The counter has to be atomic, because many gateways ask about the same caller at once, and it has to be bounded, because the check itself can be overloaded or unreachable. Everything else in this design — token buckets, Redis scripts, shards, leases, fail modes — is a consequence of those two sentences.

Start with the harm rather than the mechanism. Northstar API is a fictional company, and every name, limit and capacity in this module is a teaching input, not a real policy or a measurement. Its backend comfortably serves about 2,000 requests per second of cheap reads like GET /catalog. A report export (POST /reports/export) costs roughly ten times a read. Atlas, a large partner, starts a backfill job that hammers both. Birch, a small and well-behaved customer on the same API, starts seeing slow checkouts.

Notice what is absent from that picture: no attack, no malformed input, no broken authentication. Every request from Atlas is legitimate and authorized. The backend simply has finite capacity, and without an admission decision in front of it, whoever sends fastest takes the largest share. Latency rises first, because requests wait in queues; timeouts and retries follow, which add more load; eventually the success rate for everyone falls. The Visual Design Journey on this page starts from exactly this picture — scenes 1 and 2 show the first request and then the burst that breaks the backend.

A rate limiter is the smallest mechanism that restores fairness here. It does not make the backend faster. It decides, per caller and per request, whether the request may consume backend capacity right now, and it answers cheaply enough that saying no costs far less than serving the request would have. If you want the single-process fundamentals — what a limit is, why it exists, the four classic algorithms in isolation — the Core Concepts lesson on rate limiting covers them. This page picks up where that one stops: what changes when the caller can reach many gateways, many regions, and a coordination layer that can itself fail.

Before drawing anything, decide what the limiter is responsible for. A limiter that tries to be billing, bot defense and a request queue at once ends up doing each badly.

What the limiter must do

Admit or deny each request using a policy keyed on tenant, user or API key, route class, and optionally the cost of the work. Several limits can apply at once and all of them must permit the request. Every decision carries metadata — whether it was allowed, which policy version decided, how much remains, a retry hint when denying, and a decision ID for debugging. State must be bounded in memory and in time. Policies are published centrally with versions and rollback. Identity comes from authenticated claims or a trusted edge, never from a header the client can set.

What it is not

Several neighbouring mechanisms are easy to blur together with it, and the design stays honest only if they stay separate.

The last row deserves emphasis. A per-tenant limit shields the backend from a legitimate customer going too fast. It does not stop a volumetric attack, because the attack saturates your network or your gateways before any identity is known, and it must not be sold as a substitute for bot detection or authorization. Your API gateway is where the limiter usually runs; edge rate-limiting products sit in that edge layer and are a complementary, coarser tool. Billing is similar: counters here may be approximate or lose a few increments in a failover, which is fine for protecting a backend and unacceptable for invoicing.

The policies this page will use

The algorithm decision, made up front

The design uses a token bucket for rate, evaluated atomically in one store authority, with a semaphore or lease for concurrency and a weighted cost for expensive routes. A fixed window survives only for coarse anonymous rules such as P-ANON, where its weakness (the boundary burst, next chapters) is acceptable. Sliding log and sliding counter appear as comparison points, and GCRA — an arithmetic way of writing the leaky bucket as a meter — as an optional equivalent formulation. Naming the choice now keeps the rest of the page from presenting five algorithms as a menu.

Load that the design must survive

Assume, as a planning input rather than a statistic, a baseline of 50,000 admission calls per second across the fleet and a 10× surge to 500,000 per second. That is one decision per request before retries, hierarchical checks or skew. The journey's load-math scene (scene 4) walks through these numbers; chapter 12 below turns them into shard counts.

The first version is a counter in the gateway's own memory. It is worth building, and worth understanding exactly where it stops being true.

With one gateway, a dictionary keyed by API key, K1 → 7, incremented per request and compared with a limit, is a complete limiter. No network hop, no serialization, no new failure mode. For a single instance, an internal tool, or a prototype this is the right answer, and a team that reaches for Redis on day one has bought coordination it cannot yet use.

Its failure mode is precise rather than vague. The count is truthful about one process only. It is lost on restart, which resets everyone's allowance. It never expires unless something sweeps it, so memory grows with every key ever seen. And it assumes that all of a caller's traffic reaches this process — an assumption that holds right up to the moment a second gateway is added behind the load balancer. Scenes 5 and 6 of the Visual Design Journey show the counter in a single window filling up and the first denial.

A fixed window is the simplest counter that expires: count requests in aligned intervals, reset at the boundary, deny above the limit.

Take P-ANON: 20 requests per 10 seconds, with windows aligned to multiples of 10 seconds since the epoch. The key for a client is the IP plus the window index, floor(now / 10); increment it, compare to 20, and let it expire. State is one integer per active window, the arithmetic is trivial, and old windows disappear on their own. That cheapness is why the design keeps it for anonymous traffic.

The weakness is the boundary. Because the counter resets at an arbitrary line in time, a caller can spend the full limit just before it and the full limit just after.

Whether that matters depends on what the limit protects. For a coarse anonymous rule meant to slow scrapers, a 2× spike at a boundary is tolerable and the saved state is worth it. For P-READ, which exists to keep a user at a sustainable average, it is the wrong shape — chapter 7 replaces it. The journey's boundary-escape scene (scene 7) steps through these exact timestamps.

Two cheaper repairs exist before reaching for a different algorithm. A sliding-window counter keeps the previous and current window counts and weights the previous one by how much of it still overlaps the last 10 seconds: at t = 10.1 s the estimate is 20 × (1 − 0.1 / 10) + 0 = 19.8, so the second burst of 20 is refused outright where the fixed window admitted all of it. The price is an assumption that the previous window was evenly spread, so the result is an estimate, not a count. A sliding log is exact but stores one entry per admitted request. Both close the boundary hole. Neither says how much burst a client is entitled to, and neither lets a request cost more than one unit, which is the question the next chapters take up.

The limit has to travel with the identity, not with the process that happened to receive the request. Add a second gateway and the V0 counter breaks in a way nobody sees on a dashboard.

Traffic grows, so Northstar puts gateways A and B behind a load balancer. K1, Atlas's API key, now spreads across both. Each gateway has its own counter and its own view of the world. Suppose the limit for K1 in the current window is 20, and traffic has been landing evenly, so that each gateway has locally counted 19. From each gateway's point of view exactly one slot remains.

Both gateways are doing their job and neither is buggy. The policy simply cannot be expressed as a collection of independent local counters. Pinning each key to one gateway (sticky routing) would shrink the problem, but it ties capacity and failure domains to key placement and still breaks on gateway failure and rebalancing. The decision needs one authority that every gateway consults for the same key. Journey scenes 8 and 9 replay this timeline with distinct requests.

The shared store is introduced to give both gateways the same answer. Sharing alone is not enough: how the check is evaluated matters as much as where.

Move the counters into Redis, called R here. The first attempt most teams write is read, compare, write: GET the count, if it is below the limit SET count+1. Two gateways interleave and the same mistake appears again, only now across a network.

The repair is to make read, compute and write one indivisible step on R. A Lua script (or a server-side Redis Function) runs to completion on a single Redis instance without other commands interleaving; while it runs, other clients wait. That is why scripts should be short and never loop over unbounded data. Redis scripting works this way. What that buys is exactly one thing: on that instance, two requests cannot both consume the same remaining budget.

Be careful about how much that atomicity covers. It is atomic with respect to one Redis instance. It is not, by itself, durable (replication is asynchronous by default, so a failover can lose recently acknowledged writes — chapter 13), and it says nothing about a second region (chapters 14–15). The guarantee is narrow and real: a single atomic authority cannot double-admit against the same remaining budget.

Cost, honestly: each decision is now a network round trip to R. The design goal is a p99 admission latency in the low single-digit milliseconds within a region — a target to measure on your own network, not a benchmark. Pipelining, connection pooling and keeping the script small are how that budget is protected. Scenes 10, 11 and 12 show the shared store appearing, the naive race, and the atomic script resolving it.

The fixed window's boundary flaw and its blindness to burst shape are what push per-user limits towards a token bucket.

A token bucket holds up to a capacity of tokens and gains tokens at a constant refill rate. Each request needs its cost in tokens; if enough are present they are removed and the request is allowed, otherwise it is denied. For P-READ the capacity is 20, the refill rate 2 tokens/s and the cost 1. A user who has been idle holds the full 20, so a burst of 20 is allowed immediately, which is the point: real clients are bursty and a burst allowance is friendlier than a flat per-second ceiling.

What a bucket does not promise

Over any span of T seconds, a bucket can admit up to 20 + 2T requests: 40 in 10 s, for instance, where a strict 20-per-10-seconds sliding window would have allowed 20. The bucket enforces a burst and a long-run average, and it is not a strict sliding-window ceiling. If a contract genuinely says no more than 20 in any 10-second window, a bucket with those parameters is the wrong implementation. The other candidates are cheaper or costlier in different ways: a sliding log keeps every timestamp, so Atlas at 1,000 per minute means up to 1,000 entries for one key and 20,000 active users × 20 entries is 400,000; a sliding-window counter estimates prev × (1 − elapsed/W) + curr — with W = 10 s, prev = 18, curr = 5, elapsed = 2.5 s that is 18 × 0.75 + 5 = 18.5, so admitting one more gives 19.5 ≤ 20 and is allowed — an approximation that assumes the previous window was evenly spread, never an exact count. The full comparison lives in the Architecture Deep Dive.

So why does P-READ get a bucket? Its contract is a sustainable average with room for the bursts real clients make, such as a catalog page that fires 20 calls on load and then goes quiet. A bucket states that in two numbers (capacity 20, refill 2 per second), stores two fields per user, charges a request any number of tokens, and can compute the exact wait for a denial. A sliding-window counter is the better fit when the promise is literally "no more than N in any W seconds", for example a contractual ceiling or an abuse rule that must not allow the 20 + 2T spread. The design therefore keeps a bucket for P-READ and a coarse fixed window for P-ANON, and treats the sliding-window counter as the stricter option when a contract requires it.

Doing it atomically

The refill is computed at request time from elapsed time rather than by a background timer, so no process has to tick for every key. That makes the bucket one small state record — tokens and a last-updated timestamp — updated by one script. Two details prevent subtle bugs. Tokens are stored as integers in millitokens, because floating point drifts: a 300 ms refill at 2 tokens/s is 600 millitokens, exactly. And the timestamp comes from one clock; letting each gateway pass its own now makes the bucket depend on clock skew between machines.

A worked example with a different cost: an export costs 10 tokens, charged to a bucket shaped like P-READ's, holding 3 tokens. The deficit is 7, and 7 tokens at 2/s is 3.5 s, which rounds up to a retryAfter of 4 s — integer seconds, rounded up so the hint is never too early. That hint is a prediction from current state, not a reservation: other requests from the same bucket in the meantime can use the tokens first. Scenes 13 to 18 compare the algorithms and settle on this choice.

An atomic counter answers “how many?” correctly for whatever key it is given. Choosing the key — who counts as one caller — is where fairness and security are won or lost.

Northstar's keys follow a convention such as rl:v3:{t:atlas}:u:U-17:read: a namespace, a policy version, the tenant in braces, the principal type and identifier, and a route group. The identifier of a user or API key is hashed in real systems so raw keys and IPs never appear in logs or store dumps. The idle TTL is the time to refill fully plus slack, and it is set in the same script that writes the key, so state cannot outlive its usefulness. The braces matter later: they are a hash tag, and chapter 9 and 12 depend on them.

Choosing the principal

The anonymous case is the awkward one. P-ANON allows 20 requests per 10 s per IP, but two hundred people on one campus network share that IP and are all throttled by one noisy neighbor, while an attacker with a pool of addresses sidesteps it entirely. That is not a reason to drop the rule — it is a reason to keep it coarse, generous, limited to the pre-authentication surface, and to move to a verified identity as soon as one exists. Scene 20 shows the NAT unfairness case.

Trust and ordering

The client IP is only trustworthy if it comes from a proxy chain you control. A raw X-Forwarded-For value is whatever the caller wrote; the gateway should take the address appended by its own trusted edge, and ignore the rest. The same applies to tenant and user identity: derive them from an authenticated token's claims, never from a header or body field the client fills in (scene 19).

That produces a two-stage pipeline. A cheap edge or gateway limit by IP runs first, before authentication, so unauthenticated floods and credential-stuffing attempts cannot burn expensive verification work or limiter state at will. After authentication, the principal and tenant limits apply. Two consequences follow: login endpoints need their own pre-auth rule keyed on IP and on a hashed target username, since no authenticated user exists yet; and responses to unauthenticated callers should not reveal tenant quotas or predictable reset times, which would be an information leak. A request that fails authentication gets 401 or 403, not a quota answer.

Some traffic must not be limited like everyone else: load-balancer health probes and a small set of privileged internal operations. The safe form is a narrow, separately authenticated and audited route with its own small cap, not a general “skip the limiter” flag. A bypass that is not itself metered becomes the unmonitored path that floods your backend later.

A real request is subject to more than one limit at once, and all of them must say yes.

A GET /catalog from U-17 of Atlas is governed by the fleet-wide protection P-GLOBAL, by Atlas's tenant quota P-TENANT, and by U-17's own P-READ bucket. The verdict is the conjunction: allowed only if every applicable rule has room.

The “debited?” column is the engineering problem. If the tenant counter is charged and the user check then fails, the tenant has paid for a request that was never admitted, and a refund can itself fail, so counters drift in the safe direction until they are wrong. The clean answer is to evaluate every rule first and then debit all of them, inside one atomic step. That is possible only when all the keys live on one shard. In Redis Cluster, a script touching keys in different hash slots is rejected with a CROSSSLOT error. Hash tags — the {t:atlas} in the key — place all of a tenant's keys in the same slot, so tenant, user and route buckets for Atlas can be decided together. Redis Cluster defines these slot and hash-tag rules.

The price is that a tenant's keys now share a slot, so a very hot tenant makes a hot slot (chapter 12). P-GLOBAL cannot join that atomic step at all: it is shared by every tenant, so it lives elsewhere. It is handled either by a separate check with compensation or, in the mature design, by token chunks leased to gateways. Doing five checks in five different stores would cost 5 × 500,000 = 2.5M store interactions per second at peak; the same five tenant-scoped rules in one same-shard script cost 500,000 script calls per second, one round trip each. Rule precedence also needs a fixed order, so that the reason returned to clients and logs is deterministic: global protection, then tenant, then user, then route. Scene 21 shows these limits applied, and scene 22 the partial-debit and cross-slot trap.

Counting requests treats a catalog read and a report export as equals. Backends do not.

An export costs about ten times a read. If P-TENANT counted both as one request, Atlas could consume a thousand exports per minute while the backend, sized for 2,000 cheap reads per second, falls over. The first fix is weighted cost: meter in units where a read is 1 and an export is 10. The bucket mechanics stay identical — the request needs 10 tokens instead of 1 — and a cost larger than capacity is rejected at policy-validation time, not at runtime.

Weights cap how fast expensive work can be started, but not how much of it is running at the same time. Three exports that each take a minute to run can be started well within any reasonable rate and still occupy a worker pool. So P-EXPORT adds a second, separate mechanism: at most 3 in flight per tenant.

A concurrency limiter has a failure that a rate limiter does not: a slot taken by a worker that crashes is never given back. The standard defense is to treat each slot as a lease with a deadline — for example a sorted set whose members are request IDs and whose scores are expiry times — acquire it with an atomic script that first drops expired members, and renew or release it explicitly. Slot lifetimes must exceed the longest legitimate export, or you free a slot while its work is still running. A fourth concurrent export is denied even when the request-rate budget is untouched (scene 24). For some workloads queuing the fourth request, or turning exports into asynchronous jobs, is a better answer than denying; that is a different component with its own memory bound, and not the limiter. Scene 23 shows the 10-unit charge, and the downstream bulkhead lesson covers isolation of the worker pool itself.

A denial is an API response, and clients react to it. A good one tells them what happened and when to try again without making things worse.

When a valid caller exceeds policy the response is HTTP 429 Too Many Requests, defined in RFC 6585. It may carry a Retry-After header, whose syntax in RFC 9110 is either a number of seconds or an HTTP date. In this design it is the integer-seconds value from the bucket arithmetic (the 4 s export example in chapter 7). It is a hint based on current state, not a reservation: the capacity it points to may be taken by someone else, so a client that retries at exactly that moment can be denied again.

There is also the question of advertising limits. Fields named RateLimit and RateLimit-Policy, and older RateLimit-* forms, are being specified in the IETF Internet-Draft draft-ietf-httpapi-ratelimit-headers. It is a draft — a work-in-progress document, not an RFC — and as of this writing it is still treated that way, so confirm its current status on the IETF datatracker. The widely used X-RateLimit-* names are a convention without a standard behind them. If you expose remaining quota, remember that doing so to unauthenticated callers leaks plan details and predictable reset points.

What clients do next

If 10,000 clients are each told Retry-After: 1, a polite client retries one second later — and all 10,000 do it in the same instant, a spike aimed at the thing that just said it was busy. With jittered exponential backoff, or with the server adding some jitter to the hint, the same 10,000 retries spread over roughly one to two seconds, averaging about 10,000 per second instead of arriving as one wall. Scene 25 shows the response and backoff. The Core Concepts pages on retry and backoff and the production module on the retry storm go further on the client side. Remember also that an allowed request is not a successful one: a downstream service can still return its own 429 or 503 after the gateway admitted the call, so not every 429 a client sees came from this limiter.

One Redis primary cannot carry 500,000 decisions per second, and even a fleet of them cannot carry a single key that everybody wants.

Take a hypothetical planning budget of 40,000 script executions per second per primary — a figure to replace with your own measurement, not a benchmark. At the 10× surge of 500,000 admission calls per second, 500,000 / 40,000 = 12.5, so at least 13 shards; plan 20. With uniform hashing each shard then averages 25,000 per second, 62.5% of the budget, leaving room for imbalance and replication. Keys are placed by hash slot, in the spirit of consistent hashing, so that adding shards moves only some of the key space.

Memory is a separate calculation. With 2M active keys and an assumed 200 bytes per key (a small hash, key name and overhead, to be confirmed with MEMORY USAGE on real data), 2,000,000 × 200 B = 400 MB. Allow 2× headroom for allocator fragmentation and replication buffers, so about 800 MB per copy and roughly 1.6 GB for a primary and one replica. Since eviction can silently delete counters — and a deleted counter is a counter reset — the store runs with TTLs on every key and an eviction policy chosen deliberately, with memory alarms before it matters. Redis's own production guidance covers the operational side.

Average balance does not protect against skew. Atlas's flood attempts 75,000 requests per second, 15% of peak, all against one tenant's key set and therefore one shard — above the 40,000 per second budget. Note that the limiter's load is attempts, not admits: most of those requests are being denied, but each denial is still a script execution. Two mitigations, aimed at different cases.

Neither mitigation changes the correctness rule inside a shard; they reduce how often the shard is consulted. Scenes 26 and 27 show the peak, the hot key, and the cardinality case in which an attacker's fake principals — keyed before authentication — fill memory, which the TTL and a cardinality cap are there to bound.

The limiter sits in front of every request. When it cannot decide, the gateway must still answer, and “fail open” and “fail closed” are not one policy but a per-endpoint choice.

A timeout to R has two meanings that must be kept distinct: no authority is available, and the authority's data is stale. The gateway cannot tell from the timeout alone whether the script ran. Three fallback behaviors are available, and they are chosen by what a wrong answer costs.

The bounded local fallback is the nuanced one. When R is unreachable, each gateway enforces a conservative local cap, derived so the whole fleet cannot exceed the target. For P-READ with two gateways, a local bucket per gateway of capacity 10 and 1 token/s totals at most 20 and 2 per second if traffic splits evenly, and under-admits if it does not. The honest label is that this is a reduced guarantee: the limit is approximate until R returns. “Fail open” on its own would preserve nothing of the limit; it should never be described as preserving it. Scenes 28 to 30 step through the timeout and the two outcomes. The circuit breaker pattern is how the gateway stops waiting on a dead store, and tight timeouts and bounded connection pools stop the limiter's own slowness from becoming the outage (see also cascading failure).

Failover and the ambiguous write

Redis replication to a replica R2 is asynchronous by default. If R fails after acknowledging an admission that R2 had not yet received, promoting R2 forgets that admission, and the counter goes backwards. WAIT can narrow that window but does not turn the pair into a strongly consistent store. A restore from RDB or AOF persistence can likewise return counters to an older state. For a token bucket the consequence is bounded: a reverted bucket can hold at most its capacity, so the extra admissions are at most 20 per affected P-READ bucket. A fixed-window counter that reverts can re-admit up to its limit again within the affected window. Scene 31 shows the lost write.

The other ambiguity is a gateway that timed out and does not know whether the script ran. A blind retry can charge twice. The remedy is to attach a stable logical request ID and have the script remember recently seen IDs for a bounded time, so a replay returns the earlier decision instead of debiting again. The retention window must be short and finite, which bounds both memory and the risk of ID collisions. The related Core Concepts lesson on idempotency covers the principle.

Everything so far assumed one atomic authority a short network hop away. Add a second region and that assumption becomes a decision.

Atlas sends traffic into both EU and US. Its plan says 1,000 requests per minute globally; call that G. Suppose each region simply runs its own limiter with the full limit. EU admits up to 1,000 and US admits up to 1,000, so Atlas can get up to 2,000 per minute — twice the contract (scenes 32 and 33). Regional independence is fast and available, and it does not honor a global number.

The strict answer is a single home-region authority for each tenant's global counter. It restores the single-authority guarantee from chapter 6 — no over-admission — at two costs. The remote region adds a WAN round trip to every decision; illustratively 80 to 120 ms, a hypothetical figure that sits badly beside a low-single-digit-millisecond goal. And when the link is partitioned, the remote region cannot admit at all, or must decide on a weaker rule. That is the unavoidable trade: zero overshoot and unrestricted availability through a partition cannot both be promised.

Home-region authority suits quotas with legal or financial consequences and tenants whose traffic is concentrated in one region. Leases suit high-volume, latency-sensitive paths where a small stated overshoot is cheaper than WAN latency. Envoy's global rate limiting is an example of the related idea of a central rate limit service that proxies consult, alongside local rate limiting, which is the same trade expressed for service proxies.

A lease turns a global promise into a local one with a visible edge: a region may spend only what it has been explicitly granted, until the grant expires.

A global allocator holds the pool for the current minute: 1,000 tokens for Atlas. Regions ask for chunks of 100. Each grant has a 15-second TTL and is clamped to the end of the window, so tokens granted near the boundary cannot be spent in the next minute's budget. The allocator never grants more than remains in the pool at the instant of the grant. Inside a region, gateways spend the leased tokens through their regional store as before, with no cross-region call on the hot path. When a region's chunk runs low it asks for another, in proportion to its recent demand.

Lease depletion

A fixed split looks simpler and fails visibly. Divide G 50/50 in advance: 500 each. If EU demand is 900 and US demand is 100, EU is denied 400 requests while US leaves 400 tokens unused, even though globally 400 remained. Demand-weighted chunk renewal repairs this, because EU can keep asking for chunks while the pool lasts. The residual unfairness is the time between asking and receiving — scene 36 shows both cases.

Partition and overshoot

Now cut the link between US and the allocator while US holds 200 unspent tokens. US can admit at most those 200 until they expire, and receives nothing new. If leases are clamped to the window, grants never exceed the remaining pool, and US has no local fallback, the overshoot is 0: the global total cannot exceed G. The cost is stranded capacity — up to 200 tokens, 20% of G, that legitimate US callers are denied. That is the trade: safety bought with possible unfairness, which is why the lease size and TTL are tuning knobs and not constants.

The overshoot bound in general is explicit rather than zero, and stated with its assumptions:

Scene 35 shows the leases being granted, scene 37 the partition, and scene 38 the reconciliation of actual allowances against the bound. The allocator is the one place cross-region coordination happens, and it is off the request path: it only sees one chunk request per region per chunk-sized consumption.

Everything above runs on policy: numbers, scopes, costs, fail modes. Policy changes need the same care as code.

Policies are authored in a control plane with role-based access, validation (no cost above capacity, no refill of 0 without a documented reason), a staged rollout and an audit history. What gateways receive is an immutable, versioned snapshot, applied atomically, so a request is evaluated under one coherent policy version, recorded on the decision (scene 39). A gateway that receives an older or malformed snapshot rejects it and keeps the version it has; if it has been disconnected too long it applies a declared behavior rather than inventing one (scene 40). Instant global revocation during a partition is not promised: a tightening policy reaches connected gateways within the propagation time, and disconnected ones keep the last version they accepted. The version in the key, v3 in rl:v3:…, has a side effect worth knowing: bumping it creates fresh, full buckets, which can briefly loosen enforcement during a rollout.

A new or tightened limit should reach enforcement in stages. Shadow mode (also called log-only or dry-run) evaluates the new rule beside the live one, against its own counters under a new key version so it cannot debit real buckets, and records a wouldDeny flag on the decision while the request is still admitted. Measure before enforcing: which tenants and routes the rule would have refused, whether well-behaved callers such as Birch appear in that list, and how the count compares with what the policy author predicted. Then enforce gradually, starting with one route or a few tenants and widening while watching deny rate, retry traffic and support contacts, with the previous policy version one revert away. Treat shadow results as an estimate rather than a forecast: clients that have never received a 429 have not yet started to back off, so real enforcement changes the traffic it measures. Shadow evaluation also costs extra store calls, so run it for the rules being changed rather than for the whole fleet.

Observability tells you whether the design is doing what it claims. The signals worth alerting on are per-rule deny rates, p99 admission latency, store error and timeout rates, fail-mode engagement, policy version skew between gateways, store memory and evictions, replication lag, lease exhaustion by region, and top keys by attempts. Telemetry here is not a ledger: log pipelines drop and duplicate events, so counts derived from them are fine for dashboards and wrong for billing or contractual quotas, which need a stronger accounting path. Decision logs also carry privacy and cardinality cost, so they hash principals and sample (scene 41).

Security closes the loop. Administration is least-privilege and audited. Tenants are separated in key naming so a collision cannot cross-contaminate counters. The exempt route from chapter 8 is audited and capped. And the limiter is one layer among several: it does not replace authorization, bot defense, DDoS protection or client discipline. Scene 42 replays the original Atlas flood and shows Birch staying healthy.

The final architecture is the sum of the failures above. Each box is there because an earlier version broke.

Request flow, in order: the edge applies cheap abuse protection; a gateway authenticates and derives the tenant and principal; it reads its local policy snapshot; the admission step consults the regional store atomically for all tenant-scoped rules and consults its leased chunk for global rules; the request is forwarded or denied with a 429 and guidance; telemetry is emitted off the decision path. The allocator and control plane are asynchronous with respect to that flow, and the picture deliberately does not imply that the allocator shares a transaction with every regional store.

Mental model

Identity first, then policy, then an atomic or explicitly bounded admission decision, then feedback the client can act on safely, then monitoring that tells the truth. Scene 43 shows the assembled architecture, and scene 44 the same model as an interview whiteboard.

Guarantees, with their authority and scope