Rate Limiter Tradeoffs: Algorithms, Redis, Fail Modes & Multi-Region

Every design in the other pages of this module is the survivor of a comparison. This page reopens those comparisons: for each of 25 choices it names the incident that forces the question, the realistic alternatives, what each one costs, what Northstar picked and the circumstances under which a different team should pick differently.

The running example is fictional. Northstar API serves GET /catalog, POST /checkout, POST /login, POST /reports/export and POST /search; Atlas is a large partner tenant that floods, Birch is a well-behaved one that must stay healthy; Atlas's API key is K1, one of its users is U-17; requests enter through gateways A and B, whose limiter state lives on Redis primary R with replica R2, in regions EU and US. Every capacity, rate, latency and memory figure on this page is a hypothetical teaching input chosen so the arithmetic can be checked by hand — none is a benchmark, a vendor measurement or an industry statistic. Where a number matters it is the same number used in Complete Design and Architecture Deep Dive, which is where the mechanisms are built; this page assumes you have met them and argues about the alternatives.

The policies the decisions refer to

Five policies recur. P-ANON limits an anonymous IP address to 20 requests per 10 seconds in a fixed window aligned to multiples of 10 s. P-READ is a per-user token bucket with capacity 20 and a refill of 2 tokens per second, cost 1, applied to GET /catalog. P-TENANT gives Atlas 1,000 requests per minute across all of its identities; other plans differ. P-EXPORT charges POST /reports/export a weighted cost of 10 units *and* allows at most 3 exports in flight per tenant. P-GLOBAL protects GET /catalog fleet-wide at 40,000 requests per second, shared by 100 gateways. Load is assumed at 50,000 admission calls per second at baseline and 500,000 at a 10× surge.

The question to keep in mind throughout is the one the limiter exists to answer: may this request spend backend capacity now? Each decision is really about the failure mode you are choosing to live with — too many requests admitted, too many refused, or too much machinery between the caller and the answer. A rate limiter built from the other pages' reference design will behave well in the scenarios there; the decisions here are what you re-derive when your numbers, your contracts or your failure tolerance are not Northstar's. For the plain algorithm fundamentals see the Core Concepts lesson on rate limiting.

The first three decisions are about what “20 per 10 seconds” means once time is involved. The algorithms differ in whether a burst can slip across a boundary, in how much state each caller costs, and in how truthfully the answer can be explained to the person who just got a 429.

Decision 1 · Fixed window or sliding window?

Picture a script hammering POST /login from a single IP. P-ANON allows 20 requests per 10 s in a fixed window that resets on epoch multiples of 10. The script sends 20 requests at t = 9.9 s, all admitted, then 20 more at t = 10.1 s, all admitted again because the counter just reset: 40 requests in 0.2 s, twice the limit. Nothing was broken; that is the fixed window working exactly as defined, and a fixed window is the only algorithm in this module for which that burst is not a bug.

Three options were on the table. A fixed window keeps one integer and one expiry per IP: INCR, set the TTL when the count is 1, compare. A sliding window counter keeps two integers (the current and previous window) and estimates the rolling count; with W = 10 s, limit 20, 18 requests last window, 5 so far in this one and 2.5 s elapsed, the estimate is 18 × 0.75 + 5 = 18.5, and admitting one more gives 19.5 ≤ 20, so it allows. A sliding log stores every timestamp and is exact. The sliding variants remove the boundary doubling but cost more state, more arithmetic per call, and explanations that are harder to give: “you used 18.5 requests” is not a sentence a support engineer wants to defend.

For P-ANON Northstar keeps the fixed window. The rule only has to stop the front door being used as a cheap way to create state and burn CPU; a worst case of 2×L for one 0.2 s spike is tolerable when the backend behind it can absorb 40 login attempts, and the state is as small as state gets — one key per active IP with a TTL of window plus slack. Authenticated traffic, where fairness and burst shape actually matter, moves to token buckets (Decision 2).

A sliding window wins when the contract says “no more than N in *any* rolling interval”, for instance a downstream partner API that bans callers who exceed a rolling-minute ceiling. The operational consequence of choosing fixed windows is synchronized behavior: every client's window resets at the same instant, so a Retry-After computed as “seconds to the next boundary” sends all rejected clients back together. Spread the hint (Decision 20) or the reset itself becomes a small recurring spike.

Decision 2 · Token bucket, leaky bucket or GCRA?

Birch's mobile app opens the catalog screen and fires 12 GET /catalog calls at once, then goes quiet for a minute. A limit that rejects on the strictest reading of “2 requests per second” would refuse 10 of those, and Birch would see a perfectly healthy client misbehave. The decision is whether the limiter should tolerate bursts, flatten them, or be indifferent as long as the long-run rate holds.

The token bucket stores a number of tokens, refills it at a steady rate up to a cap and spends tokens per request. P-READ uses capacity 20 and refill 2 per second: an idle user holds 20 tokens, can send all 20 at t = 0, and the 21st is denied with retryAfter = ceil(1/2) = 1 s. Ten seconds idle restores the full 20, and five seconds restores 10. The leaky bucket has two readings: as a *queue* it releases work at a fixed rate and therefore delays, and as a *meter* it admits a request only if the bucket would not overflow, which behaves like a token bucket with a different bookkeeping. GCRA (the Generic Cell Rate Algorithm) is an equivalent formulation of the leaky bucket as a meter that keeps one value per key, the theoretical arrival time of the next conforming request, so state is a single timestamp.

Northstar chose the token bucket for rate limiting because its burst parameter is a first-class, readable number: capacity 20 says “a user may spend up to 20 at once”, and the denied response can be explained in the vocabulary of tokens. The price is worth stating precisely: over any T seconds a P-READ bucket can admit up to 20 + 2T requests, for instance 40 in 10 s, so it is not a strict sliding-window ceiling of 20 per 10 s. If the backend breaks at an instantaneous rate rather than a long-run average, shrink the capacity or pick a different tool.

A leaky bucket in its queue form wins when the *downstream* needs a smooth input, such as a payment processor with a fixed throughput contract, and delay is acceptable. GCRA wins when the smallest possible state matters and strict spacing is the goal, but its stored timestamp is harder for operators to read than “7 tokens left”. Operationally, the token bucket's fractional refill pushes you towards integer fixed-point storage — 2 tokens per second over 300 ms is 600 millitokens — so the script never accumulates floating-point drift.

Decision 3 · Exact sliding log or approximate weighted counter?

Suppose Atlas's contract says: no more than 1,000 requests in any rolling minute, enforced strictly, because Atlas resells that capacity and disputes overage. Token-bucket semantics will not satisfy that sentence literally, so the question is how to build a true rolling count. This is a comparison of the two sliding approaches, not an alternative to the primary design: Northstar's hot path stays on token buckets.

The sliding log keeps one entry per admitted request and counts those inside the window. It is exact, and its memory is O(requests in the window): up to 1,000 entries for Atlas's tenant key, but for P-READ-sized rules across 20,000 active users × 20 entries that is 400,000 entries, each one a timestamp to write, expire and scan. Denied requests are not recorded, so a flood does not inflate it. The weighted counter keeps two integers per key regardless of volume; its estimate assumes the previous window's requests were spread evenly. If all 18 of the previous window's requests actually arrived in its final second, the true rolling count at 2.5 s into the new window is 23 while the estimate still says 18.5 and admits the next request. The error is bounded and usually small, but it is real, and it is never described as exact.

The rule of thumb that falls out: use the log where the key count is small and exactness has contractual weight — a few hundred tenants with a 1,000-per-minute rolling clause is a few hundred thousand entries — and the weighted counter where keys are numerous and a few percent of error is acceptable. In both cases, size memory from *active keys × entries*, not from request rate. The operational cost of the log is a script whose work grows with the window's contents, which matters because a Redis script blocks other commands on that shard while it runs.

Counting correctly on one machine is easy. The moment two gateways serve the same caller, the question becomes who owns the number, and what “owns” has to mean when two requests arrive in the same microsecond.

Decision 4 · In-process counters or a Redis-backed shared limiter?

Gateways A and B each keep a local P-READ bucket for U-17. A load balancer sends U-17's 20 rapid requests half to A and half to B; each gateway sees 10, finds plenty of tokens and admits everything. In general, with N gateways and a local limit L the fleet admits up to N × L, and the factor depends on how the balancer happens to spread the caller. Splitting the limit evenly (10 each) repairs the worst case but under-admits whenever the balancer is uneven: U-17 pinned to A is held to 10 although entitled to 20.

The options are an in-process limiter (zero network hops, accuracy proportional to how sticky routing is), a shared Redis authority (one round trip, exact per key within one store) and a hybrid where the gateway spends locally and syncs. Northstar uses Redis for P-READ, P-TENANT and P-EXPORT because their whole purpose is a number that does not depend on which gateway answered. Local logic is not discarded: it reappears as the negative cache that absorbs denial floods (Decision 11), as leased chunks for P-GLOBAL, and as the bounded fallback when R is unreachable (Decision 16).

In-process wins in three honest cases: a single instance; a coarse abuse guard where an error factor of N is tolerable; and any protection of the process itself, such as a cap on its own concurrent connections. The operational consequence of choosing Redis is that the limiter becomes a dependency with its own availability, timeout and pool settings. The design goal of a low single-digit-millisecond p99 for the decision is something to measure in your own environment, not a promise inherited from this page. See rate limiting for the single-process baseline.

Decision 5 · Redis atomic script or a database conditional update?

The incident that motivates this decision is the check-then-write race: two gateways read “tokens = 1”, both decide to spend it, both write 0, and two requests were admitted on one token. Any design has to make read, decide and write one indivisible step. There are three ways to get there.

A Redis Lua script or function runs on the shard that owns the key, to completion, without interleaving other commands; it reads the bucket, refills it, compares, debits, sets the TTL and returns allow, remaining and retryAfter in one round trip. A database conditional update — UPDATE quota SET remaining = remaining - :cost WHERE key = :k AND remaining >= :cost — achieves the same atomicity through row locking and returns the affected-row count. Optimistic transactions (watch the key, queue commands, retry on conflict) avoid long scripts but retry under contention, which is exactly the situation a hot limiter key is always in.

For per-request admission Northstar uses the script: the work is a handful of arithmetic operations, so it holds the shard for microseconds rather than milliseconds, and the multi-rule version (Decision 9) can check every applicable bucket in one call. Its limits are real: a script blocks the shard while it runs, so it must stay short; the script cache is volatile, so clients should be ready to reload it after a restart or failover; and atomic is not durable, a point Decision 24 returns to.

The database wins when the quota is a *balance* with financial meaning — say a prepaid bundle of 1,000,000 API calls per month for a tenant, decremented in the same transaction as the billing record. There the durability and auditability of a relational commit outweigh its latency, and rates are low enough that row contention is manageable. A hot tenant row under a database lock is the failure to watch; see transactions and concurrency. Operationally the choice splits one product into two systems: fast admission state that may be lost in a failover, and slow accounting state that may not.

Decision 6 · Where in the request path should the limit be enforced?

Suppose Atlas's key K1 sends 5,000 requests per second (a hypothetical flood). If the check runs inside the checkout service, every request has already crossed the CDN, the gateway, authentication and a connection to the service before it is refused. If it runs at the CDN, a refusal is cheap, but the CDN sees an address and headers: it cannot tell U-17 from the rest of Atlas or charge an export ten units. Placement decides what identity the limiter can see, how early a refusal happens, and which paths can skip the check entirely.

Northstar enforces at the gateway. It is the first point where tenant, user and route are verified, one fleet shares one policy snapshot and one fallback behavior, and a refusal costs only a store round trip. Two additions cover its gaps without replacing it. A coarse edge rule on source address, in the style of P-ANON, absorbs volumetric noise before it uses gateway capacity, and the origin accepts connections only through the edge and gateway so neither can be skipped. Each service also protects itself with its own concurrency cap (see Decision 4), because a gateway cannot see the work a single request fans out into.

The recommendation changes with the topology. If most traffic is service-to-service and never crosses a gateway, a sidecar or shared library is the only place that sees it. If a request's cost is known only after work has started (Decision 25), the service has to take part in the decision. If an organization has several ingress paths, send them all to one shared limiter so a tenant's quota is not multiplied by the number of paths. The usual result is a coarse outer layer plus a precise inner one, and the price is two policies that must agree, so state which layer is authoritative for each rule.

A counter is only as meaningful as the key it hangs from. The next three decisions settle whose quota a request spends, how little you can trust an IP address, and how several limits are evaluated for one call without leaving half-spent state behind.

Decision 7 · Global, tenant or principal quota keys?

Atlas's integration, using key K1, starts a nightly sync with 50 worker users all hitting GET /catalog. Each user stays inside P-READ's 2 tokens per second, so no per-user bucket ever denies. Together the workers can still produce up to 100 requests per second, which is 6,000 per minute against a plan that says Atlas may have 1,000. A limit keyed only on the principal cannot see the tenant, so Atlas's plan is not enforced at all.

The reverse failure is just as real. If the only rule is P-TENANT's 1,000 per minute, one runaway user such as U-17 can burn the whole minute's allowance in 10 seconds at 100 requests per second, and every other Atlas user is then denied until the window refills. Keying on the tenant protects Northstar and the plan; it does not protect Atlas's users from each other. A purely global key per endpoint is the third option and fails the second test worst: Atlas's flood can starve Birch, who never did anything wrong.

Northstar evaluates all three levels together — principal, tenant, and a fleet-wide guard for the shared catalog (P-GLOBAL) — and the first denying rule is named in the response so each caller learns only about the scope it can act on. A tenant-only design wins for business-to-business APIs where the tenant is solely responsible for how its users share the allowance; a principal-only design wins for consumer products with no tenant concept; a global-only design is a reasonable emergency brake when nothing finer exists yet. The cost of the layered design is operational: more keys per request, more policy to author, and the need to publish *which* rule denied so that an engineer at Birch is not told the answer is “your tenant is over quota”.

Decision 8 · IP-based anonymous limits or verified identity?

POST /login and the public parts of the catalog are reached by callers Northstar cannot yet identify. P-ANON keys on IP: 20 requests per 10 s. Then 200 students on one campus network, all NATed behind one public address, share that allowance: a hypothetical 20 per 10 s divided among 200 people is one request every 100 seconds each, and the first few to log in in the morning use it all. An attacker has the opposite experience, because rotating through a pool of addresses costs little and each address arrives with a fresh budget.

Options: key on IP alone; key on a verified identity (impossible before authentication, which is why anonymous routes exist); or layer them — a deliberately coarse IP rule at the front door, then identity-keyed limits after the caller has authenticated. Northstar layers. The IP rule exists to stop the edge being used as a cheap way to create limiter state and burn CPU, so failed authentications count against the source IP budget and never against the tenant the attacker claims to be; otherwise a stranger who guesses Atlas's key id could drain Atlas's quota. Forwarded-for headers are trusted only when they were appended by a proxy the gateway controls, because a client can send any value it likes.

IP-only limits win for simple public endpoints with no accounts and for volumetric defense in front of the app. A per-target limit — failures per username, say — is a further option for credential stuffing and complements IP limits, at the price of a new place where one attacker can lock out a victim. Operationally, treat any IP rule as a defense for the front door rather than a fairness mechanism, give known corporate or carrier ranges an explicit, reviewed policy rather than a bypass, and avoid revealing quotas or predictable reset times to unauthenticated callers.

Decision 9 · A single atomic key or a hierarchical multi-key policy?

U-17 calls GET /catalog. Five rules apply: user, route, tenant, plan and one more guard, say. If each rule lives in its own store and is checked in turn, the second rule can deny after the first has already been debited, so U-17 loses a token for a request that was never served — a *partial debit*. Repairing it needs a refund, and an ambiguous refund after a timeout is how quotas get double-credited.

There are three approaches: collapse everything into one composite key (simple, but you can no longer tell a user's limit from a tenant's); evaluate sequentially with compensation (works only with idempotent undo, which a token refund is not); or put every applicable bucket on the same shard and decide in one script that reads all of them and writes only if all allow. A denial anywhere debits nothing. The arithmetic of cost points the same way: at 500,000 admission calls per second, five rules in five stores are 5 × 500,000 = 2.5 million store interactions per second, while the same five in one same-shard script are 500,000 calls, one round trip each.

Northstar uses the single-script form with the tenant hash tag {t:atlas} in every key, so the user bucket, tenant bucket and route budget share a Redis Cluster slot and avoid a CROSSSLOT error. A single composite key is fine for a route with exactly one rule — a health probe, or a plain anonymous IP limit. Sequential-with-compensation survives where the rules *must* live in different systems, and then the compensation must be built on an idempotent operation such as releasing a lease, not on handing tokens back. The operational price of colocation is that a heavy tenant becomes a heavy slot — the subject of Decision 10.

Adding shards divides load only when load is made of many keys. The limiter's hardest scaling problem is the opposite shape: one key that attracts a disproportionate share of traffic, including traffic that will be refused.

Decision 10 · Colocate a tenant's keys or spread the key space?

Capacity planning first. Assume a planning budget of 40,000 script executions per second per primary, a hypothetical figure that you must replace with a load test of your own. At 500,000 calls/s at the surge peak, 500,000 / 40,000 = 12.5, so at least 13 shards; planning 20 gives an average of 25,000 per shard, 62.5 % of the budget. That only holds if hashing spreads traffic evenly, and the hash-tag choice decides whether it can.

With tenant colocation every key of Atlas contains {t:atlas}, lands in one slot and so on one shard; multi-rule scripts work. With per-principal hashing each user's keys hash independently, load spreads almost perfectly, but the tenant rule is now a cross-shard operation that needs leasing or compensation. A hybrid sub-divides a very large tenant into several tags — {t:atlas:0} … {t:atlas:7} — and splits its tenant budget among them, trading exact tenant accounting for more slots.

The break-even is arithmetic. One shard's budget is 40,000 of 500,000 peak calls, 8 % of fleet load; a tenant that can attract more than that in attempts cannot live on one slot. Atlas's flood is 75,000 attempts per second, 15 % of peak. Northstar still colocates, because most of those attempts are denials and denials can be removed at the gateway (Decision 11) without moving a key. Switch to the hybrid when a tenant's *admitted* traffic alone nears the shard budget, which is a statement about the product's largest customer, not about the technology. Operationally, hash-tag layouts make “hot tenant” and “hot shard” the same alert, and the tag scheme is part of the key schema, so changing it later means migrating keys or accepting a window of reset buckets.

Decision 11 · Let the shard serialize a hot key, or lease token chunks?

Two different hot keys share a symptom and need different cures. Atlas's flood is deny-heavy: 75,000 attempts per second at a single shard that can do 40,000, almost all refused. The limiter's load is determined by attempts, not by admits, so being mostly denied does not make it cheap. P-GLOBAL is admit-heavy: 40,000 requests per second on GET /catalog, nearly all allowed, all against one key.

For the deny-heavy case the gateway honours the store's retry hint with a short negative cache: after a denial with retryAfter, it denies the same key locally for the hint, capped at 250 ms. The cap means the gateway never denies for longer than the store would have, so the cache can only remove load, not wrongly admit. With 100 gateways each asking at most four times a second, 75,000 attempts per second fall to at most 400 store calls per second for that key. For the admit-heavy case, denying locally is useless; instead each gateway leases a chunk of 100 tokens by calling the same bucket script with cost 100, spends it without a store call and asks again when empty. With 100 tokens per chunk, store operations fall from 40,000 per second to 40,000 ÷ 100 = 400 per second.

The price of leasing is a bound you can write down: each gateway may hold up to 100 tokens, so 100 × 100 = 10,000 tokens may be parked, which is 0.25 s of global capacity. That is the overshoot and unfairness ceiling, acceptable only if the backend tolerates a quarter-second of extra load. Near exhaustion, a bucket holding 60 tokens cannot satisfy a 100-token request, so production leases retry with smaller chunks or grant partial ones; unspent tokens expire with the lease rather than being refunded, which would reintroduce the double-refund problem.

Letting the shard serialize every request is the right answer when exactness matters more than throughput and the key's rate is modest: for a billing-grade rule at a few hundred requests per second, a lease buys nothing and costs precision. Approximate local counters synchronized periodically sit between the two. Operationally, watch parked tokens, chunk renewal rate and negative-cache hit rate alongside the shard's script rate; a lease size is a tunable with a safety meaning, and changing it changes the overshoot bound.

A limit that was exact inside one region becomes a negotiation once two regions serve the same tenant. The three decisions here are about who decides, how much error you will pay for in exchange for latency, and how regions learn what others have spent.

Decision 12 · A central global authority or independent regional enforcement?

Atlas's P-TENANT limit is 1,000 requests per minute globally, and Atlas has traffic in both EU and US. If each region simply enforces the full 1,000 on its own, the combined admission can reach 2,000 per minute — twice the limit — and nothing in either region's logs looks wrong. The failure is invisible from inside any one region.

The central option sends every decision to a single home region. It is exact, but a US request pays a WAN round trip to EU for each admission; as a hypothetical illustration, 80–120 ms against a design goal of a few milliseconds locally. During a partition the remote region cannot admit at all. The independent option is fast and partition-proof and overshoots by up to a factor of the region count. A static split (500 each) fixes the overshoot and creates its own injustice: if EU demand is 900 and US demand is 100, EU is denied 400 while 400 US tokens go unused.

Northstar picks neither pure form for plan-level quotas. The authority is central for *allocation* and regional for *enforcement*, which is decision Decision 13. For rules where the overshoot has no cost — a cheap, fleet-protective read limit where the backend independently absorbs 2× — independent regional enforcement is entirely reasonable and is the simplest thing that works. Central per-request authority wins where the rule is rare, the value of exactness is high, and the callers tolerate WAN latency: administrative operations, or a hard monetary cap. The operational result of the central form is that one region's health becomes every region's admission health, so that region's runbook needs the language of a Tier-0 service.

Decision 13 · Strict global limit or bounded-overshoot regional leases?

Once an allocator is in the picture the question is what a region may do with what it was given. In the leased design the allocator holds the pool for the minute window (1,000 tokens for Atlas). Regions request leases in chunks of 100 tokens with a TTL of 15 s, clamped to the window end; grants never exceed the remaining pool at the moment of the grant. EU and US spend locally, with no WAN round trip per request, and go back to the allocator when a chunk runs out. Demand-weighted renewal addresses the depletion unfairness above: EU, being busy, simply renews more often.

What is the overshoot? The general bound is admitted in a window ≤ G + Σ(unclamped carry-over across the window boundary) + Σ(local fallback allowances) + (grant records lost by asynchronous allocator failover). With clamped leases the first term is zero. If leases were *not* clamped, two regions each carrying up to a 100-token chunk across the boundary would add up to 200, +20 % of G. A bounded local fallback of 100 tokens per region during a partition adds up to another 200. In a partition where US holds 200 unspent tokens, it can admit at most those 200 until the leases expire, gets nothing new, and — with clamping and grants capped by the pool — causes zero overshoot; the cost is that up to 200 tokens (20 % of G) may sit stranded and denied.

The strict alternative is the home-region design of Decision 12: zero overshoot by construction, a WAN price on every remote decision, and no remote admission in a partition. It wins when overshoot is expensive or illegal: a contractual hard cap, a regulated quota, a per-account spending limit. Leases win when low latency and partition tolerance matter and a stated, bounded excess is affordable. The operational consequence is that every term in the bound becomes a configuration choice that someone has to own: chunk size, lease TTL, the clamp, the fallback allowance, and how allocator failover records its grants.

Decision 14 · Synchronous reservation or asynchronous aggregated counts?

Regions need to learn how much of the pool each of them is spending. The strict way is a synchronous reservation: a region spends tokens it has reserved at the authority, so the authority's ledger is always at least as large as what has been spent. The cheap way is asynchronous aggregation: each region counts locally and ships counts to a peer or a central service every second or so, and each region subtracts what it hears.

Asynchronous counts are tempting because they cost nothing on the hot path. The trouble is what happens near the end of the pool. Suppose both regions have been told that 100 tokens remain, and each decides, locally and independently, that it may spend them before the next exchange. Admission can reach up to 100 above the pool — a hypothetical 10 % of G for two regions — and the bound scales with (regions − 1) × remaining, which grows quickly with more regions or slower exchanges. The reserved design pays for one authority interaction per chunk, not per request, but never lets two regions spend the same token.

Northstar uses synchronous reservation at chunk granularity as the admission authority, and asynchronous counts only for *demand signals*: which region is busy, how fast it is draining a chunk, what the renewal weights should be. Async wins when the quota is soft — a free-tier suggestion, analytics-grade usage meters, a fairness estimate — and an overshoot of a few percent is invisible. The operational point: when you read a dashboard of aggregated counts, know which kind of number it is. An aggregated view can be stale and eventually consistent, and it must never be what decides a contractual limit.

Sooner or later the Redis shard that owns Atlas's keys times out. The limiter then has to answer a question it was built to avoid: what should happen to a request when the limiter cannot decide?

Decision 15 · Fail open or fail closed — chosen per endpoint

The store is unreachable, circuit breakers have opened and the gateway holds requests it cannot evaluate. A single global setting is the most common mistake here, because routes are not equal. Consider the five Northstar endpoints. Letting POST /reports/export through unchecked would let Atlas queue exports that each cost about 10 reads of backend capacity and last minutes: failing open there turns a Redis problem into a backend outage. Refusing GET /catalog entirely would turn the same Redis problem into a customer-visible outage of the product's front page, when the backend can comfortably serve 2,000 requests per second of reads.

Three behaviors are available. Fail open admits everything and is defensible only where the backend independently survives the traffic; no Northstar endpoint qualifies, so it is not the default anywhere. Fail closed returns 503 for the route: users see an outage for that route, a trade made deliberately for routes where admitting an unbounded amount is worse than refusing. Bounded local fallback enforces a conservative local cap derived so that the fleet-wide excess is a computable number; it is a weaker, stated guarantee, not a preserved one. Northstar's assignment: exports, checkout and login fail closed (503), because admitting an unmetered amount there costs more than a short outage; catalog reads and search use a bounded local fallback with caps scaled to their cost; health probes and the privileged internal route sit on a narrow, separately authenticated path with its own small cap and do not depend on the quota store.

The alternative wins in the right place. A fail-open choice is sound for a low-cost, idempotent endpoint whose backend is itself protected by a bulkhead or load shedder, since there the limiter is a courtesy. Fail-closed everywhere is sound for a system where a duplicate admission is far worse than downtime — a safety-critical command channel. Operationally, the matrix is policy: it should be versioned, reviewed with the owners of each backend and *rehearsed*, because the first time you find out a route's fail mode is the day the store dies. See circuit breaker and load shedding for the surrounding patterns, and keep 503 distinct from 429: one says the service or its dependency is unavailable, the other says a valid caller exceeded a limit.

Decision 16 · A fixed local fallback cap or deny on dependency loss?

Once a route uses a bounded fallback, its size is its own decision. Take P-READ with two gateways: each runs a local bucket of capacity 10 and 1 token per second, so the pair admits at most 20 and 2 per second in total when traffic is split evenly. If U-17 sends a steady 2 requests per second, exactly within P-READ, and the load balancer sends 80 % of them to A, A receives 1.6 per second but can sustain only 1, while B admits its 0.4: 1.4 per second get through and about 0.6 per second of legitimate traffic is denied.

Options: a static per-gateway cap (limit divided by configured gateway count — predictable, simple, wrong when traffic is unbalanced); a dynamic share (limit divided by the number of healthy gateways, which needs membership information and itself misbehaves when it is wrong); or denial, which is exact in its own way, since no unbounded admissions occur, but converts a partial outage into a total one for that route. At fleet scale the same arithmetic applies to P-GLOBAL: 40,000 ÷ 100 gateways is 400 requests per second per gateway, and a gateway receiving 3× its share would deny two-thirds of its traffic while the others idle.

Northstar uses the static cap for reads because predictability beats cleverness during an incident, and accepts under-admission in exchange for a bound on over-admission. Denial wins on the routes of Decision 15 that fail closed. Dynamic shares win in fleets with reliable membership and very uneven load. The operational consequences are two, and both are easy to forget: engagement of the fallback must be a first-class metric, because a silent fallback changes what “20 per 10 s” means; and recovery needs care. When the store returns, buckets that sat idle may hold their full capacity — up to 20 tokens per active user — so a burst of catch-up demand is possible. Ramp gateways back to the store rather than switching all at once.

Counting requests treats a catalog read and a ten-minute export as the same thing. The three decisions below are about measuring what a request actually costs, limiting how much of it runs at once, and choosing what happens to the traffic that exceeds the limit.

Decision 17 · Weighted cost or plain request count?

Northstar's backend comfortably serves 2,000 GET /catalog requests per second, and an export costs about ten times a read. Atlas sends 100 exports per second. A request counter sees 100 requests per second, far below the 2,000 the backend handles for reads, and approves. Backend load, though, is equivalent to 1,000 reads per second: half the capacity, consumed by one tenant on one route, while Birch's reads slow down.

Choices: a plain count (one unit per request, trivially explained); separate buckets per route (exports have their own small budget, so cheap traffic can never be starved by heavy traffic and vice versa, at the price of more keys and a rigid split); or a weighted cost against a shared bucket, where POST /reports/export charges 10 units and a read charges 1. With weights, a user holding 3 tokens who requests an export has a deficit of 7, and 7 ÷ 2 tokens per second = 3.5, so the response says retry after 4 seconds, rounded up to whole seconds. The cost must be positive and no greater than the bucket's capacity, or the request could never succeed.

Northstar uses weighted cost for the shared budget and a small separate mechanism for concurrency (Decision 18). Plain counts win when requests are genuinely uniform, and are the right default for a young API where nobody has measured per-route cost. Separate buckets win when you want strict isolation between route families. The operational consequence of weighting is that the cost table is policy: calibrate it from measured backend cost, version it like any other policy, and tell customers when a cost changes, because changing a weight from 10 to 20 silently halves the exports Atlas can run.

Decision 18 · Concurrency semaphore, rate limit or queue?

Atlas starts one export per second, within every rate limit, but each export runs 60 seconds. After a minute 60 exports are executing at once, holding workers, memory and database connections, and the system is overloaded although the request rate never rose. A rate bucket measures arrivals; this problem is about *simultaneous work*.

A concurrency semaphore is the direct fix: P-EXPORT allows at most 3 in-flight exports per tenant. Acquire takes a lease with an expiry, release gives it back, and expiry reclaims leases whose holder crashed; releasing is idempotent, so releasing twice or after expiry changes nothing, which is what makes it safe to compensate. The ordering is lease first, token debit second, release if the debit denies. A queue accepts the work and runs it later under a worker pool of fixed size: the caller receives a job id rather than a refusal. A rate limit alone would have to be set so low to keep the 60-second case safe that it would starve short exports.

Northstar combines the weighted rate (how much capacity per unit time) with the semaphore (how many at once). The queue wins for work that can legitimately be asynchronous: if an export may complete in minutes and be fetched later, accepting a job and rate-limiting job *creation* is a better product than any amount of 429s, and it moves overload handling to an explicit backlog with its own bounds. Operational caveats for the semaphore: lease TTL must exceed the longest honest export or long jobs lose their slot mid-run (heartbeat renewal handles this), but be short enough that a crash does not hold a slot long; and a concurrency denial has no computable retry time because it clears when some export finishes, so the 429 carries a policy-chosen Retry-After. See the bulkhead pattern for the general idea of capping resource use per tenant.

Decision 19 · Reject with 429, or delay, shape and queue?

The 21st rapid GET /catalog from U-17 finds the bucket empty. The gateway can answer 429 Too Many Requests with Retry-After: 1, or it can hold the request for about a second and then serve it, which looks friendlier. Both are defensible; they differ in where the excess waits.

Rejecting (429, defined in RFC 6585, with Retry-After as defined in RFC 9110 — a delay in seconds or an HTTP date, a hint rather than a reservation) pushes the waiting to the client and costs the gateway almost nothing. Delaying keeps a connection, a buffer and a request context alive per waiting request; if 5,000 requests are held for a second — a hypothetical figure — the limiter has converted a rate problem into a connection-count and memory problem inside the gateway, and the held requests may time out upstream anyway. Queueing with a bounded queue and an explicit drop policy is the principled form of delay, and belongs to a worker or message system, not to a front-door gateway.

Northstar rejects at the edge. Delaying wins for internal batch callers on the same trust boundary, where a client is under your control and smoothing is worth a held connection, and for background work behind a queue with backpressure — see backpressure. Operationally, give Retry-After an integer number of seconds rounded up, make it honest for token-bucket denials, avoid promising a reset time for dynamic quotas, and do not expose tenant-level quotas or reset boundaries to unauthenticated callers. Decide separately what the client should do next, which is the subject of Decision 20.

A limiter is operated by people and used by software that did not choose to be limited. These four decisions concern the behavior of clients on rejection, the way policy is written and shipped, and what the limiter records about itself.

Decision 20 · Fixed retry delay or jittered exponential backoff?

A partner outage ends and 10,000 clients, all told Retry-After: 1, wait one second and retry. All 10,000 arrive in the same instant. The limiter denies most of them, they are told to wait again, and the pattern repeats: a self-sustaining retry storm in which each round is as sharp as the first.

A fixed delay that honours the hint exactly is simple and maximally synchronized. Exponential backoff without randomness spaces successive attempts but keeps clients that started together moving together. Jittered exponential backoff adds a random component, so the same 10,000 clients, spread over roughly one to two seconds, arrive at about 10,000 per second on average rather than as a single spike, and later attempts spread further. The server can help by adding a little jitter to the hints it emits on bulk denials, so even clients with naive logic do not synchronize.

Northstar's client guidance is jittered exponential backoff with Retry-After as the lower bound, a cap on attempts and, for non-idempotent calls, an idempotency key so that a retry after an ambiguous timeout cannot double-execute. A fixed delay is fine for a small number of cooperating internal clients where the load is known. The operational point is that the limiter cannot fix a retry storm from the inside — it can only avoid worsening one — so publish the backoff contract and test your own SDKs against it. See retry with backoff, the retry storm module and idempotency.

Decision 21 · Fine-grained policy targeting or operational simplicity?

Atlas's account manager asks for a higher export allowance for Black Friday week. Then Birch asks for something similar, then a partner wants a different login limit, and within months the policy file has a rule for each of 200 tenants on each of 5 routes — 1,000 hypothetical hand-written rules, with no one able to say which of them still matter.

The options: fine-grained targeting (any tenant, user or route can have its own number — expressive, and every rule is a thing to review, test and eventually forget); plan tiers (a few templates such as free, standard and enterprise, with tenants assigned to them — less expressive, easy to reason about); or tiers plus a small, audited override list whose entries carry an owner and an expiry. Northstar's P-TENANT of 1,000 requests per minute for Atlas is a plan-level value, and other plans differ.

The third option is the working compromise: tiers carry the 95 % case, overrides carry exceptions and expire by default, so a Black Friday raise disappears on its own. Pure fine-grained targeting wins for a platform with a handful of very large customers who negotiate contracts. Tiers alone win for self-serve products. Whichever you choose, make the effective policy for a caller explainable — “you are on the standard plan, this rule denied you” — because unexplainable limits become support tickets, and an operational rule of thumb: if an override has no expiry, it is a plan you forgot to define.

Decision 22 · Centralized policy publishing or application-embedded config?

An incident needs the export cost doubled right now. If limits are constants in the gateway's configuration, that is a code change, review, build and rolling deploy across 100 gateways, during which some gateways enforce the old value and some the new. If the rolling deploy also changes the algorithm or its parameters, the eligibility of callers shifts with whichever gateway they happen to hit.

Alternatives: embed policy in the application and ship it with each release (simple, versioned with the code, slow to change); fetch policy per request from a central service (always fresh, a new dependency on the hot path); or a control plane that publishes immutable, versioned snapshots which gateways cache and switch to atomically. Northstar chooses the last: a bundle with a monotonically increasing version and a checksum is validated, replayed against sample traffic offline, rolled to a canary slice, then released, and one request is evaluated against exactly one bundle with its version recorded in the decision. Gateways accept only higher versions, so a delayed old snapshot cannot overwrite a newer one; a rollback is a new bundle with older content and a higher version.

Embedded config wins for a single small service with infrequent changes, where the control plane would cost more than it saves. What the snapshot model adds is a staleness question: a disconnected gateway keeps its last bundle unless told to expire it, so give snapshots a maximum age and a conservative behavior after it, and treat time-to-tighten (push plus poll) as an SLO. Stale *policy* and stale *counters* are two different problems; keep them in separate runbooks.

Decision 23 · Detailed decision logs or cost, cardinality and privacy?

After the Atlas flood, security asks for every decision: who, which rule, what remained. At a hypothetical 500,000 decisions per second and 200 B per event, that is 100 MB per second, close to 8.6 TB a day, before indexes. It also puts hashed identifiers, and in careless implementations raw ones, into a long-retention store.

The options are to log everything (complete, expensive, and a privacy surface); to emit aggregated metrics only (cheap and cardinality-safe, but you cannot answer “what did K1 do between 02:00 and 02:05?”); or to combine aggregates with sampled event logs — low sampling for allowed requests, high for denials and fallbacks, and a rate limit on the logging itself so that a flood does not become a log flood. Northstar takes the combination. Tenant, rule and route group are bounded dimensions and fine as metric labels; user ids, API keys and IP addresses are not, and appear in logs only as a keyed hash alongside the decision id.

Full logging wins in a regulated domain that requires it, with retention and access controls built for it. The point that does not change by choice: decision telemetry travels over a lossy, asynchronous pipeline and may drop or duplicate events, so it is evidence for dashboards, alerting and investigation, not a ledger, and billing or contractual quotas need a stronger accounting path. Overshoot, too, cannot be read from the store; it is estimated offline from sampled logs grouped by rule and true window and compared to the design's documented bound. See Failure and Scale Scenarios for incidents where this matters.

The last two decisions look at the edges of what an atomic, in-memory counter can promise: what is left after a primary fails, and what to do about requests whose price is unknown when they arrive.

Decision 24 · A single-region Redis, or multi-zone replication with explicit failover semantics?

Primary R in one availability zone loses its host. If R is the only copy, every bucket on that shard vanishes. Gateways then see a store that is down (Decision 15) and, when a replacement starts, a store that is empty, which means every caller has a full bucket. For a rate limiter that is a lenient failure, but it is a failure: enforcement was reset exactly when load may already be high.

A single instance is simplest and fails completely. Primary plus replica in another zone (R and R2) keeps state through a host loss, but Redis replication is asynchronous by default: after promotion, writes acknowledged shortly before the failure may be missing, so a few debits are forgotten and callers get slightly more than their share. Waiting for replica acknowledgement (WAIT) can narrow that window at a latency cost, yet it does not turn the setup into a strongly consistent store. A hypothetical bound: at an average 25,000 calls per second per shard and a replication lag of 100 ms, a failover could forget up to 2,500 debits — at most 2,500 tokens of excess across thousands of keys, negligible per key. Persistence adds a further wrinkle: restoring from an old snapshot gives callers older, fuller buckets.

Northstar uses a replica in a second zone, accepts asynchronous replication, and treats state loss as a bounded, temporary leniency rather than a correctness violation. A single instance wins when the limiter protects something where a reset is harmless and the extra node is not worth running. Synchronous waiting wins for a small set of rules where a lost debit costs real money. Operational implications: keep script loading robust (the cache is volatile across restarts and failovers), monitor replication lag and evictions, and be careful with eviction policies, since a silently evicted counter is a silently reset limit. See Redis production engineering and distributed data.

Decision 25 · Pre-admission reservation or post-execution accounting?

An export's true cost is known only when it finishes: the estimate was 10 units, the actual run scanned a large dataset and cost the equivalent of 14. Charging only the estimate means heavy exports are under-billed, and charging afterwards means the work was already done.

Three options. Pre-admission reservation charges a conservative estimate up front, so the budget bounds work before it starts. Reserve and settle charges the estimate and then trues up: for 14 actual against 10 reserved, debit 4 more (allowing a bounded negative balance); for 6 actual, credit 4 back. Settlement must be idempotent, keyed on the decision id, or a retried completion event will double-refund. Post-execution accounting records actual cost after the fact and enforces it on later requests — accurate and unable to stop the expensive request that is already running.

Northstar reserves a class-based cost up front and settles only for exports, where the work is expensive enough to justify an idempotency record; cheap reads pay their cost at admission and are never refunded, because the extra keys and the refund risk would exceed the benefit. Post-execution accounting is the right shape for *metering* — billing, usage dashboards — where nobody intends the limiter to block. Operationally, decide how much negative balance is tolerable, and note that settlement events can be lost: the limiter's budget is then slightly generous, which is a bounded error you can state.

The 25 decisions are not independent. Choosing a weighted token bucket (Decision 2, Decision 17) constrains the script (Decision 5, Decision 9), which constrains the sharding layout (Decision 10), which in turn limits what you can do about hot tenants (Decision 11) and how regions share their budgets (Decisions 12–14). It is worth seeing the net effect for the five endpoints at once.

Two themes recur. The first is that every mechanism has a failure that is *bounded* and one that is *unbounded*, and the design work is making sure the system lives in the bounded one: the lease size bounds overshoot at 0.25 s of capacity, the clamp bounds regional excess, the negative-cache cap bounds how long a gateway can deny. Where a decision leaves a failure unbounded, as an unclamped lease would, that is a defect to fix, not a trade to explain. The second is that the cheap path should stay cheap: one store round trip per decision for the common case, no idempotency record for cheap reads, and heavy machinery — settlement, semaphores, home-region calls — only for routes whose cost justifies it.

• Write down, for each rule, the failure it accepts and the number that bounds it.
• Calibrate weights and fallback caps from your own measurements and label planning figures as assumptions.
• Test fail modes and recovery ramps before the incident that needs them.
• Give every override an owner and an expiry.

• Copying capacity figures from this page into a design without a load test.
• One global fail-open or fail-closed switch.
• Refunding tokens after an ambiguous timeout.
• Using sampled decision logs for billing.