Distributed Cache Trade-offs: TTL, Eviction, Sharding & Failover

A cache that works in the demo and a cache that survives a Tuesday-evening sale differ in about twenty small choices. This page takes each of them in turn: the incident that forces the question, the realistic alternatives, what each one costs in latency, money, freshness and durability, what a fictional retailer picked, and the circumstances in which you should pick differently.

The running example is fictional. Meridian Commerce is a B2C storefront whose source of truth is a relational product database; its Product API runs as a hypothetical 20 pods in front of cache shards A, B and C (and later D), each with an asynchronous replica A', B', C'. The hot item is product:4821, a product whose price the shopper sees as $99 (record version v5) until the database changes it to $109 (v6). Two tenants appear: Cedar, a large merchant that can be noisy, and Birch, a well-behaved one that must stay healthy. Every rate, latency, size and memory figure is a hypothetical teaching input chosen so the arithmetic can be checked by hand. None is a benchmark, a vendor measurement or an industry statistic. The sibling pages Complete Design and Architecture Deep Dive use the same numbers and build the mechanisms step by step, but you do not need them first: every term is defined where it first appears below, and each decision stands on its own.

The workload every decision is measured against

Meridian averages 12,000 requests per second and peaks at 60,000. Eighty percent of requests are reusable product reads that may be cached (cache-eligible), so the peak has 48,000 eligible and 12,000 non-eligible requests per second. The baseline hit rate (hits divided by eligible lookups) is 90%, which leaves 4,800 eligible misses per second at peak and a database-facing load of 4,800 + 12,000 = 16,800 reads per second. The database is assumed to handle 24,000 reads per second, so that load is 70% of capacity (operators here keep the database under an 80% ceiling, 19,200 reads per second, to leave room for spikes); without any cache the same peak would ask for 60,000, which is 2.5 times what it can do. Eligible requests cost 3 ms on a hit and 85 ms on a miss, so their mean is 0.9 × 3 + 0.1 × 85 = 11.2 ms. The 99th percentile (p99, the latency that one request in a hundred exceeds) sits near the miss time, around 85 ms.

The value stored for a product averages 1.5 KiB (1,536 bytes), and there are 8,000,000 potentially hot objects. Payload alone is 11.44 GiB, or 12.288 GB in decimal units; GiB uses powers of 1,024 and GB powers of 1,000, and mixing them is a roughly seven percent mistake waiting to happen (1 GiB is 1.074 GB). With assumed key and metadata overhead plus an allocator factor the memory bill becomes 15.27 GiB, spread over shards of 8 GiB budget each.

One sentence from the cache's job description should stay in mind throughout: a cache holds a derived, best-effort copy, and the database stays the authority. A hit therefore proves only that a copy exists. It does not prove the copy is fresh, complete, or that the caller may see it. Most of the decisions below are really choices about which of those gaps you are willing to live with, and who closes them when it matters. For the single-process vocabulary see the Redis lessons on fundamentals and production engineering.

Before any question about TTLs or shards there are three earlier ones: which process holds the copy, which code fetches it on a miss, and which data is allowed in at all. Get these wrong and the later tuning is decoration.

Decision 1 · A cache inside each pod, or one shared cache?

Meridian's first cache was a plain in-memory map inside each Product API pod, a local or L1 cache. It was fast, because a hit costs a hash lookup and no network, and it was free to operate. Then product 4821's price changed from $99 to $109. Pod A refreshed its copy and showed $109; pod B still held v5 and showed $99 for the rest of its TTL, so two shoppers behind the same load balancer saw different prices within a minute. The same design also multiplied misses: with 20 pods each key must be fetched once per pod rather than once overall, so a cold key costs up to 20 database reads per TTL period instead of 1, and every pod stores its own copy of the working set.

The alternatives are a local cache only, a shared remote cache (a separate cluster of cache servers that every pod talks to), or both stacked. A shared cache gives one copy per key, so a fill benefits every pod and invalidation has exactly one place to act; the price is a network round trip on every lookup, a new fleet to run, and a new way to be down. Meridian chose the shared cache as the primary layer, because consistency of a visible price across pods mattered more than shaving a network hop, and because one copy of the 15.27 GiB working set is far cheaper than twenty. The local layer is not abandoned; it returns in Decision 12 for the single hottest keys, with a 2-second TTL.

A local cache wins outright when the data is nearly immutable and small (a country list, a feature-flag snapshot, compiled templates), when the network hop is the dominant cost, or when the process must keep working while the shared tier is unreachable. The operational consequence of choosing shared is that the cache becomes a dependency with failure modes of its own, which Decision 17 and Decision 19 deal with. If you do keep an L1 beside it, the invalidation path has to reach L1 as well; forgetting that is how a perfectly invalidated shared entry coexists with a stale local one.

Decision 2 · Cache-aside, or read-through?

Three teams at Meridian each wrote their own loader for product data. One forgot to set an expiry, so the entry lived until eviction pushed it out; another cached an empty result when the database timed out, turning a transient error into 30 minutes of “product not found”; the third did not coalesce concurrent misses at all. The question was who should own the miss path.

In cache-aside (lazy loading) the application checks the cache, and on a miss reads the database itself and writes the result back. In read-through the application asks only the cache layer, and that layer, through a registered loader, fetches and stores the value. A write-through cache extends the idea to writes by updating the cache and the database together; it is a different decision and shows up in Decision 4. Open-source Redis and Memcached are stores, not loaders, so read-through generally means a library, sidecar or service you run on top of them. That is not an argument against it, only a reminder of where the code lives and who deploys it.

Meridian chose cache-aside, implemented once in a shared Product API cache library that all services use. The application keeps control of what a miss means (a not-found is cached as a negative entry, a timeout is not cached at all), while the library owns TTL assignment, single-flight (Decision 11), version-checked fills (Decision 6) and metrics. That gets most of read-through's consistency of behavior without making a new tier the thing that must scale with every service. Read-through is the better fit when many independent services read the same dataset through one loader, when you want coalescing and negative caching enforced centrally, or when a managed caching product supplies the loader hook.

What gets harder with cache-aside is discipline: the rule only holds if every caller goes through the library, and a direct call to the cache from a new service quietly bypasses TTL classes and version checks. A lint rule or a thin client wrapper that is the only code allowed to talk to the cache is cheap insurance. What gets harder with read-through is that the loader layer is now critical infrastructure whose capacity must cover the miss rate, and a bug in it fails every consumer at once.

Decision 3 · Cache everything, or only selected reads?

When the cache first proved its worth, the temptation was to put a cache in front of every endpoint. The ones that hurt were not the obvious ones: a cart read cached for a few seconds sometimes showed an item that had just sold out, and a one-off export crawler touching two million rarely-used products pushed useful entries out of memory. Caching indiscriminately spends scarce RAM on data that never earns a second read and, worse, spends correctness on data that cannot tolerate staleness.

The options are to cache every read, to cache selectively by data class, or to admit an object only after it has been requested more than once. Meridian picked selection by class and wrote it down. Descriptive content (C1) gets a 15-minute TTL with ±20% jitter, price on the product card (C2) gets event invalidation plus a 300 s TTL backstop, and the availability badge (C3) gets 30 s. A fourth class, C4, is never served from a TTL-only cache: inventory reservation, payment instruments and secrets, and authorization decisions. A cached permission that outlives its revocation is a security defect, not a performance trade.

Selectivity also puts an honest ceiling on the benefit. Only 80% of requests are cache-eligible, so at peak 12,000 of the 60,000 requests per second never touch the cache and the database-facing load can only fall to 28% of the no-cache baseline (16,800 of 60,000), not toward zero. Cache everything wins when the entire dataset is read-mostly, small, public and cheap to rebuild, such as static content behind a CDN, where the selection step costs more than it saves. The consequence for Meridian is a registry of classes that every new endpoint must be placed in during review, with hit rate and staleness tracked per class rather than as one comforting aggregate.

Every cached value starts to go stale the moment the record behind it changes. The four decisions in this chapter decide how fast the copy is corrected, what happens when the correction goes wrong, and how much staleness you deliberately keep as a safety net.

Decision 4 · Invalidate on write, or update the cache on write?

The price of product 4821 changes from $99 (v5) to $109 (v6). Two Meridian services touch the cache around that moment. In the first design the writer, after committing, computed the new product card and stored it under the key. A second change to v7 landed seconds later, the two cache writes crossed in flight, and v6 overwrote v7: the cache confidently showed a value that was already two versions old. Nothing was wrong with either write; they simply arrived out of order.

There are three ways to react to a write. Invalidate deletes the key, so the next read misses and refills from the database. Update (the cache-side half of write-through) puts the new value in directly, so the next read is a hit. A guarded update does the same but only if the incoming version is newer than what is stored. Deleting is simple and makes the database the only producer of cached content, but it converts every write into a miss, and on a key served at 8,640 requests per second with an 85 ms miss path that miss lets 8,640 × 0.085 ≈ 734 requests pile up before the refill finishes. Updating avoids that miss but needs the writer to produce the exact cached representation (joins, tenant variants, serializer version), and unguarded updates can reorder.

Meridian invalidates, and it does so from a subscriber rather than from the writer. After the database commit, a transactional outbox (a table written in the same transaction as the change) feeds an invalidation stream; a subscriber with read access to the authority deletes the key or, for flagged hot keys, refreshes it from the database with a version check. An event does not edit caches by itself; a subscriber that holds authority does. Update-on-write wins when the cached shape is trivial to derive, writes are rare, reads are enormous and even one miss is unacceptable, provided versions guard against reordering. It is also the natural model when the cache sits in front of a store you own end to end and writes always pass through it. The operational cost of invalidation is a pipeline to monitor: outbox lag, subscriber errors and dead-lettered events all become freshness incidents. See the outbox pattern for the mechanism.

Decision 5 · TTL only, or events with a TTL backstop?

Early on Meridian relied on time to live (TTL, an expiry timer attached to an entry) alone, with a 300 s TTL on prices. It needed no extra infrastructure, and a price change was simply seen when the old entry expired. Support tickets arrived anyway: a merchant changed a price, then saw the old one on their own storefront for most of five minutes and assumed the change had failed.

The choices are TTL only, events only, or events with TTL as a backstop. TTL-only bounds staleness by the TTL itself and needs nothing to go right except the clock. Events-only gives fast propagation, but a single lost or dropped event leaves the stale value in the cache until something evicts it, which can be indefinitely. Combining them gives seconds-scale freshness on the normal path and a hard bound when the event path fails. For the price class C2 the bound is the jittered TTL: 300 s ±20% means an entry lives between 240 and 360 s, so the worst case when an event is lost is about 360 s, whereas normal propagation after a commit takes seconds.

Meridian uses events with a backstop for C2, TTL only for the availability badge (30 s is short enough that the event pipeline would cost more than it saves, and the badge never reserves stock), and TTL with stale-while-revalidate for descriptive C1 content. TTL-only wins whenever tolerated staleness is greater than the cost of building and watching an invalidation path, or when the source cannot emit changes at all, as with a third-party feed. Events-only is rarely right for a cache, precisely because a lost event has no upper bound.

None of this makes the displayed price binding. Checkout revalidates price and availability against the authoritative service, so a stale price on the card is an annoyance with a bounded lifetime, not a charge at the wrong amount. What gets harder is observability: you now measure two freshness mechanisms, and the TTL is the number you quote when someone asks how wrong the page can be.

Decision 6 · Direct invalidation, or version-aware fills?

Deleting a key on write sounds airtight until you draw the interleaving. Reader R misses on product 4821 and reads v5 ($99) from the database. Before R stores it, the writer commits v6 ($109) and the subscriber deletes the key, which does nothing because there is nothing there yet. R now writes v5 into the cache. The invalidation has been spent, and the stale entry stays until its TTL expires, up to about 360 s. Nothing in the system misbehaved; this is simply what a delete does against a fill that began earlier.

The choices: accept the race and rely on the TTL to bound it; make fills version-aware, where the value carries the record version assigned by the authority and a fill is applied only if it is not older than what the cache or a recent invalidation already knows; or put the version in the key (…:product:4821:v6) behind a small pointer that names the current version, so a late v5 fill lands on a key nobody reads. The first costs nothing and is the right answer for data where a few minutes of staleness is acceptable. The second needs a compare-and-set primitive: Memcached's cas command with a token is one, while on Redis a short script or a transaction can do the same job. It also needs an invalidation to leave a short-lived marker (“do not accept versions below 6”) rather than a bare delete. Versioned keys are the most robust, but each read costs a pointer lookup or an extra hop and old versions linger until they expire.

Meridian runs version-aware fills for C2, because the price on a popular product is exactly where a 360 s race is visible. The version must come from the database, never from a cache clock or an application timestamp, otherwise two pods can disagree about which write is newer. Versioned keys win for immutable artifacts such as images and compiled bundles, and are also what the v2: schema prefix in Meridian's key format does for serializer changes: an old application version and a new one can never read each other's bytes. The consequence to budget for is the marker traffic and the discipline of carrying recordVersion in every envelope.

Decision 7 · Fixed TTL, jittered TTL, or stale-while-revalidate?

After a deployment Meridian warmed 6,000 catalog keys in the same few seconds, all with a flat 300 s TTL. Five minutes later all 6,000 expired together, and the database absorbed 6,000 refills at once, a self-inflicted stampede with no hot key involved. The fix that costs almost nothing is jitter: draw each TTL uniformly from 240 to 360 s, a spread of 120 s, so the same 6,000 keys expire at roughly 6,000 ÷ 120 = 50 per second.

The third option, stale-while-revalidate (SWR), lets the cache return an entry that has just expired while one background request refreshes it, so users never wait on the miss. It trades freshness for latency and needs somewhere to keep expired entries for the grace period, plus a bounded refresh path. A flat TTL is the baseline: predictable, trivial, and synchronized by construction. Meridian jitters every TTL and allows SWR only on C1 descriptive content, for up to 5 minutes; price (C2) and availability (C3) never serve beyond expiry, because a longer window would defeat the bounds in Decision 5.

Be clear about what jitter does not fix: a single hot key still expires at one instant (that needs single-flight, Decision 11, or early refresh), a mass purge or a bulk invalidation still empties everything at once, and a cold cluster has nothing to spread. A fixed TTL is the better choice when expiry is meant to line up with an external event, for example a rate card that is published on the hour and should disappear on the hour. Operationally, watch the distribution of expiry times, not just the average TTL, and put a ceiling on concurrent background refreshes so SWR cannot become a stampede of its own.

RAM is the scarce resource of a cache, and the workload is almost always larger than it. These decisions say what leaves when memory fills, which absences are worth remembering, and whether to spend CPU to make each entry smaller.

Decision 8 · LRU, LFU, random, or TTL-first eviction?

When a cache reaches its memory limit it must evict, that is, discard entries to make room. At Meridian a nightly catalog-export job started reading about two million rarely-used products once each. Under a recency-based policy those one-time reads looked like the most recently used items in the cluster, pushed out genuinely popular products, and the hit rate fell from 90% to something much worse until the day's traffic rebuilt it. The scan was hypothetical in size, but the pattern, a burst of cold reads displacing warm data, is a classic.

LRU (least recently used) evicts the entry that has gone longest without access; LFU (least frequently used) evicts the one accessed least often, using a counter; random picks a victim blindly; TTL-first policies evict entries closest to expiry. Redis offers each of these as a setting, in allkeys-* forms that consider every key and volatile-* forms that consider only keys carrying a TTL. Two details from its documentation matter in practice. LRU and LFU are approximations: LRU samples a handful of keys (maxmemory-samples, 5 in the docs) rather than keeping an exact list, and LFU uses a small probabilistic counter with a decay time. And a volatile-* policy, when no key has a TTL, behaves like no eviction at all, so writes begin to fail.

Meridian chose an LFU allkeys policy for the product cache because its popularity is stable and heavy-tailed, and the scan is the worst case for LRU. The wrinkle is trending items: a product that suddenly goes viral starts with a low counter and takes a while to look valuable. For that reason hot-key handling (Decision 12) is a separate mechanism rather than something eviction is trusted to do. allkeys rather than volatile is a defense against the day someone writes a key with no TTL. LRU wins when demand shifts quickly and scans are rare; random wins when access really is uniform and you want zero bookkeeping.

The consequence is that eviction becomes a normal, measurable event rather than an alarm: eviction rate, hit rate by data class and memory headroom (also needed for replication buffers and, if persistence is on, for snapshot behavior) become the three graphs that tell you whether capacity is right. The sizing arithmetic lives in the worksheet below.

Decision 9 · Negative caching, or ask the database every time?

A marketing e-mail linked to 40,000 products that had since been delisted. Every click was a miss, because there was nothing to cache, so every click went to the database for an answer that was always “not found”. Misses that can never become hits are the cheapest load to remove and the easiest to overlook.

Negative caching stores the absence: an entry that means “this product does not exist” for a short time. The options are to always resolve misses from the database, to cache the negative result briefly, or to put a probabilistic filter (such as a Bloom filter) in front that says “definitely not present” without any lookup. Always asking is simple and correct, and exactly as expensive as the miss rate. A short negative entry removes the repeated lookups, at the price that a product created within that window looks missing, so creation must delete the negative entry. A filter handles huge id spaces without storing every absent id, but it adds a structure that must be rebuilt as the catalog grows and it has false positives.

Meridian caches negative results for 30 s with a per-tenant cap on how many negative entries one tenant may hold. The cap exists because an attacker, or an innocent crawler, that enumerates ids can otherwise fill memory with “not found” markers and evict real data, and because Cedar's keys must not be able to crowd Birch's. Negative entries are scoped by tenant, and a 401 or 403 is never cached as “not found”: an authorization failure for one identity must not be replayed as a miss for another. Always asking the database wins when misses are rare, when create-then-read must be immediate, or when the answer depends on who is asking. Expect to monitor negative-entry counts per tenant and the rate at which they are hit.

Decision 10 · Compress values, or spend memory instead of CPU?

The memory worksheet showed 15.27 GiB for 8,000,000 objects, and finance asked whether compression could halve it. The temptation is real, but Meridian's average object is only 1.5 KiB, and compressing small values saves little while paying CPU on every read and write and adding a decompress step to the hit path.

The options are no compression, compression of every value, or compression only above a size threshold, performed by the client library before the bytes are sent. As an illustration with a hypothetical 2:1 ratio on the payload, a 1,536-byte value becomes 768 bytes, the item footprint falls from 1,640 to about 872 bytes including the assumed key and metadata overhead, and 8,000,000 items take roughly 6.50 GiB, or about 8.12 GiB after the 1.25 allocator factor. That is a real saving, but it depends entirely on an assumed compression ratio that must be measured on real data; much of what is stored (prices, flags, numbers) compresses poorly and gains nothing.

Meridian compresses only large C1 descriptions above a size threshold it tunes from measurement, and leaves C2 and C3 alone, since those values are tiny and live on the hottest path. The envelope's compression field records which codec was used, so a reader never has to guess. Compression wins when values are large (documents, rendered fragments), when memory or cross-zone bandwidth is the dominant cost, and when the application has CPU to spare. The ongoing consequence is that you pay CPU in the Product API pods rather than in the cache, that raw values can no longer be read with the engine's own tooling, and that a codec change is a rollout with a schema-version bump.

A cache is least comfortable when many requests want the same thing at the same moment, or when it has nothing yet. The three decisions here cover coalescing duplicate misses, relieving one overloaded key, and deciding whether to fill the cache before traffic arrives.

Decision 11 · Single-flight in each process, or a distributed lease?

A flash sale puts product 4821 on every home page. Its entry expires, and 5,000 concurrent requests miss at the same instant. Each independently queries the database for the same row. At the steady 8,640 requests per second this key sees, the 85 ms refill alone would let about 734 requests through before the first one finished; in a burst, the count is the number of concurrent callers. This is a cache stampede (or thundering herd): the cache's protective effect disappears at exactly the moment demand is highest.

Single-flight (request coalescing) lets one caller perform the fetch while the others wait for its result. Inside each process it needs only a map of in-flight loads and costs nothing to operate: with 20 pods, 5,000 misses become at most 20 database fetches, around 250 waiters sharing each one. A distributed lease goes one step further: a caller takes a short-lived lock on the key in the shared cache (a set-if-absent with an expiry, for instance) and only the holder fetches, so the database sees 1 fetch, plus a few more if the lease expires while the holder is still working. The lease costs an extra round trip on the miss path, makes the cache a coordinator as well as a store, and brings every distributed-lock problem along: a holder that crashes leaves the others waiting until the lease expires, and a holder that is merely slow lets a second one in.

Meridian uses per-process single-flight for everything, because the failure it cannot absorb (20 identical queries) is cheap, and reserves a distributed lease for a short list of keys whose miss is genuinely expensive, such as an aggregate that takes seconds to compute. Waiters carry deadlines and, once those pass, serve stale data where the class allows it or fall through to the database under the global fallback cap rather than queueing forever. Locks are harmful in exactly the cases they are tempting: they do nothing for many different keys expiring together (jitter, Decision 7, does that), they turn an owner crash into a latency cliff, and a lease service that is slow can itself become the outage. The distributed option wins when 20-fold duplication of a heavy query is still too much for the database. Read more under distributed locks and the cache stampede module.

Decision 12 · Copies of a hot key, or a single owner?

Sharding spreads keys, but not requests for one key. At peak, 18% of eligible traffic goes to the top product: 48,000 × 0.18 = 8,640 requests per second on one key. The shard that owns it serves those plus its quarter of everything else, 8,640 + (48,000 − 8,640) ÷ 4 = 18,480 per second, which is 73.9% of a hypothetical 25,000 ops/s per-shard budget while the others carry 9,840. If a trending item takes 45% of eligible traffic, that is 21,600 per second on the key and 28,200 on its shard, beyond the budget while the other shards sit at 6,600. (The number 8,640 also happens to be the average-load hit count; here it is the single key's peak rate, and the two are unrelated.) Adding shards does not move the problem, because a key lives on exactly one.

There are three ways out. Keep single ownership and accept the load, which is fine until it is not. Put a short-lived local L1 copy of just the hot key in every pod: with a 2-second TTL on 20 pods, each pod refreshes at most once every 2 s, so the shard sees at most 20 ÷ 2 = 10 GETs per second for that key instead of 8,640. Or replicate the key across several shards under suffixed names (product:4821#1 … #k) and have readers pick one at random, which spreads reads but multiplies every write or invalidation by k and needs routing logic of its own. Replica reads from the shard's asynchronous replica help too, and are stale by whatever the replication lag is.

Meridian flags hot keys from sampled access counts and gives only those an L1 copy with a 2 s TTL, with the invalidation stream also reaching L1 so a price change does not wait out the 2 s. The cost is explicit: that key can be up to 2 s stale on a pod, and the copies live in pod memory, so a flagged list must stay short. Single ownership is right when the key's rate fits comfortably, when it changes often enough that copies are constantly invalidated, or when the class demands the freshest read; such classes are the ones that should not be in a TTL-only cache to begin with. Watch for the day detection lags a real spike: a flash sale does not announce itself, so the L1 TTL and the shard budget need headroom for the first seconds.

Decision 13 · Warm the cache eagerly, or let it fill lazily?

A new shard D joins the cluster after a capacity review, or a deploy replaces the cache fleet, and the entries it should own are gone. During that time a large fraction of lookups miss. At peak, a fully cold cache would send all 60,000 requests per second to the database, 2.5 times its 24,000 capacity, and even a partly cold cache can push the 16,800 normal load above the 19,200 ceiling.

Lazy fill needs no machinery: each miss loads its entry, and the cache warms in the order demand arrives, which is also the order that matters. Eager warming pre-populates the cache before traffic is switched, from a list of known-hot keys, yesterday's access sample or a replay, at a rate the database can take. A mixed approach warms the top of the distribution eagerly and lets the tail fill lazily, usually combined with a gradual traffic ramp. Warming everything is rarely sensible: reading 8,000,000 objects at a hypothetical 2,000 database reads per second takes 4,000 s, about 67 minutes, and most of it will never be read.

Meridian fills lazily in steady state and warms a short list of hot keys before cutting traffic to a new shard or fleet, with the warmer rate-limited against the database budget and stopped by a breaker if latency climbs. Lazy alone wins when the working set is small, the database has real headroom, or the warm list goes stale faster than it is built. The risk of eager warming is that it can load stale data, spend memory on the wrong keys and add load at the worst time. It also needs the same version-aware fill rules as ordinary reads, so it cannot overwrite something fresher.

One node cannot hold 15.27 GiB at 48,000 requests per second with room to grow, so the keyspace is split. How the split is defined decides what an added node costs, who has to know about it, and how large a single failure is.

Decision 14 · Proxy in front, or a smart client?

After shard D was added, a few older Product API pods kept sending some product reads to the node that no longer owned them. Each such request was redirected, retried and answered correctly, but for a short while the cluster spent noticeable effort on redirects, and the slow ones showed up in p99. The pods were running an older routing table.

In a smart-client design the client library knows the topology: in Redis Cluster it keeps a slot-to-node map, sends each command to the owner, and treats -MOVED as “update your map” and -ASK as “try the new owner once during migration, preceded by ASKING”. In a proxy design the clients talk to a thin tier that holds the topology and forwards requests, so clients stay simple and unaware of resharding. Cluster nodes themselves do not proxy; they answer with a redirect, which is why the client needs to understand it. The proxy adds a hop to every request and a fleet that must scale and fail over; the smart client adds a library that must be kept current everywhere, in every language you run.

Meridian runs a single stack, so it uses smart clients and refreshes the slot map on MOVED, with each retry bounded by the request deadline and a limit on how many redirects one request may follow. The routing table has a version, and pods log it so a stale straggler is visible on a dashboard. A proxy wins when many languages or short-lived clients (serverless functions, for example) make per-client topology awareness impractical, when connection counts to the cache nodes need pooling, or when you want one place to enforce per-tenant limits. The consequence of either choice is the same retry hazard: after a timeout the client does not know whether the command ran, so retries of non-idempotent operations such as an increment or a compare-and-set need a token or an explicit ambiguity rule. See retry and backoff.

Decision 15 · Modulo, a consistent-hash ring, or fixed hash slots?

The first sharding scheme Meridian tried was the obvious one: node = hash(key) mod N. With three nodes it worked. When the team added a fourth, the fraction of keys that changed owner was N ÷ (N + 1) = 3/4, so 75% of the cache was suddenly “on the wrong node” and every lookup for those keys missed. A tiny sample of twelve illustrative key hashes (17, 22, 35, 41, 58, 64, 70, 83, 96, 101, 112, 120) shows it: ten of the twelve change node when going from three to four, 83% in this small sample against an expected 75%.

A consistent-hash ring places nodes (through many virtual nodes, or tokens) at positions on a circle and assigns each key to the first node clockwise from its hash, so adding a node steals ranges only from its neighbors. In an illustrative 0–99 space with A at {10, 60}, B at {25, 80} and C at {40, 95}, product 4821 hashes to 33 and belongs to C (the next token is 40). Adding D at {20, 70} moves 20 of 100 positions, all from B, and nothing else, so 4821 stays on C. But the same example shows the catch of few virtual nodes: ownership becomes A 35, B 15, C 30, D 20, uneven until there are many tokens per node. These are made-up hash values, not a real hash function.

Fixed hash slots, as in Redis Cluster, split the keyspace into 16,384 slots by HASH_SLOT = CRC16(key) mod 16384, and a table assigns slots to nodes. This is not a ring and should not be described as one: ownership is an explicit table, and growth is an explicit migration of slots. With three primaries holding 0–5460, 5461–10922 and 10923–16383, product:4821 lands in slot 13107 (node C) and tenant:cedar:product:4821 in slot 8298 (node B). Adding D by moving 1,365, 1,365 and 1,366 slots from A, B and C transfers 4,096 slots, 25% of the space, and 4821's slot moves from C to D while the Cedar key stays on B. Hash tags ({tenant:cedar}:…) force related keys into one slot so multi-key commands work, with the downside that the whole tenant now sits on one slot and one shard.

Meridian picks fixed slots for its primary cache because the cluster must grow online, migrations must be observable and resumable, and MOVED and ASK give clients an explicit way to follow ownership. A ring is the natural fit for a Memcached-style fleet in which the clients do the hashing and the servers know nothing of each other; Memcached itself has no built-in clustering, so distribution is conventionally client-side (check your version's documentation). Modulo is acceptable only for a fleet that never changes size, or one so cheap to refill that a flush is a non-event. Whichever you choose, the topology needs a version number so a client can tell that its map is old, and the key schema must keep hash tags rare.

Decision 16 · A few large shards, or many small ones?

With four primaries of 8 GiB, a node failure removes a quarter of the cache. Suppose a primary dies at peak and its keys all miss. The 43,200 hits per second at peak, divided evenly, put 10,800 of them on that shard. Those become database reads on top of the normal 16,800, giving 27,600 per second against a 24,000 capacity, 115% of what the database can serve, from one lost node.

Splitting the same 32 GiB into 16 shards of 2 GiB cuts the lost fraction to 1/16: 43,200 ÷ 16 = 2,700 extra reads, for a total of 19,500, or 81% of capacity, just above the 80% operating ceiling but nowhere near overload. Rebalancing is gentler too, since a new node takes a small slice from many peers instead of a quarter from three. Many small shards are not free: 16 primaries and 16 replicas is 32 nodes to monitor and upgrade, each node carries fixed overhead and reserve, clients keep a bigger slot map, and a failover storm after an incident affects more nodes at once. A few large shards are simpler to run, and bigger snapshots or resyncs (where persistence is on) are the price of that simplicity.

Meridian starts with four primaries on 16,384 slots, which is already the right unit of work, and treats the failure arithmetic above as the trigger to split slots across more, smaller nodes before the database margin disappears. Few large shards win when the workload is small, operations are the scarce resource, or the database can absorb a node's worth of misses without help. The practical output of this decision is a per-shard dashboard of memory use, ops per second and slot count, so uneven placement is noticed before the hottest shard (4.39 GiB, 54.9% of its budget in the worksheet) turns into the first one to fail.

A cache node will fail. What matters is how much of the cache goes with it, what the database is asked to do next, and which writes may disappear. Three decisions cover the layers of that answer: whether to keep a replica, how hard to push on write acknowledgement, and what each data class does when the cache is not there.

Decision 17 · Primary only, or an asynchronous replica per primary?

Shard C's primary crashed during the evening peak. Without a replica, a quarter of the keyspace was cold and the database felt Decision 16's arithmetic. A cluster with a replica per primary can fail over: a replica is promoted to take over the slot range, holding a warm copy of most of the data, so the cache keeps most of its value through a node loss.

The options are primary only, one asynchronous replica per primary, or replicas that also serve reads. Replication in Redis is asynchronous by default: the primary does not wait for the replica, and the replica acknowledges the offset it has processed afterwards. Partial resync uses a replication ID and offset, with a full resync from a snapshot when that is impossible. The price is memory: a full copy per primary doubles the 15.27 GiB worksheet to about 30.5 GiB across 8 nodes. Reading from replicas spreads load, but replica data can lag, so such reads are potentially stale on top of any TTL.

Meridian keeps one asynchronous replica per primary and does not read from replicas for C2 price, since a lagging replica adds staleness to a class whose staleness is already budgeted. Consider a concrete loss: SET product:4821 $109 v6 is acknowledged at time t; the replica, lagging by a hypothetical 120 ms, has not applied it; the primary dies, the replica is promoted and holds v5 at $99. The write was acknowledged and is gone. Redis's own cluster documentation says that small windows where acknowledged writes can be lost exist. For a cache this is survivable because the database holds v6, the invalidation event or the TTL backstop will correct the copy within the freshness bound, and checkout revalidates. Primary only wins when the database comfortably absorbs a shard's worth of misses and the memory bill matters more than the warm start. The consequence of replicas is not only cost: promotion, split-brain and fencing rules become something you operate.

Decision 18 · Stronger write acknowledgement, or low write latency?

The lost write in Decision 17 raised an obvious question: can the primary wait until the replica has the write before replying? Redis offers WAIT, which blocks the client until a number of replicas have acknowledged. Its replication documentation is careful about what that buys: it reduces the probability of losing an acknowledged write dramatically, but it does not make the system strongly consistent, because a failover can still choose a replica that lacks the write. Memcached has no built-in replication at all, so this choice does not exist there.

Meridian does not use WAIT on cache writes. The cache holds a copy of data that the database already has, so the right way to make a lost write harmless is not a stronger ack but a recovery path that re-derives the truth: invalidation, the TTL backstop and revalidation at checkout. Spending replica round trips on every cache write would raise write latency, make slow replicas into slow writes, and still not guarantee anything. WAIT wins when the data is expensive to recompute or has no other copy yet, such as an in-flight result or a rate counter whose loss would matter. The moment a cache is the only place a value lives, it has stopped being a cache; write-behind designs that acknowledge before persisting make that loss a business error, not a freshness blip. And a timed-out write leaves the outcome unknown to the caller, so retries of non-idempotent writes need tokens.

Decision 19 · When the cache is down: fail open, shed, serve stale or fail closed?

The incident everyone fears: the cache cluster is unreachable at peak. Up to 48,000 eligible requests per second now point at the database, which is 2,880,000 requests for a one-minute outage. The database has a planned 19,200 reads per second ceiling and 12,000 are already non-eligible traffic, which leaves a fallback admission cap of 19,200 − 12,000 = 7,200 per second. Against 48,000 that is 6.7 times oversubscribed: 40,800 requests per second, 85% of eligible traffic, must be shed, answered stale from a class that allows it, or degraded.

The options are fail open (send everything to the database and hope), fail closed (return errors for all cacheable reads), or a per-class policy. Fail open is the instinctive choice and the dangerous one, because it converts a cache outage into a database outage, and retries amplify the load; see cascading failure and retry storms. Fail closed protects the database and fails customers who could have been served. The class policy for Meridian: C1 descriptive content is served from L1 or stale copies, because a product description that is minutes old is harmless; C2 price may read the database within the 7,200 per second cap, with the rest degraded to a last-known price flagged as unconfirmed, since checkout revalidates anyway; C3 availability is simply hidden, because it is display only; C4 always goes to the authority with reserved capacity, and authorization fails closed.

Mechanically the cap is enforced by a circuit breaker, a bulkhead that partitions database connections by class, and explicit load shedding at the API; see circuit breaker, bulkhead and load shedding. Fail open is fine when the database has real headroom over the whole peak, as it does for an internal tool with a fraction of Meridian's traffic. Fail closed is correct for authorization and money. The operational consequence is that the degraded behavior must be built and rehearsed in advance: a game day that actually removes the cache is the only way to learn whether 7,200 is a real number.

The last three decisions concern things that scale with the organization rather than with request rate: users in more than one region, a choice of cache engine that other teams will inherit, and tenants who share the same memory.

Decision 20 · One cache per region, or one cache for everyone?

Meridian serves shoppers in two regions, EU and US, with the product database writing in one place. A single cache cluster in the EU would make every US read cross an ocean. If a wide-area round trip is a hypothetical 80 ms, a hit that costs 3 ms locally costs about as much as the 85 ms miss, and the cache has stopped helping the region that is far away.

The alternatives are one central cache, a cache per region fed by cross-region invalidation, and per-region caches with TTL only. Per-region caches keep hits local and the miss path to a regional read path, and they let a region keep serving when the link is down. Their freshness is bounded by how fast and how reliably invalidations cross the WAN, which is never instant: events can be delayed, reordered or lost. Meridian runs a cache per region, fans the invalidation stream out across regions, and relies on the same 240–360 s TTL backstop; in the worst case, US shoppers see the old price for up to about 360 s while checkout still revalidates against the authority.

A central cache wins when only one region produces most traffic, when the dataset is small and freshness requirements are strict enough that cross-region delay is unacceptable, or when the cost of a second cluster outweighs the benefit. The costs to plan for are an invalidation-lag metric per region, a cold regional cache after a regional failover (the Decision 13 problem again), and a policy for a partitioned region: it keeps serving from cache until entries expire and then follows the Decision 19 rules. See multi-region architecture.

Decision 21 · A Redis-like engine, or a Memcached-like one?

A product team wanted to keep each tenant's top-ten list as a single cache entry and update one element at a time without reading the whole list. Another team only needed to store and fetch serialized product blobs at high volume. They argued over one engine for both.

A Redis-like engine offers data structures (hashes, lists, sorted sets), scripting, replication, an optional cluster mode and the configurable eviction policies described in Decision 8. A Memcached-like engine is a simpler key-value server: opaque values, cas for conditional updates, slab-based memory with an LRU and crawler, and, as typically deployed, no built-in replication or clustering, with distribution handled by client-side hashing. The details are version-specific and worth checking: Memcached keys are limited to 250 characters, an exptime of zero means “never expires” though the item may still be evicted, expiry values above 30 days are read as an absolute Unix time rather than seconds, and the item size limit is configurable and by default about 1 MiB.

Meridian uses a Redis-like cluster, mainly for properties it needs elsewhere on this page: slot-based sharding with online migration, replicas for failover, per-key TTLs, and scripts for version-checked fills. Those same richer features are the cost, since a rarely used command on a large collection can delay other commands on the same node in the usual single-threaded execution model, and a team can easily start treating a flexible store as a database. A Memcached-like engine wins for blob get/set with very large volume and no need for replication, when the simplicity is the point, and when a lost node is simply a smaller cache. Whichever you choose, record the version you validated against, because eviction policies and cluster behavior change between releases.

Decision 22 · A shared pool for all tenants, or dedicated shard tiers?

Cedar launched a large promotion and its entries, including thousands of negative entries from a crawler, filled a shared cluster. Under LFU and the shared memory limit, Birch's perfectly healthy entries were evicted along with everyone else's, and Birch's hit rate and latency fell for reasons that had nothing to do with Birch. Prefixing keys with a tenant name had separated the namespaces, but not the capacity.

The options are a shared pool with per-tenant accounting and quotas, a dedicated shard tier for tenants over a threshold, or a dedicated cluster per tenant. A shared pool is the cheapest and fills evenly, but isolates nothing by itself: memory, CPU, connections and hot slots are all common. Dedicated tiers isolate capacity and failure domains at the price of idle headroom per tier and more clusters to operate. A naming convention is also not a security boundary: preventing one tenant from reading another's keys requires verified identity and access controls on the engine or in a gateway in front of it, and no secrets or unredacted payment data belong in a shared cache at all. One placement trap: tagging every key with a tenant hash tag gives that tenant a single slot, so a noisy tenant becomes a hot shard that cannot be rebalanced.

Meridian keeps a shared pool for the long tail, with tenant-scoped keys, per-tenant limits on entries (including negative ones) and request rate at the cache wrapper, and moves a tenant such as Cedar to a dedicated shard tier once its share of memory or traffic crosses a threshold. Hash tags are used only where multi-key operations need them, never for a whole tenant. Shared wins when tenants are many and small and the cost of dedicated headroom would dominate; dedicated wins for contractual isolation, regulatory separation or a tenant large enough to be its own cluster. The operational price is accounting: the engine does not give per-tenant byte counts for free, so you sample or count in the client library, and you need a documented process for promoting a tenant between tiers.

The 22 decisions are not independent. Choosing a fixed-slot cluster (Decision 15) is what makes hash tags (Decision 22) and migrations (Decision 16) possible; choosing invalidation (Decision 4) is what forces version-aware fills (Decision 6); and choosing to keep the database as authority is what makes losing a write (Decision 18) and a node (Decision 17) survivable. It helps to see what they add up to for each data class.

Two themes recur. The first is that every mechanism has a failure that is bounded and one that is not, and the design work is arranging for the system to live in the bounded one. The TTL jitter bounds simultaneous expiry; the TTL backstop bounds a lost invalidation at about 360 s; the L1 TTL bounds hot-key staleness at 2 s; the fallback cap bounds database load at 7,200 per second. Where a decision leaves a failure unbounded, such as events-only invalidation or an unguarded update, that is a defect to repair, not a trade to explain.

The second is that distribution adds failure modes even while it makes reads faster. A single local map cannot be partitioned, mis-routed, half-migrated, split-brained or cold after a failover; a shared, sharded, replicated cache can be all of these. Every added layer in this page earns its place by fixing a named weakness of the layer before it and creating a new thing to monitor, and that bookkeeping is the actual content of the decisions.

• Write down, for every data class, the staleness you accept and the number that bounds it.
• Carry the authority's record version in every cached envelope and apply fills conditionally.
• Rehearse the cache-down case with a hard fallback cap before you need it.
• Track hit rate, eviction rate, and invalidation lag per data class and per region, not as one aggregate.
• Label every capacity figure as an assumption until measured on your own workload.

• Treating a hit as proof that the value is fresh, complete or authorized.
• Caching inventory reservation, payment data or permission decisions on a TTL alone.
• Retrying non-idempotent cache operations after a timeout without a token.
• Using a tenant name prefix as a security boundary, or a tenant-wide hash tag as a placement strategy.
• Copying numbers from this page into a design without a load test.