Distributed Cache Failure Scenarios: Stampedes, Hot Keys & Recovery

A cache fails in two directions at once: it can disappear and push all its traffic onto the database, or it can stay up and quietly say something false. Thirty incidents, grouped by mechanism, walk through both.

Everything here is a hypothetical scenario on a fictional storefront, Meridian Commerce. The Product API runs as 20 pods against a relational product database; cache shards A, B and C (D after resharding) have asynchronous replicas A′, B′ and C′; two regions, EU and US; tenants Cedar (large, sometimes noisy) and Birch (well-behaved); the hot product is product:4821, a $99 item at record version v5 that becomes $109 at v6. Nothing describes a real outage at a real company, and every number is a planning input or a derivation from one, not a measurement. An interactive Visual Design Journey with 44 scenes sits on the Complete Design page if you want to watch the machinery before watching it break.

Each incident is told as a short story and then pinned down in a table: trigger, timeline, visible symptom, root cause, detection, containment, recovery, prevention, the guarantee that survives, and the trade-off you accepted. One more row matters in a cache more than in most systems: whether the data lost or served wrongly can simply be rebuilt from the product database, or whether somebody treated the cache as the only copy. In most incidents the answer is 'rebuild it'; the exceptions are the ones where the cache was trusted to remember something (F16, when it held the only copy) or to authorize something (F20, F22), and there the damage is a business or security error, not a cold cache.

Load shifted versus truth shifted

Failures that remove the cache (a dead shard, a cold start, a retry storm) are mostly load failures: the answers are still correct, they just arrive from the database, which was sized assuming the cache absorbed 72% of the read work. Failures where the cache stays healthy but wrong (a stale refill, a lost invalidation, a collided key, a lagging region) are truth failures: latency looks excellent, hit rate looks excellent, and the dashboard stays green. The two need different instruments. Load failures show up in database queries per second and queueing; truth failures show up only when you compare cached content with the authority, or when a customer notices.

Baseline numbers used throughout

Peak traffic is 60,000 requests per second, 80% of them cache-eligible (48,000 per second) with a 90% hit rate, so 4,800 eligible misses plus 12,000 non-eligible requests give the database 16,800 reads per second. The hypothetical database read capacity is 24,000 per second (70% utilised at peak, 19,200 as the 80% operating ceiling). Four shards share the 48,000 GETs, 12,000 each against a hypothetical 25,000 ops/s per-shard budget. A hit costs about 3 ms and a miss about 85 ms, so the mean is 11.2 ms while p99 sits near the miss latency. These are the same figures the other four pages use.

A few terms are used from here on without further introduction. Cache-aside means the Product API checks the cache first and, on a miss, reads the database and writes the result back. TTL (time to live) is the lifetime after which an entry expires. Stale-while-revalidate serves an expired value for a short extra period while one request refreshes it in the background. L1 is a small process-local cache inside each API pod, in front of the shared cache. Cached data falls into four classes with different rules: C1 descriptive content (title, images; TTL 15 minutes, stale-while-revalidate allowed), C2 the price shown on the product card (event invalidation plus a TTL of 300 s ± 20%; always revalidated at checkout), C3 the availability badge (display only, TTL 30 s) and C4 data that must never be served from a TTL-only cache: inventory reservation, payment secrets and authorization decisions.

When cache capacity vanishes, the database inherits the read traffic it was never sized for. The interesting work is deciding who gets served, who waits, and who is told no.

One piece of arithmetic recurs in this chapter. At peak the database has 24,000 − 16,800 = 7,200 reads per second of spare capacity before it hits its limit, but the planned 80% ceiling (19,200) leaves only 19,200 − 16,800 = 2,400 per second of headroom above normal operation. Every outage below is a question of what happens to the demand that exceeds that headroom.

F01 — A cache node dies in the middle of peak traffic

At 20:14 shard B, which owns a quarter of the key space, stops answering. Its replica B′ is also in the failed rack, so there is nothing to promote. (If instead the replicas sit in a different zone from their primaries, a zone loss costs only the failure-detection gap plus the F16 loss window, because a replica is promoted rather than a cold shard rebuilt; the fallback arithmetic below then applies to that short gap rather than to the whole recovery.) Clients see connection errors within a timeout or two, and the Product API's cache-aside code does what it was written to do: treat an error as a miss and read the database. Of the 43,200 hits per second at peak, a quarter, 10,800, were B's. They are now database reads on top of the 16,800 already flowing, 27,600 in total, 115% of the 24,000 capacity (derived, assuming uniform distribution).

Unbounded fallback turns a quarter-cache outage into a full database incident. With a 2,400/s admission budget for the extra misses, the API admits those and sheds or degrades the other 8,400 (78% of the lost hits) by class: stale-while-revalidate content for C1 descriptive data, last-known badge for C3, an honest 'price temporarily unavailable, confirm at checkout' for C2, and refusal rather than guessing for anything in C4. Shedding is chosen on purpose rather than discovered when the database falls over.

F02 — Cold start after a rolling deployment

A platform engineer restarts the cache shards one at a time to pick up a configuration change, and the shards keep their data only in memory. When shard A comes back empty, its quarter of the traffic turns into misses. Hot product 4821 refills with a single fetch; the long tail of millions of objects warms only as shoppers ask for them, so the hit rate on A climbs from 0% along the popularity curve rather than in one step.

Timing decides whether this matters. At the 12,000 req/s average, A sees 2,400 eligible GETs per second and, fully cold, sends all 2,400 to the database: 3,360 + 2,160 extra = 5,520 reads per second, about 23% of capacity (the 90% of A's traffic that was previously hit; derived). At peak the same restart adds 10,800 per second and pushes the database to 27,600, the same 115% as F01. The restart was safe at 03:00 and an incident at 20:00. Restarting a replica first and promoting it, or restoring a snapshot, avoids the empty start altogether.

F03 — Whole cluster cold after a regional restart

After a power event takes down the whole EU cache tier, every shard returns empty at 19:30, just as evening traffic reaches its peak. The cache's job was to absorb 72% of read work; with nothing cached, the eligible 48,000 requests per second all become database reads, 60,000 per second in total, 2.5× the 24,000 capacity. Connections queue, miss latency climbs from 85 ms to seconds, the API's own thread pools fill with waiters, and the cache cannot warm because every fill requires a database read that is now timing out. A cache that is cold at peak does not recover by waiting.

Admission control is the only way out. The database planned ceiling minus non-eligible traffic leaves 7,200 per second for cache-eligible fallback, so 40,800 per second (85%) of eligible demand has to be shed, served stale from an allowed class, or degraded. A practical sequence: admit about 15% of regional traffic (7,200 ÷ 48,000) with priority to the hottest keys, let those fill the cache, then ramp. If every admitted miss were a distinct key, 1,000,000 keys would take about 139 seconds to fill at 7,200 per second (illustrative); because popularity is skewed, the hit rate recovers much faster than a uniform count suggests, but it is a ramp, not a switch.

F09 — Repeated cache outage becomes a retry storm

Shard C flaps: up for ten seconds, down for ten. The client library retries each failed GET three times with no backoff, and every retry that fails still holds a request thread for the 50 ms timeout. A request that would have taken 3 ms now spends up to 200 ms stuck on the cache before it even reaches the database. The service's concurrency is spent waiting on a dependency that was supposed to be optional.

The recoveries are worse than the failures. When C returns, 20 pods reopen a hypothetical 50 connections each, 1,000 at once, onto a freshly started process that is simultaneously rebuilding its memory. Meanwhile the fallback database reads from the previous flap are still draining. The loop feeds itself: flap, retry, queue, flap. The repairs are standard but must all be present: a circuit breaker that stops calling an unhealthy shard, jittered exponential backoff with a retry budget (for example no more than 10% extra attempts), connection ramp-up, and a short cache-client deadline so the miss path starts early.

F24 — Cache and database are slow at the same time

A noisy neighbor on the cache hosts pushes hit latency from 3 ms to 40 ms while a long-running report pushes database miss latency from 85 ms to 400 ms. Nothing is down, so nothing alerts. Little's law shows why the service falls over anyway: at the 60,000 requests per second peak, with eligible reads averaging 0.9 × 40 ms + 0.1 × 400 ms = 76 ms and the other 20% (assumed to read the database) taking 400 ms, the mean request time becomes 0.8 × 76 + 0.2 × 400 ≈ 141 ms. Multiplied by the arrival rate, about 8,450 requests are in flight, roughly 420 per pod across 20 pods, against a hypothetical concurrency limit of 200 per pod (the healthy mean of about 26 ms needs only about 78 per pod). The pods saturate, queueing begins, and callers retry, which raises the arrival rate on an already slow system.

Every layer needs a bound: a 50 ms deadline on the cache call, a database deadline derived from the caller's remaining budget, and a breaker on each. Past that, the service sheds by class. C1 and C3 can go stale for a while; C2 price falls back to 'confirm at checkout'; C4 stays strict. A degraded page that arrives in 150 ms is a better product than a complete page that arrives never. The cascading-failure and load-shedding lessons cover the same ground for general services.

F28 — Service-discovery outage while clients are warm

The control plane that publishes the cache topology (DNS records or a discovery service) goes down for twenty minutes. Nothing breaks at first: running pods keep their connections and their last-known topology, hits keep flowing, and the dashboards look normal. Then a primary fails during the same window. Failover happens inside the cache cluster, but the clients that would normally learn the new address cannot, so they keep sending to the old one. A shard outage (F01) arrives through a side door. (This is the case for clients that take their topology only from the discovery service; a cluster-aware Redis client also learns new owners from -MOVED replies, which is one reason to prefer one.)

Worse, the autoscaler adds three pods for the evening spike. They start with no topology, cannot find any cache, and either fail readiness or run with an all-miss cache that hammers the database. The fix is to make discovery a soft dependency: pods persist a last-known-good topology snapshot and a static seed list, refuse to downgrade to an empty map, and recover topology from the cluster itself when it supports it (a Redis Cluster client may query any node for its slot map). Deploys and scale-ups are paused while discovery is unhealthy.

Expiry is how a cache stays honest, and it is also how a cache arranges for hundreds of identical database queries to arrive in the same instant.

A stampede (or thundering herd) is a burst of simultaneous misses for the same data. For a single key the size of the burst is easy to estimate: key 4821 at 8,640 requests per second with an 85 ms miss path sees 8,640 × 0.085 ≈ 734 requests arrive before the first refill finishes. Single-flight (request coalescing) answers that by letting one caller fetch while the others wait for its result. The three incidents below are about how the protection itself fails.

F04 — Synchronized TTL expiry across many keys

A catalog sync job finishes at 02:00 and warms 6,000 popular product entries in a single pass, all with a fixed 300-second TTL. Five minutes later all 6,000 expire at once. Single-flight does not help: it coalesces requests for one key, and these are 6,000 different keys, each needing its own fetch. If all 6,000 refills land inside one 85 ms miss window, the instantaneous demand is 6,000 ÷ 0.085 ≈ 70,600 reads per second against a 24,000 capacity, nearly 3× (derived, illustrative), and then it repeats every five minutes because the new entries are also synchronized.

Jitter spreads the expiries of a synchronized cohort, the pattern often called a cache avalanche. With TTL = 300 s ± 20%, each key expires uniformly between 240 and 360 seconds, a 120-second spread, so the 6,000 keys expire at about 50 per second rather than 6,000 at one instant. Jitter has limits that are worth stating: it does nothing for one hot key, for a mass purge that deletes everything together, or for a cold cluster, since those synchronize on something other than the TTL.

F05 — The single-flight leader crashes mid-fill

In a flash sale, 5,000 concurrent requests miss on the same expired key. Each of the 20 pods runs a per-process single-flight, so about 250 waiters per pod queue behind one local fetch and the database sees at most 20 fetches instead of 5,000. If you want one fetch for the whole fleet, a distributed lease adds a second layer: the pod that holds the lease fetches, and everyone else waits or polls for the filled value.

Now the leader dies between taking the lease and writing the fill. A local single-flight is the simple case: the process and its waiters die together, and other pods run their own flights. With a distributed lease the other pods' waiters are looking at a lease that nobody is working on. They must not wait forever. Every waiter carries a deadline (for example, a few multiples of the 85 ms miss time), and when it fires the waiter either serves a stale copy if the class allows, fails fast, or, after the lease expires, one waiter takes over as the new owner. At 8,640 arrivals per second on the key, a 250 ms hypothetical lease means up to 8,640 × 0.25 ≈ 2,160 requests have to be handled by the deadline and stale paths while the system waits for the lease to expire. (The diagram above draws a 2-second lease; 250 ms is used here only to keep the arithmetic small, and a lease that long would mean 8,640 × 2 = 17,280 requests to absorb.)

F06 — The lease expires during a slow database fetch

The lease is 250 ms (hypothetical) and the database fetch usually takes 85 ms. A slow query takes 1,000 ms. At 250 ms the lease expires while the original owner is still waiting; a second caller takes it and begins a second fetch; at 500 ms and 750 ms two more do the same. The database now runs four identical slow queries, which makes it slower, which makes more leases expire: positive feedback with the database as the victim.

Two further problems hide in the same trace. The earlier fetch may finish after the later one and overwrite it with data read earlier. And a lock without a fencing check cannot tell the cache that its holder lost ownership. The practical defense is layered: lease length comfortably above the p99 fetch time, one renewal at most, a cap on concurrent fetches per key, and writes that are versioned so that an older read cannot replace a newer one (the same conditional fill that fixes F07). See distributed locks for why a lease cannot, by itself, provide mutual exclusion.

These are the incidents where the cache is healthy, fast, and wrong. Nothing alerts on a hit that returns the previous price.

Invalidation means removing or replacing a cached entry because its source changed. Meridian publishes a post-commit event from a transactional outbox into an invalidation stream, and a subscriber with authority deletes or refreshes the key. The event does not edit caches itself. For C2 (the product-card price) this is paired with a TTL backstop of 300 seconds ± 20%, so the worst case when an event is lost is about 360 seconds, and checkout always revalidates against the authoritative service.

F07 — A delayed fill overwrites a fresh invalidation

Reader A misses on key 4821 at t = 0 and reads the database: $99, version v5. Before A's cache write, the merchant commits $109, version v6 at t = 5 ms, and the outbox event reaches the subscriber, which deletes the cache key. There is nothing to delete, because A has not written yet. At t = 30 ms A finishes and SETs v5 into the cache with a fresh TTL. The invalidation arrived, was honoured, and was then undone by a slower request. The cache now holds v5 for up to 240–360 seconds while the database says v6.

Deleting twice or after a delay narrows the window but does not close it. The reliable fix is a versioned conditional fill: the value carries recordVersion from the authority, and a fill succeeds only if the cached version is lower than the incoming one. There is a trap: if the invalidation simply deletes the key, the key is absent and the late v5 fill is accepted anyway. So the invalidation must leave a tombstone, a small marker that records v6 and lives for a short time (longer than the slowest plausible fill), and the conditional fill compares against it. In Redis the compare-and-set is an atomic script on the owning node. Memcached has no compare-by-version command, so the version is embedded in the value and a gets/cas loop does the comparison (cas succeeds only if the item is unchanged since it was read), with the same tombstone idea. Versions must come from the authority (a database row version or commit sequence); timestamps from application servers can disagree. Alternatively accept the bounded staleness explicitly and rely on the TTL, which is a legitimate choice for non-critical content but must be a choice.

F08 — Invalidation event lost, replayed or reordered

The invalidation stream is at-least-once at best, and nothing in it is exactly-once. A lost event means the key keeps its old value until TTL. A replayed event for v5 arriving after the cache already refilled v6 deletes a fresh entry, which costs one extra miss and nothing else. The dangerous case is reordering when the subscriber updates in place instead of deleting: an update for v5 arriving after v6 overwrites the newer value.

The design consequences are cheap but must be chosen: prefer delete-on-invalidate, ideally leaving a short-lived versioned tombstone (idempotent and safe to replay), keep a per-key version so any in-place write is conditional, track consumer lag, and keep the TTL backstop for the lost-event case. A periodic reconciler that samples keys and compares versions turns 'hopefully no events were lost' into a measured rate.

F21 — A stale price after a database update; checkout must revalidate

At 11:00 Meridian drops 4821 from $99 to $109 for a supplier cost change. The invalidation is lost to a consumer restart. Product cards keep showing $99 until the TTL expires, between 240 and 360 seconds later. A shopper adds the item to the cart at $99. At checkout the authoritative service computes $109 and the shopper sees the difference before paying. That step is the real protection: the card price is a display, and the order total is computed by the authority.

The cost is trust, not correctness. A shopper who sees a price change at checkout may leave, and some jurisdictions have rules about advertised prices; Meridian's legal and product teams, not the cache design, decide the acceptable window. For items where even a few minutes is too long (flash prices, regulated pricing), shorten the TTL, push updates through a refresh rather than a delete, or exclude price from the card cache altogether and pay the latency.

F22 — A revoked permission is served from cache

A support agent's access to Cedar's order data is revoked at 15:00. The authorization service's answer 'allowed' was cached for 300 seconds with the rest of the session context. For five minutes the agent can still read orders. This is why authorization decisions belong to class C4: they are never served from a TTL-only cache. Staleness here is not a freshness inconvenience, it is a security failure.

Two safe patterns exist. Re-check the authority on each sensitive action (the cache then only saves work on non-sensitive reads), or cache decisions against a revocation epoch: each cached grant records the epoch it was issued under, the caller reads the current epoch from a small, highly available authoritative source, and a mismatch forces a re-check. If the authority is unreachable, the system must fail safe for sensitive actions: deny or require re-authentication, never 'serve the stale permission'. Stale-while-revalidate does not apply here.

F23 — Regional lag and residency restrictions

EU and US each run their own cache tier in front of one product database. A price change commits in the EU. The EU invalidation lands in seconds; the US invalidation crosses a link that is lagging at about 40 seconds (hypothetical), then the US consumer retries a failed delivery. For that window a US shopper sees v5. If the cross-region link is down, the TTL backstop (≤ ~360 s) still bounds it per region, which is why every region keeps its own TTL instead of depending on the event.

A second failure mode is residency. If US pods fill their cache from an EU-only dataset (merchant tax data, say) because a read path was written for one region, restricted content is now stored where it is not permitted. Region must be part of the key and the fill rule, not a deployment detail. 'Instantly consistent across regions' is not a property the design offers, and a design that claims it needs a very different cost and latency contract.

A cache has a fixed amount of memory and an unbounded appetite for data. Eviction decides what to forget, and each policy has an access pattern it handles badly.

Eviction removes entries when memory is full. LRU (least recently used) removes what has gone longest without access; LFU (least frequently used) removes what is accessed least often. Redis implements LRU and LFU as approximations (LRU by sampling keys; LFU with a probabilistic counter that decays), so the behavior below is a tendency, not an exact algorithm. Remember the memory worksheet: 1,536 bytes of payload per object becomes about 1,640 bytes with key and metadata, and 15.27 GiB across the cluster once a 1.25 allocator factor is applied (all overheads assumed). The sample cluster is 4 × 8 GiB = 32 GiB nominal.

F12 — LRU thrash when the working set outgrows memory

A sale multiplies the number of active variants per product, and the active working set grows to 2.5× the planned 15.27 GiB, about 38.2 GiB, against 32 GiB of nominal memory. Under LRU, entries are evicted a moment before they are requested again, and hit rate decays from 90% toward 70% or lower. At 70%, eligible misses are 14,400 per second, database reads 26,400, which is 110% of capacity; mean latency moves from 11.2 ms to about 27.6 ms (0.7 × 3 + 0.3 × 85) and p99 stays near the miss latency.

The cache did not break; the model did. The capacity plan assumed an 8,000,000-object working set. Practical responses are to add shards (a resharding exercise in itself), shrink payloads through compression or smaller views, shorten TTL on cold content, and be selective about what is cached at all: a key that is read once costs memory without saving a read. Hit rate, eviction rate and working-set growth should be watched together.

F13 — Never-expire data starves the evictable space

A batch job writes enrichment data with no TTL, about 3.6 GiB per shard on its first run (derived: it takes the 8 − 4.39 GiB the hottest shard had left), and each later run adds more under new keys. With an allkeys-* policy this is survivable, because every key, including the job's, is a candidate for eviction. With a volatile-* policy it is not: those policies consider only keys that have an expiry. Every normal fill is then paid for by evicting the cache's own entries, so the cache shrinks to whatever the job leaves, hit rate sinks, and once the non-expiring data alone reaches the memory limit there is nothing evictable, the policy behaves like noeviction, and writes fail with an out-of-memory error. The cluster looks full of data nobody remembers writing.

Two defences are cheap. Make a TTL mandatory in the cache client (reject or default a missing TTL), and monitor the share of keys without an expiry. In memcached an exptime of 0 means never expires, but the protocol notes such an item may still be removed to make room, so the exposure differs by engine and version; check the documentation for the one you run.

F14 — Oversized values and compressed payload bombs

A new feature caches the full product page, images inlined, at about 1 MiB per entry, instead of the 1.5 KiB card. Meridian's policy rejects values over 1 MiB, so values of exactly that size slip through. At 8,640 requests per second for the hot key, one shard must send 8,640 MiB per second, about 8.4 GiB per second, over a hypothetical 10 Gbit/s link of about 1.16 GiB per second: 7.2 times the capacity of the NIC. A single hot key is enough to saturate a node's network before CPU or memory notice. Large values also block other commands on a single-threaded engine while they are serialized, and they inflate garbage collection and memory in the pods that decode them.

The second hazard is compressed content: a small stored value that expands to something enormous when the client decompresses it. The wrapper Meridian stores around every cached payload (its envelope: schema version, record version, expiry, compression flag, bytes) includes a compression field and the envelope should also record the uncompressed length so clients can refuse to inflate beyond a bound. Protect at the boundary: cap sizes on write, cap decompressed sizes on read, and never put the large body in the shared cache when a reference or a CDN can carry it.

F26 — Eviction quality collapses under a looping access pattern

A nightly job walks the product catalog in id order through the cache-aside path, and a downstream consumer does the same every fifteen minutes. A loop over slightly more items than fit is the classic worst case for strict LRU: each item is evicted just before the loop comes back to it, so the hit rate approaches zero for that traffic while the loop also pushes out the hot set that real shoppers need.

Different policies fail on different patterns. LFU keeps frequently used keys through a one-time scan but is slow to forget keys that were popular earlier, which is why it has a decay setting. Random eviction ignores access order and does tolerably on near-uniform access. The most effective single measure is to keep scans out of the cache: batch consumers read the database or a replica directly, or mark their reads as non-cacheable. Choose a policy against a real trace of your access pattern, not against folklore.

Sharding spreads keys, not requests. One very popular key lives on exactly one shard, however many shards you add.

A hot key is a key whose request rate is far above its peers. The planning budget is 25,000 operations per second per shard (hypothetical); an even spread of the 48,000 peak GETs puts 12,000 on each of four shards, 48% utilised. The incidents below break that comfortable arithmetic.

F10 — A hot key concentrates request CPU on one shard

The top product takes 18% of eligible peak traffic, 8,640 requests per second, all addressed to one key and therefore one shard. The other 39,360 eligible GETs spread over four shards at 9,840 each, so the hot shard carries 8,640 + 9,840 = 18,480 per second (73.9% of budget) while the others sit at 9,840, a 1.88× skew. It still fits, but a trigger that was acceptable at 73.9% has no room for a failover, a snapshot or a deploy on that node. (The 8,640 here is the hot-key rate, unrelated to the 8,640 hits/s at average load; the numbers coincide.)

Adding shards changes nothing for the hot key. What helps is serving it from somewhere other than its owning shard: a tiny per-pod local cache (L1, a process-local cache in front of the shared one) with a 2-second TTL means 20 pods each fetch the key at most every two seconds, 20 ÷ 2 = at most 10 shard GETs per second for that key instead of 8,640. The price is up to two seconds of staleness on that key and an invalidation path that must reach every pod's L1. Acceptable for a display price with checkout revalidation; not acceptable for anything in C4.

F11 — A trending SKU overloads its shard and then stampedes

A viral video sends 45% of eligible peak traffic to one product: 21,600 requests per second. Its shard now serves 21,600 + (48,000 − 21,600) ÷ 4 = 28,200 operations per second, above the 25,000 budget, while the other three shards idle at 6,600. Latency on the hot shard rises, other keys on that shard suffer alongside the trending one, and clients that time out retry, which adds load.

The overload alone is bad. Add expiry and it gets worse: at 21,600 requests per second and an 85 ms refill, about 1,836 requests arrive before the first refill finishes, and if the owner of the refill is itself slow they pile up. Layered defense works best here: the L1 from F10, single-flight per pod (at most 20 fetches fleet-wide), a distributed lease if the database cost of 20 is still too much, and stale-while-revalidate for descriptive content so the value never looks absent. Price on the card can use the L1 as well, with checkout still revalidating.

F25 — Skewed distribution from a poor shard hash

An engineer 'optimizes' shard selection to keep a tenant's keys together by hashing only the tenant prefix. Cedar, 70% of traffic in this hypothetical, now maps entirely to one shard: 0.7 × 48,000 = 33,600 GETs per second against the 25,000 budget. This is the same hazard as a Redis hash tag such as {tenant:cedar}, which forces every key with that tag into one slot (860 in the sample), and it is the right trade when you need multi-key operations on a tenant and the wrong one when the tenant is hot.

Distribution problems also come from weak hash functions (hashing only the last digits of numeric ids, or using key length) and from too few virtual nodes on a ring, where one node can own noticeably more than its share (a toy ring of 100 positions with one token per shard can end uneven: ownership of 35/15/30/20 across A, B, C and D after adding D). The test is mechanical: hash a sample of real keys offline and chart per-shard counts and per-shard request rates before deploying a new scheme.

A shared cache is shared memory, shared CPU and a shared namespace. Each of those can be consumed or confused by a tenant who did not mean to.

Isolation in a cache is made from three separate things: a key namespace that cannot collide, resource limits that cannot be exhausted by one tenant, and access control tied to verified identity. Naming conventions alone provide none of the three.

F15 — Negative-cache poisoning and random-key misses

A negative cache stores 'not found' for 30 seconds so repeated lookups for a missing product do not reach the database. Two attacks work against it. In the first, a scraper (a cache penetration attack) requests random product ids at 5,000 per second (hypothetical). Every id is new, so every request misses, the negative cache never gets a repeat, and 5,000 reads per second pass straight to the database. The 5,000 × 30 = 150,000 negative entries that accumulate are a smaller problem for memory than the reads are for the database (even at the full 1,640-byte footprint they are about 0.23 GiB, 1.5% of the 15.27 GiB plan, and a real 'not found' marker is smaller), but at a higher rate or a longer negative TTL they begin to displace useful entries.

In the second, poisoning, a product is created at 10:00:05 but a negative entry written at 10:00:02 says it does not exist, so shoppers see 'not found' until the 30-second TTL ends. And if a 403 or 401 is cached as 'not found' across identities, one user's lack of permission becomes everyone's missing product. Defences: validate id shape and existence cheaply (for example against a bloom filter or id range) before the database, rate-limit by caller, cap negative entries per tenant, delete a negative entry when the product is created, and key negative entries by tenant and, for authorization failures, by principal.

F19 — One tenant dominates shared memory and operations

Cedar imports its full catalog and a scraper touches four million objects once each. At 1,640 bytes and a 1.25 allocator factor each, that is about 7.6 GiB, half of the 15.27 GiB planned for the whole cluster (derived). Under LRU, Birch's moderately warm entries are older than Cedar's fresh inserts and are evicted first. Birch's hit rate falls, its latency rises, and Birch did nothing.

A single engine instance has one memory pool, so per-tenant limits must be built from other pieces: accounting by key prefix in the client with a write quota, rate limits per tenant at the API, a cap on one-touch inserts, and, for tenants who justify it, a dedicated shard tier so their entries never share eviction with a neighbor. A shared pool is cheaper and more flexible, and its price is exactly this class of interference.

F20 — A cross-tenant key collision serves the wrong tenant's content

A new endpoint builds its cache key as product:4821 instead of the schema's v2:<tenant>:product:4821. Cedar and Birch both have a product 4821, and they are different products. Cedar's shopper loads the page first and fills the key. Birch's shopper hits the same key and is shown Cedar's product, price, and perhaps private supplier notes. The cache did exactly what it was told; the name had no tenant in it. The unscoped key also hashes to a different Redis Cluster slot (13,107) than the simplified tenant-scoped form tenant:cedar:product:4821 (8,298; the full v2:-prefixed key would land on a third slot), so the leaked entry lives apart from the correct one.

This is a security incident, not a performance bug. The content is no longer private to its owner and the exposure window is the TTL of every colliding entry. Prevention is structural: one key-builder function is the only way to produce keys, it takes the tenant from the authenticated identity rather than from a request header, a CI check rejects direct cache calls with literal keys, and tests run two tenants with the same ids. Where responses depend on the caller (permissions, personalised content), the key must include whatever the response varies by. Access control lists on the cache itself restrict which service can touch which prefix; naming conventions do not.

A replica gives you a warm standby and read capacity. It does not give you a promise that the standby has everything the primary acknowledged.

Redis replication is asynchronous by default: the primary answers the client without waiting for replicas, and the replicas acknowledge the offset they have processed in the background. The Redis Cluster documentation says plainly that acknowledged writes can occasionally be lost, and that the windows are larger for clients in a minority partition. WAIT can block for replica acknowledgements, but it lowers the probability of loss; it does not make the system strongly consistent.

F16 — Failover promotes a lagging replica

The primary of shard C acknowledges SET product:4821 $109 v6 and fails before the write reaches the replica. The replica is promoted, holds v5 ($99), and the acknowledged write is gone. In a cache this is usually ordinary: with replica lag of 120 ms (hypothetical) and about 1,200 fills or deletes per shard per second at peak (4,800 eligible misses over 4 shards), roughly 144 operations sit inside the window at most: the replica keeps whatever had already reached it, so the real loss is whatever was still unsent when the primary died. The subtle ones are the deletes. A lost invalidation DEL means the stale value that the primary removed is back on the promoted replica, and no further event will remove it.

For a cache that is acceptable only because the authority can repair it. After a failover, replay the invalidation stream from before the failure time minus the lag plus a safety margin; the TTL backstop covers anything missed. Versions in the envelope let replays be idempotent. The picture changes completely if the cache held the only copy of something, such as a cart or a session. Then those 144 operations are lost business data, which is why such state belongs in a durable store, and why write-behind caches need explicit durability design. Do not describe the design as 'no acknowledged-write loss'.

F27 — Replica catch-up pressure after failover or restore

After a failover, a fresh replica must attach to the new primary. If the replication backlog still covers the replica's position, a partial resync (using the replication ID and offset) is cheap. If not, a full resync is needed: the primary produces a snapshot, ships it, and then streams the changes made meanwhile. On a primary at 4.39 GiB of an 8 GiB budget (54.9%) that already serves peak traffic, snapshot creation, copy-on-write memory growth during heavy writes, and replication buffers all compete for the remaining headroom, and a primary that runs out of memory during resync becomes the next outage.

A restore after the F03 restart has the same shape on a larger scale. Rebuild replicas one at a time, off the shard's busiest hours when possible, leave memory reserved for buffers (the Redis eviction documentation recommends leaving room for replication and persistence), and accept running a shard without a replica for a short, visible period instead of risking the primary. Choosing between 'redundancy sooner' and 'primary safer' is an operator decision that a runbook should settle beforehand.

Every request must find the shard that owns its key. When the clients' idea of ownership drifts from the cluster's, reads miss and, worse, invalidations go to the wrong place.

Modulo hashing, consistent hashing and Redis Cluster's 16,384 fixed hash slots are different mechanisms. Moving from N to N + 1 nodes with modulo changes the owner of N ÷ (N + 1) of keys (75% from 3 to 4, 80% from 4 to 5), while a balanced ring moves roughly 1 ÷ (N + 1). Redis Cluster uses a slot table with explicit migration, answering -MOVED (permanent: update your slot map) or -ASK (one-time during migration).

F17 — Partition and stale client routing

Pods are being deployed. Half run a client configured with a 4-node server list; half run a new config listing 5. Under modulo placement, 80% of keys have a different owner depending on which pod asks. A pod on the new list refills key 4821 on the fifth node; the invalidation consumer, still on the old list, deletes it from the fourth. The fifth node keeps v5 until its TTL, and nothing is wrong from any single pod's point of view.

A real network partition adds the Redis Cluster case: a primary stranded with a minority of the cluster keeps accepting writes from clients that can still reach it until it notices the partition (after the node timeout) and refuses them, while the majority side promotes a replica. Writes in that window are lost. With a cluster-aware client, MOVED responses repair the map and the client should refresh it; with static client-side hashing, nothing repairs anything. Treat topology as a versioned object: one source, a version in every client, and a skew metric across pods.

F18 — Resharding concurrent with writes

Slots 12288–13653 are migrating from C to the new shard D; key 4821, in slot 13107, is inside the range. During migration the source answers for keys it still holds, and for keys already moved it replies -ASK, which the client follows once by sending ASKING followed by the command to the target. An invalidation consumer that treats any non-OK reply as a failure and gives up, or an old client that does not handle ASK at all, silently skips the DEL. The key then lands on D with v5 while the database holds v6.

With a client-side ring and dual-write migration the problem is duplicates: both old and new owners hold a copy, and an invalidation that reaches only one leaves the other to be read after a rollback or a boundary shift. The rules are the same either way: invalidation consumers must follow redirects and treat an unconfirmed delete as a retry, a migration must be observable (slots moved, keys remaining, errors), and the TTL backstop plus versioned fills handle whatever slips through. A key never teleports: after a slot moves, exactly one node owns it.

The last two incidents are about the systems around the cache: the telemetry that should reveal a failure, and the deploys that cause one.

F29 — A metrics outage hides a hit-rate collapse

A key-builder change ships at 19:00 with a bug that sends part of the lookups to the wrong key, and the hit rate drops from 90% to 60%. At peak, misses become 19,200 per second, database reads 19,200 + 12,000 = 31,200, which is 130% of capacity, and mean latency becomes 0.6 × 3 + 0.4 × 85 = 35.8 ms. The metrics pipeline is down for an unrelated reason, so the dashboard that would show the hit rate is blank, and the alert based on it cannot fire. A blank panel looks calm.

Detect it independently of the thing that failed. The database reports its own read rate, the load balancer reports request latency, and a synthetic probe asks for a canary key and a known-missing key every few seconds and records hit/miss from the response. 'No data' on a critical alert should itself page, because silence is also a signal. The principle for a cache is that its health must be provable from a vantage point outside it.

F30 — Key-schema migration with incompatible serializers

Meridian changes the value encoding and bumps the key schema version from v1: to v2: for a rolling deployment. Because the version is the first component of the key, new pods read only v2: keys and old pods read only v1: keys: neither can deserialise the other's values. The price is a gradual cold start: the v2: space is empty at the beginning, so each upgraded pod starts with misses, and in a fast rollout at peak the F02 arithmetic applies. Had both versions shared one key space, the failure would be worse: exceptions when an old pod tries to decode a new envelope, or silent corruption.

Rollout safely in two steps. First ship a reader that understands both versions and treats an unknown schemaVersion as a miss; then, once all pods run it, switch writes to v2:. Both key spaces then coexist until the v1: entries expire, so in the worst case the hot set is held twice, 15.27 × 2 = 30.5 GiB, 95% of the 32 GiB nominal: very little headroom, so watch evictions during the overlap or migrate in stages. A rollback is clean because the old keys are still there until their TTL.

The incidents repeat the same few questions: is the load shifting to the database, is the cache telling the truth, and can a failure grow beyond the shard it started in? An operating setup is the set of signals that answer them.

• Kill one shard and its replica at peak load in a test environment and confirm the fallback cap holds the database at or below 19,200 reads per second while excess demand is shed by class (F01).
• Restart the whole tier empty at peak and rehearse the ramp: admit ~15%, warm hot keys first, then raise admission in steps (F03).
• Replay the F07 interleaving with a delayed fill and confirm the conditional fill rejects v5 after v6.
• Fail over a primary with simulated lag and verify the invalidation replay window covers it (F16).
• Run two tenants with identical product ids through every cache path and assert no cross-tenant hits (F20).
• Disable service discovery during a failover drill and watch whether pods keep their last-known topology (F28).
• Turn off the metrics pipeline and check that independent signals still page someone (F29).

• Treating a cache error as a plain miss with no admission limit.
• Using a TTL as the only mechanism for permissions, inventory ownership or anything in C4.
• Believing a hit means fresh, complete or authorized.
• Calling an asynchronous replica 'durable' or claiming the design never loses an acknowledged write.
• Retrying cache calls immediately and without a budget.
• Letting a shared pool run without per-tenant visibility.