Distributed Cache Architecture: Sharding, Eviction, Replication & Consistency

A cache service looks trivial from the outside — get a value, put a value, delete a value — but each verb hides a decision about what the caller may assume. Writing the contract down first is what keeps later chapters honest.

Every name here belongs to a fictional teaching company, Meridian Commerce, a B2C storefront. Its source of truth is a relational product database; its Product API runs as a hypothetical 20 pods; the hot item is product:4821, and the large merchant Cedar and well-behaved merchant Birch share the cache. All rates, sizes and latencies are illustrative inputs, not measurements. A *cache* is a fast, memory-resident, derived copy of data that lives authoritatively somewhere else; a *hit* means the copy was found, a *miss* means it was not and the caller must go to the authority.

Two real engines are referenced throughout as comparison points. Redis is an in-memory data-structure server that can run as a single node, as a primary with replicas, or as a sharded *Redis Cluster*; Memcached is a simpler in-memory key-value cache that does no replication or clustering of its own. They are examples, not a shared feature set. The HTTP surface below is deliberately illustrative: Redis speaks its own binary-safe protocol (RESP) and Memcached has text and binary protocols; neither exposes this REST shape. Teaching it as REST lets us discuss status codes, headers and limits without tying the lesson to one engine. A real deployment would put this contract in a client library that speaks the engine's protocol.

Three details carry most of the weight. First, a miss is a normal answer, so it travels as a 404-style result that the caller must handle with a fallback rather than as an exception. Second, a write can carry ifVersion, the record version taken from the database; the cache refuses to replace v6 with v5 (chapter 4 shows why). Third, every response reports the remaining TTL, so a caller can decide whether the value is fresh enough for *its* purpose — a product card tolerates minutes, a checkout total tolerates none.

Size limits belong in the contract too. Meridian's illustrative policy is: reject values over 1 MiB and keys over 250 bytes. The key figure is borrowed from Memcached, whose protocol documentation limits keys to 250 characters; Memcached's maximum item size is configurable and by default about 1 MiB, but check the value for the version you run. A cache that accepts arbitrarily large values lets one buggy writer evict everyone else's data.

A key is both a name and a routing instruction, and the value is more than the bytes: the metadata wrapped around it is what lets a reader decide whether it may use what it found.

The schema version comes first because two deployments will overlap during a rollout: pods on the new serializer write values that pods on the old one cannot read, and the reverse. If the version is part of the key, the two generations simply never see each other's entries and the cold start of a new version is a deliberate, observable event rather than a deserialization storm. The cost is a one-time miss wave when the version bumps, which is why rollout plans warm or ramp traffic gradually.

The tenant segment gives every merchant its own namespace, which makes per-tenant quotas, purges and metrics possible. It is a naming convention, not a security boundary: chapter 13 explains why identity and access control must be enforced separately, and why the tenant segment must come from the verified caller, never from request input.

The envelope

recordVersion is the field that makes the rest of the design possible. It comes from the authority: a monotonically increasing number the database bumps on each committed change to the row (v5 is the $99 card, v6 is the $109 card). A cache must never mint it, because a cache cannot know the order in which the database committed things. createdAt and expiresAt are for debugging and for staleness metrics; the engine's own TTL still performs the actual expiry. provenance lets an incident responder say which build wrote a bad value.

Compression is chosen per value, not globally. Gains are workload dependent, and decompression is attacker-influenceable work, so Meridian caps the *decompressed* size as well as the stored size to avoid decompression bombs. Sensitive data is not welcome in the envelope at all: no payment instruments, no secrets, and no unredacted personal data in a store whose access model is a shared password. (Where an entry does need to be tied to an identity, store a reference or a redacted projection, and keep the personal fields in the database.)

• Derive keys from verified identity and a fixed schema, never by concatenating raw request strings; reject keys over the length cap.
• Store the authority's version, and compare versions rather than timestamps when two writers race.
• Bound the decompressed size as well as the stored size.

• Using wall-clock `createdAt` to decide which value is newer. Pods' clocks disagree.
• Putting the tenant in the key and calling it isolation.
• Caching serialized objects with language-native formats that execute code on load; use a plain data format.

Cache-aside, also called lazy loading, puts the application in charge: look in the cache, and only on a miss ask the database and then store the answer. Everything difficult about it lives in the word *only* — what happens when many callers miss together, or when the database is slow.

At the illustrative numbers, a hit costs 3 ms and a miss 85 ms. With a 90% hit rate on cache-eligible reads the mean is 0.9 × 3 + 0.1 × 85 = 11.2 ms, but the p99 (the latency that 99% of requests beat) sits near the miss cost, around 85 ms. The mean looks wonderful and still hides a tail that one in ten shoppers lives in. Design for the tail.

(1) The cache call has its own short deadline. A cache that answers slowly is worse than one that is down, because every request would wait for it; on timeout the code treats the result as a miss and moves on. (2) A negative cache entry ("this product does not exist", TTL 30 s, capped per tenant) stops repeated lookups of ids that are not there, such as a crawler walking ids. It is a separate cache class from real products on purpose. (3) single_flight is the stampede brake. Callers that miss on the same key in the same process share one fetch instead of each running one; chapter 9 covers its limits. The wait is bounded: a caller that cannot get an answer in 120 ms returns an error or a stale-but-allowed value rather than hanging.

(4) The database read carries a deadline and passes through a circuit breaker, a switch that stops calls to a failing dependency so that it can recover. Without the bound, a slow database turns into a queue of waiting requests, then exhausted threads, then an outage that began as mere slowness. (5) The fill is conditional: put_if_newer refuses to overwrite a stored v6 with a v5 payload that was read earlier. That line fixes the race the next chapter walks through. The TTL is the 300 s base with ±20% jitter, spreading expiry across 240–360 s.

On a write the authoritative database is updated first and the cache is told afterwards. The reverse order can leave the cache holding a value the database never had. Even in the right order there is a race that every cache-aside system has to deal with.

Take Meridian's price change. A shopper sees $99 (record v5). An operator commits $109 (v6). The cache still holds v5 and will keep serving it until something removes it. The correct trigger is a *post-commit event*: the writing transaction also inserts a row into an outbox table, a relay publishes it to an invalidation stream, and a subscriber with the authority to do so deletes (or refreshes) the cache key. An event does not magically edit caches; the subscriber does, and it can fail, lag, or deliver twice.

Why an outbox rather than calling the cache directly after the commit? Because a process can crash between the commit and the call, and the invalidation is then lost forever; the database says v6 and the cache says v5 with nothing pending to correct it. Writing the event row inside the same transaction removes that window. The relay might publish an event twice, so the subscriber's operation must be idempotent: deleting a key twice, or deleting only if the stored version is older than the event's, is harmless. See the outbox pattern for the mechanism in general.

The refill race

Now the subtle part. Reader R1 misses, reads v5 from the database and is delayed (garbage collection, a slow network). The operator commits v6; the subscriber deletes the key. R1 wakes up and stores its v5 payload. The cache now holds an old value *after* the invalidation that was supposed to remove it, and nothing will delete it again until its TTL. Delete-after-commit alone does not prevent this; TTL only bounds it (here up to about 360 s for the price).

Two remedies exist. Version-aware fill: the cache stores recordVersion and refuses writes with a lower number. This needs the key to remember something after deletion, either a short-lived *tombstone* carrying the version or an invalidation that writes the new version in place of deleting. The version must come from the database; a cache-generated counter would just number the race. Accept bounded staleness: for a low-risk class, such as a product description, let the TTL cap the exposure and skip the machinery. Meridian does the first for C2 prices and the second for C1 descriptive content.

An entry leaves a cache for one of two reasons that are easy to confuse: its time ran out (expiry) or the memory ran out (eviction). They are driven by different policies, answer different questions, and fail differently.

A *TTL* (time to live) is a per-entry lifetime after which the entry must not be returned. It bounds staleness; it does not synchronize anything with the database. Meridian assigns lifetimes by data class: C1 descriptive content 15 minutes ±20%, with stale-while-revalidate (serve the old value briefly while one caller refreshes it) allowed for up to 5 more minutes; C2 price 300 s ±20% plus event invalidation; C3 availability badge 30 s; negative entries 30 s. Reservations, payment instruments and authorization decisions are never served from a TTL-only cache.

Lazy and active expiry

In Redis, expiry has two mechanisms, both documented. Passive (lazy) expiry checks the TTL when a key is accessed and deletes it if it has elapsed. Active expiry periodically samples keys that have a TTL and deletes the expired ones, so keys nobody reads again do not sit in memory forever. The consequence is that an expired key may briefly occupy memory but is never returned. Redis also centralises expiry on the primary: it synthesises a delete command that goes to replicas and to the append-only file (AOF, its optional write log used for persistence), so replicas do not run their own TTL arithmetic. Memcached's protocol states that a non-zero expiry is guaranteed not retrievable after it elapses, and that an item with expiry 0 may still be removed to make room for other items. Treat TTL as an upper bound on how long a value is *served*, not a promise about when memory is freed.

Eviction when memory is the limit

Redis sets a maxmemory limit (by the documentation, no limit by default on 64-bit builds) and, when it is reached, applies a configured policy: noeviction (writes fail), allkeys-lru, allkeys-lfu, allkeys-random, and volatile-lru, volatile-lfu, volatile-random, volatile-ttl, where the volatile variants only consider keys that have a TTL — and with no such keys behave like noeviction. Redis 8.6 and later add least-recently-modified (allkeys-lrm, volatile-lrm) variants, which refresh a key's timestamp only on writes; check the version you run.

Redis's LRU and LFU are approximations. LRU samples a handful of keys (maxmemory-samples, 5 in the documentation) and evicts the best candidate among them rather than keeping an exact global order, which would cost memory and time. LFU uses a small probabilistic logarithmic counter with a decay period (lfu-log-factor, lfu-decay-time). The documentation calls allkeys-lru a good default if you have no reason to prefer another. Walk a trace to see why the choice matters: with a bestseller read 100 times a minute and a crawler reading 10,000 distinct products once each, LRU lets the crawl push the bestseller out and LFU does not, whereas for a flash sale whose popularity appears abruptly LFU lags until counts build.

Memcached memory is organized differently

Memcached allocates items into slab classes, size buckets with pages assigned to each. An item lives in the smallest class that fits, which wastes some space inside the chunk but avoids general-purpose allocator fragmentation. Eviction is per slab class using an LRU, and the protocol exposes slabs reassign and slabs automove for moving pages between classes, documented as subject to change, plus an LRU crawler that reclaims expired items. The practical lesson: if object sizes shift (say, a new serializer makes values bigger), memory may sit in the wrong slab class and evictions can occur in one class while others have free pages. Redis instead uses a general allocator with configurable policies and has its own fragmentation to watch. Neither engine's behavior should be assumed from the other.

A *shard* is one slice of the keyspace owned by one node. The whole design question is how a key finds its owner, and what happens to the mapping when the number of nodes changes.

Modulo hashing

The obvious rule is owner = hash(key) mod N. It is deterministic and needs no table, but changing N reshuffles almost everything: going from N to N+1 changes the owner of N/(N+1) of keys, so 3 to 4 nodes remaps about 75%, 4 to 5 remaps 80%, and 10 to 11 remaps roughly 91%. Each remapped key is a miss on its new owner, so adding capacity triggers a stampede on the database at the worst moment.

A twelve-key illustration (hash values 17, 22, 35, 41, 58, 64, 70, 83, 96, 101, 112, 120) sees 10 of 12 change node when going from 3 to 4. That 83% is a small-sample figure; the expected value is 75%.

Consistent hashing with virtual nodes

*Consistent hashing* places nodes and keys on a ring; a key belongs to the first node position clockwise from its hash. Adding a node takes over only the arcs next to its positions, so about 1/(N+1) of keys move when the ring is balanced. Because one position per node gives lumpy arcs, real implementations give each node many virtual nodes. Here is a toy ring with hash space 0–99 and illustrative values, not a real hash function: tokens A{10, 60}, B{25, 80}, C{40, 95}. The key product:4821 hashes to 33, whose next token is 40, so it lives on C. Add D{20, 70}: D takes (10, 20] and (60, 70], both previously B's, so 20 of 100 positions move, all from B, and 4821 stays on C. The result is A 35, B 15, C 30, D 20 — uneven, which is exactly what few virtual nodes cause. See consistent hashing for more.

Redis Cluster hash slots

Redis Cluster does something else. It has 16,384 fixed hash slots; the slot is CRC16(key) mod 16384 (the XMODEM CRC16 variant), and each primary owns a set of slots. Moving data means moving slots between nodes, explicitly. This is *not* consistent hashing, and the two are not interchangeable: the ring maps by position and moves arcs implicitly, while slots are a fixed table moved by an operator or planner.

With three primaries split evenly, A owns 0–5460, B 5461–10922 and C 10923–16383. The plain key product:4821 computes to slot 13107 and so lives on C; the tenant-scoped tenant:cedar:product:4821 computes to slot 8298 and lives on B. Adding D by migrating 4,096 slots (25%: 1,365 from A, 1,365 from B, 1,366 from C, an illustrative plan) moves product:4821 from C to D while the Cedar key stays on B.

Hash tags and their cost

If a key contains a non-empty {...} section, only the text inside the first braces is hashed. {tenant:cedar}:product:4821, {tenant:cedar}:product:4822 and {tenant:cedar}:price:4821 all land on slot 860, so multi-key commands over them are allowed, because Redis Cluster requires all keys of a multi-key command to be in the same slot; otherwise the node rejects the command with a cross-slot error. The price is concentration: a tag on the whole tenant puts the whole tenant on one slot and therefore one shard, so a hot tenant becomes a hot shard, and a slot is the smallest unit of migration.

Once data is spread across shards, someone has to send each request to the right one. Two designs dominate, and either way the routing table is eventually stale.

A smart client keeps the topology (which node owns which slot or ring arc) and connects directly: one network hop, but every language client must implement discovery and redirects correctly. A proxy layer hides topology behind one endpoint: clients stay simple and topology changes are central, at the cost of an extra hop, a new tier to scale and keep available, and a new place for connection exhaustion. Redis Cluster's design has no proxies; a client may send a query to any node and be redirected. Memcached deployments typically rely on client-side hashing across servers, since the engine itself does not cluster or replicate (check the documentation for your version).

MOVED, ASK and the migration state machine

While Redis Cluster migrates a slot, the source node is in a *migrating* state and the target in *importing*. The source answers for keys it still has. For keys already moved it replies -ASK with the target address; the client sends an ASKING command followed by the original command to the target, once, without updating its table. The target accepts the key only after ASKING. When the migration finishes, a node that no longer owns the slot answers -MOVED <slot> <host:port>, which is the permanent signal: update the slot map. Clients should handle both.

The timeout branch is the dangerous one. A read that times out can be retried. An INCR that times out may or may not have been applied on the node, and retrying double-counts; a compare-and-set (CAS, which writes only if the stored token still matches) can succeed on the first attempt and then report a conflict on the retry. Non-idempotent operations need a request token the server can deduplicate, or the caller must treat the outcome as *ambiguous* and resolve it by reading. Atomicity is per command, on one node; there is no global transaction across keys on different shards (see retry and backoff).

Three practical details sit under any routing design. Discovery: a client learns the topology at start-up from a seed node or a configuration service and must re-learn it when redirected or when connections fail. Connection pools: each pod keeps a small pool of persistent connections per node, so 20 pods talking to 8 nodes hold on the order of a few hundred sockets; opening a connection per request would spend more time on handshakes (and TLS) than on the lookup. Retries and deadlines: the request carries one overall deadline that the redirects and retries all draw from, and retries are capped (and ideally budgeted across the fleet), because a retry storm against a struggling shard turns a partial failure into a full one. Resharding also has a cost model. Every moved key is either migrated explicitly (slot migration copies it) or missed and refilled (ring changes). The first uses network and CPU on both nodes; the second pushes load onto the database. Plan it off-peak, throttle migration, and watch the miss rate while it runs.

A single node is a single point of failure, so each primary gets a replica. The trap is in what the primary's "OK" tells the client.

Redis replication is asynchronous by default: the primary applies the write, replies to the client, and streams the change to replicas, which acknowledge processed offsets asynchronously; the primary does not wait. A replica can serve read-only queries, which may be stale. A reconnecting replica attempts a partial resync using the replication ID and offset, and falls back to a full resync from a snapshot when that is not possible.

Follow one write. The primary acknowledges SET product:4821 $109 v6. The replica, 120 ms behind in this hypothetical, has not yet applied it. The primary dies and the replica is promoted: it holds v5 ($99). The acknowledged write is gone. The Redis Cluster specification says so directly: it tries, best-effort, to keep writes from clients connected to a majority of primaries, but there are windows where acknowledged writes can be lost, larger when a client is in a minority partition. It also uses asynchronous replication, last-failover-wins, and no merging of values. WAIT can make the client block for replica acknowledgements, which sharply reduces the loss window, yet it does not turn the system into a strongly consistent one.

A custom design could instead acknowledge only after one or more replicas confirm (synchronous or quorum replication). That closes most of the window, but every write now pays a network round trip to the slowest required replica, a slow or partitioned replica stalls writes (or forces a downgrade to async), and it still does not make a cache a database. Most caches accept the async window because reads dominate and the tail of recent writes is cheap to repair. For a cache this loss is acceptable only because the cache is rebuildable: the database still says v6, an invalidation or TTL eventually corrects the entry, and checkout revalidates. If the same cache were the only copy of the data — a session store, a counter that is billed, a write-behind buffer — the same event would be a business error. Do not call a cache durable because it has replicas.

Split brain, fencing and epochs

*Split brain* means two nodes both believe they are primary for the same data, typically after a partition. Redis Cluster limits the damage with majority voting and with minority-side primaries refusing writes after a node timeout. A custom design must provide the equivalent: a monotonically increasing epoch (a term number) attached to each primary, replicas that reject replication from a lower epoch, and fencing, so a deposed primary's writes are refused downstream. Without these, two primaries can each accept writes and diverge.

• Say which loss you accept (the unreplicated tail at failover) and what repairs it (TTL, invalidation, revalidation).
• Place a replica in a different failure zone from its primary, and size memory for the full copy: replicas double the bill, 15.27 GiB becoming about 30.5 GiB across eight nodes in the sample worksheet.
• Rebuild gradually after failover. A promoted replica may be partially cold; ramp traffic.

• Claiming synchronous guarantees from `WAIT`.
• Reading from replicas for data that must be current, such as permission decisions.
• Calling a cache with write-behind 'durable'.

A *stampede* happens when many callers miss on the same key at once and each goes to the database, because nobody has yet refilled it. The hot product 4821 at 8,640 requests per second with an 85 ms miss path sends 8,640 × 0.085 ≈ 734 requests to the database before the first refill lands.

The flash-sale version is starker: 5,000 concurrent misses on one expired key, spread over the 20 pods. With no coalescing every one of those requests issues its own fetch, and the database sees 5,000 identical reads for one row.

Single-flight and its scope

*Single-flight* (request coalescing) makes callers for the same key in the same process share one in-flight fetch. Across 20 pods that cuts 5,000 database fetches to at most 20, about 250 waiters per pod. A distributed lease — one pod acquires a short-lived lock in the cache or a coordination store, and the rest wait or serve stale — cuts it to 1, plus races when the lease expires. The costs: waiters need deadlines or they queue forever; if the owner crashes, the lease must expire and someone else must take over; and a lease that expires while its owner is merely slow lets two fetchers run (so the fill must still be version-conditional). See distributed locks for why a lock is advisory here, never a correctness mechanism.

Early refresh and jitter

Probabilistic early refresh lets a caller occasionally refresh an entry shortly *before* it expires, with higher probability as expiry nears, so the key is rarely absent at all. Stale-while-revalidate serves the previous value while one caller refreshes. TTL jitter attacks a different shape of stampede: if 6,000 keys are warmed together with an identical 300 s TTL they all expire in the same instant. With ±20% jitter the lifetimes are uniform over 240–360 s, a spread of 120 s, so about 6,000 / 120 = 50 keys per second expire instead of 6,000 at once.

Sharding spreads *different* keys across nodes. A single key is indivisible: all of its traffic lands on one shard no matter how many shards exist.

At peak the top product draws 18% of the 48,000 eligible requests per second, which is 8,640 on one key (not to be confused with the 8,640 hits per second of the average-load case; same number, different quantity). The other 39,360 spread over four shards at 9,840 each, so the hot shard carries 8,640 + 9,840 = 18,480/s, or 73.9% of a hypothetical 25,000 ops/s budget, 1.88× the others. A trending item at 45% (21,600/s) pushes its shard to 28,200/s, over budget, while the rest sit at 6,600.

Detection comes from sampled key-level counters or per-shard metrics: shard CPU skew, one key's share of traffic. Relief has two main forms. A tiny per-pod L1 (local, in-process) cache for just that key, with a 2 s TTL across 20 pods, limits the shard to at most 20 / 2 = 10 GETs per second for it. Alternatively, replicate the hot value as copies under several key names and spread reads across them. Both trade freshness: the key can be stale for up to the L1 TTL, and invalidation has to reach every L1 and every copy, so an invalidation message that is dropped costs you the 2 s bound, not the 360 s one.

• Promote keys to L1 by measurement, with a short TTL and a size cap.
• Send invalidations to local caches too, or accept the short TTL as the whole bound.
• Keep hot-key policy per data class: a display card may be 2 s stale; a permission decision may not.

• Adding shards for a single hot key.
• Putting one tenant under a single hash tag and then being surprised that it is one slot.

Sizing a cache starts with the number of objects times their size, and then every additional term the first estimate forgot.

Meridian's illustrative workload: average 12,000 requests per second, peak 60,000, 80% cache-eligible, baseline 90% hit rate, average serialized object 1.5 KiB, 8,000,000 potentially hot objects. Payload alone is 8,000,000 × 1.5 KiB = 12,000,000 KiB = 11.44 GiB (12.288 GB decimal; GiB means 2^30 bytes, GB 10^9, and mixing them is an easy 7% error).

On four primaries of 8 GiB (32 GiB nominal) the average utilization is 15.27 / 32 = 47.7%, so there is room, but only after the model: also reserve headroom for replication and persistence buffers (Redis does not count the output buffers queued for replicas or the AOF toward maxmemory, so they sit on top of the limit), and for copy-on-write growth while a snapshot is being written if persistence is on. A claim that "11.44 GiB fits in 32 GiB" without that arithmetic is an invitation to an eviction storm. Cost follows the same worksheet: the 8 nodes of 8 GiB are what is billed, not the 11.44 GiB of payload, and replicas and failover reserve are the line items most often left out of the first quote. Weigh that bill against the database capacity it replaces, and note that object-size distribution matters: a few multi-megabyte values can consume more memory than millions of small ones, so size by percentile, not only by the 1.5 KiB average.

Throughput and the backend ceiling

At peak all 48,000 eligible requests do a GET first, so 48,000 GETs per second spread over 4 shards is 12,000 per shard, 48% of the hypothetical 25,000 ops/s budget. The database side is the other half: with a 90% hit rate, 4,800 eligible misses plus 12,000 non-eligible requests give 16,800 reads per second, 28% of the 60,000 no-cache baseline, a 72% backend reduction. Against a hypothetical 24,000 reads/s capacity that is 70%, below the 80% operational ceiling of 19,200. Note that a 90% hit rate on eligible traffic is not a 90% reduction in total traffic, since the 20% non-eligible requests never benefit.

Cache-aside is one placement of the logic. Three siblings move it into the cache layer, with different failure properties.

Write-behind deserves the longest warning. It turns the cache into the first and only copy of recent writes, so everything in chapter 8 about acknowledged loss becomes a *business* error rather than a repairable cache miss. It is a reasonable choice for low-value counters or telemetry, and an unsuitable one for orders or payments. An asynchronous buffer is not a substitute for an authoritative database write.

Write-through: the database commit succeeds and the cache update fails

Take a price change for product 4821 from $99 (v5) to $109 (v6). A write-through layer commits v6 to the database and then updates the cache. If the cache call times out or its node is failing over, the layer has two bad options. It can report failure although the database committed, which makes the caller retry a write that already happened and so requires the write to be idempotent. Or it can report success and leave the cache at v5 until the TTL ends. A sound implementation treats the database commit as the commit point, and on a cache failure deletes the key (or marks it stale) and retries that step in the background; if the retry also fails, the TTL backstop of about 360 s bounds the staleness. Reversing the order is worse: writing the cache first and the database second means a failed commit leaves a price in the cache that was never committed. No transaction spans a cache and a database, so the design has to choose which failure it prefers and bound it. Write-through also does not remove the refill race from chapter 4: a slow reader that loaded v5 before the commit can still overwrite v6, so the versioned conditional fill still applies.

Write-behind: flush ordering, crashes and what was acknowledged

Write-behind suits updates whose individual loss is tolerable, such as a view counter for product 4821. The cache layer applies each increment in memory, acknowledges, and flushes batches to the database on a timer or after a number of updates. Three details decide whether it is safe. Ordering: if two updates to one key are queued, the flusher must apply them in order or coalesce them into one value or sum, otherwise a retried older batch can overwrite newer data; that is easy while one node owns the key and harder when a failover moves it. Crash exposure: everything acknowledged but not yet flushed exists only in memory, and perhaps on an asynchronous replica. With a 1 s flush interval and 2,000 updates per second (hypothetical), a crash can lose roughly 2,000 acknowledged updates, plus whatever the replica was behind. Replay: a flush that is retried after a crash must carry a per-key sequence number so a repeated batch does not count twice. Persisting the queue in a durable log before acknowledging closes the loss window, but then the layer is a write-ahead log with its own durability cost, not a cache. Expose the unflushed backlog and the oldest unflushed update as metrics, because that age is the loss window you are carrying.

Meridian uses cache-aside for products: it is the pattern that keeps working when the cache is down, which fits the principle that the cache is a derived copy. The related CQRS read-model idea is the same split at larger scale.

A shared cache is a shared memory space, and the naming convention that separates tenants protects nothing if the wrong caller can ask for any name.

Cedar can be noisy; Birch is well-behaved. Isolation has layers. Identity: the tenant segment in the key comes from the verified credential, not from a header the client controls, so a request for Birch's data cannot be built by editing a parameter. Access control: the engine's ACLs and a separate credential per service restrict which key patterns and commands each client may use, and privileged commands (flush, config, key enumeration) are denied to application roles. Transport: TLS between pods and cache nodes. Quotas: per-tenant limits on memory, key count and write rate, so Cedar's burst evicts Cedar's entries first; for strict needs, a dedicated shard tier per tenant class.

Attacks specific to caches: *cache key poisoning* (crafting inputs that make a bad response get stored under a good key, or under a key that other users share); *negative-cache enumeration* (filling the store with not-found markers, hence the per-tenant cap on negative entries); *decompression bombs* (small stored values that expand enormously); unsafe deserialization. Authorization decisions are the sharpest case: a revoked permission must fail safe by re-checking the authority or consulting a versioned revocation epoch, never by serving a stale allow. Never cache a 401 or 403 as a not-found for other identities.

• Put verified tenant and schema version into every key, and enforce ACLs on the server.
• Cap negative entries per tenant, and keep identity out of shared negative entries.
• Keep secrets and payment instruments out of the shared cache entirely.

• Trusting naming conventions as isolation.
• Letting application roles run key-listing or flush commands.

A global hit rate averages away the failures that matter; the cache needs measurements split by key class, by shard, and by tenant.

The fallback figure comes from the outage math: with the cache fully down at peak, up to 48,000 eligible requests per second go to the backend, or 2,880,000 in one minute. The database's planned ceiling of 19,200/s minus 12,000 non-eligible leaves a 7,200/s fallback cap, so 40,800/s (85.0%) of eligible requests must be shed, served stale from an allowed class, or degraded — 6.7 times oversubscribed. Shedding and breakers are decided *before* the incident (load shedding, circuit breaker).

Error budgets should be written per class: for example, "price staleness beyond 360 s: zero tolerated, alert on any". Distinguish cache-side hit latency, API latency, backend QPS reduction and customer latency; they are four different numbers.

Everything above reduces to a short list of statements the design keeps true, and a list of phrases it must not use.

• The cache is a derived, best-effort copy. Checkout, inventory reservation and authorization use authoritative validation.
• A hit is not proof of freshness, completeness or permission. A TTL bounds staleness; it does not synchronize.
• Ownership is deterministic and topology-versioned, and modulo, rings and slots are different mechanisms.
• A hot key is not solved by sharding; L1 or copies are used, with their staleness stated.
• Failover can lose acknowledged writes; repair relies on revalidation.
• Atomicity is per command per node; ambiguous timeouts need tokens or reads.
• Database fallback is bounded by breakers, queues and shedding.
• Memory is more than payload; negative caching is capped; tenant isolation needs verified identity and ACLs.
• Cross-region invalidation has delay and can fail.