Distributed Cache System Design Interview: Questions, Answers & Trade-offs

Fix the query first when the query is the problem, and add a cache when the query is already reasonable but the same answer is requested far more often than it changes. A missing index, an N+1 access pattern or a table scan will make the database slow for every caller, and a cache only hides that on the keys that happen to be warm; the first cold or evicted key still pays the full cost. Look at the numbers before choosing. If a product read costs 3 ms from an in-memory cache and 85 ms from the database (illustrative), and 80% of 60,000 requests per second are repeats of reusable reads, then even a perfect index leaves the database serving the full 60,000 per second, 2.5 times a hypothetical capacity of 24,000. Better indexing makes each read cheaper, but it does not remove the reads. A cache does remove them: at a 90% hit rate on eligible reads the database sees 16,800 per second. A cache is justified when the read volume is high, the data tolerates a defined amount of staleness, there is meaningful repetition (skewed popularity helps), and the system can survive the cache being empty. It is a poor choice when data is written as often as it is read, when every read must be fresh, when keys are almost never repeated (a low hit rate pays the lookup cost for nothing), or when the database could simply be scaled with replicas at lower operational risk. I would also note that a cache adds a failure mode: the system can come to depend on it, so its absence at peak must be planned for.

• What hit rate makes a cache worthwhile? — There is no universal threshold. Compare the cost of a lookup plus the miss path against the saving: at 3 ms for a hit and 85 ms for a miss, a 10% hit rate would save little, while 90% cuts mean eligible latency to 11.2 ms. The more important question is whether the database survives without it.
• Would read replicas be enough? — Sometimes. They scale reads with the same consistency model as the database and no invalidation logic, but they cost a full database each and cannot match in-memory latency. Choose them first if load is the only issue.

The difference is who talks to the database and when. In cache-aside, also called lazy loading, the application owns the logic: it reads the cache, on a miss reads the database itself, then writes the value into the cache. Only requested data gets cached, the cache can be empty or down without breaking correctness, and the application can decide what to cache and for how long. The cost is that every caller must implement it correctly, and the first request for a cold key is slow. In read-through the cache sits in front of the database and loads on a miss through a loader it was given; the calling code only asks the cache. That centralises the logic and makes request coalescing easy to put in one place, but it needs a cache layer that can reach the database and enforces one way of loading. Write-through is about writes: the application writes to the cache and the cache synchronously writes to the database before acknowledging, so cache and database agree after each write, at the price of write latency and a cache that now sits in the critical write path. A fourth variant, write-behind, acknowledges after writing the cache and persists later; it is faster, but a cache crash loses writes the user was told succeeded, so it makes cache failures a business error instead of a performance problem. For Meridian's product reads I would pick cache-aside with invalidation, because the cache is allowed to be wrong or empty and the database must stay the single writer. I would choose read-through if many services shared the cache and I wanted one implementation of loading and coalescing, and write-through only for data that is read immediately after it is written and where the cache is not the sole copy.

• Does write-through remove stale data? — It removes one cause, because the cache and database are written together for that path. Other writers that go directly to the database, replicas lagging, or a failed second write still leave inconsistencies, so a TTL is still useful as a backstop.
• Where does write-behind belong? — Only where losing the most recent writes is acceptable and measured, such as counters or analytics, and never for the system of record.

Because each pod holds its own private copy, so you get three problems at once: duplicate memory, uneven freshness and a lower hit rate. Take 20 Product API pods and a request for product:4821. The load balancer spreads that product's traffic across all 20, so the key has to be fetched and stored 20 times, once per pod, and each pod misses independently after every restart or expiry. Memory is multiplied by 20 for the same data. Worse, when the price changes from $99 to $109, one pod can refresh while another keeps showing $99 until its TTL ends, so two shoppers hitting different pods in the same second see different prices; in-process caches also cannot easily be invalidated from outside without a broadcast channel. A shared cache removes duplication and gives one place to invalidate: after the first miss every pod hits. That is not a reason to throw local caches away. A tiny local cache (L1) with a two-second TTL in front of the shared cache is useful for a single very hot key, because it removes almost all of its load from one shard: 20 pods fetching at most once every two seconds is at most 10 shard reads per second for that key. The cost is up to two seconds of staleness on those entries and an extra invalidation path. Use L1 deliberately for a few hot keys, not as the main cache.

• When is local-only enough? — For small, rarely changing reference data (configuration, feature definitions) where a few seconds of divergence is harmless and the data set is tiny.
• How would you invalidate L1? — Subscribe each pod to the invalidation stream and delete the key locally; keep a short TTL as the backstop, since a pod can miss an event.

Start with a shopper requesting product 4821 from tenant Cedar. The Product API builds the key v2:cedar:product:4821 and sends a GET to the shard that owns it. The shard answers with nothing: that is the miss. The API then reads the row from the product database, which takes about 85 ms in the illustration compared with 3 ms for a hit. The database returns the row with its record version, for example v5 and a $99 price. The API serialises it into the value envelope (schema version, record version, creation and expiry times, compression flag, bytes), checks it is under the 1 MiB size cap, and sends a SET with a TTL, which for a price is 300 s with random jitter of ±20%. The reply to the shopper does not need to wait for the fill; some implementations send the response and fill asynchronously, though then a failed fill is silently lost, which is fine for a cache. Several details are where it goes wrong. If the database read fails, the API must not cache an error. If 5,000 requests miss the same key at once, they all go to the database unless the API coalesces them. If the SET times out, the outcome is unknown: the value may or may not be there, which is harmless for a plain fill but matters for conditional or counter operations. And if the product genuinely does not exist, store a negative entry for 30 s so repeated lookups do not all reach the database.

• Should the API wait for the fill? — Not usually. Return the value and treat the fill as best effort, with a short timeout and a metric for failures, so a slow cache never slows the shopper beyond the miss itself.
• What if the database is down on a miss? — Serve stale only for classes where policy allows it (and the entry still exists), otherwise return an error; never fabricate a value or cache the failure.

Take product 4821 at 8,640 requests per second with an 85 ms miss path. When its entry expires, every request that arrives before the first refill completes is a miss. That is 8,640 × 0.085 ≈ 734 requests all asking the database for the same row; in a flash sale it can be 5,000 concurrent misses on one expired key. The database does the same work 734 or 5,000 times and may slow down, which lengthens the miss path, which lets more requests pile up: the stampede feeds itself. Single-flight means that for a given key only one caller performs the load and everyone else waits for its result. Inside one process this is a map from key to an in-flight future; the first caller creates it, later callers await it. With 20 pods that cuts 5,000 database fetches to at most 20, about 250 waiters per pod. To get to one, use a distributed lease: a caller sets a short-lived lock key in the cache, and only the holder loads; others wait briefly, serve a slightly stale value, or retry. The details matter. A lease needs an expiry so a crashed holder does not block the key forever, and a lease can expire while the holder is still working, so the fill itself should be conditional. Waiters need deadlines and a fallback, or the lock simply moves the queue. Single-flight does not help when many different keys expire together: lock-per-key does nothing for 6,000 keys that all expire at one moment. That case needs TTL jitter (spread over 120 s, about 50 expirations per second), stale-while-revalidate, or refreshing before expiry.

• Is per-process single-flight enough? — For a small fleet it is often enough: 20 fetches instead of 5,000 is a different order of load. Add a distributed lease only when the remaining multiplication still hurts.
• What about probabilistic early refresh? — It lets some requests refresh slightly before expiry with a probability that rises as expiry approaches, which avoids a synchronized cliff without a lock. It comes from published work on optimal probabilistic early expiration (often called XFetch); tune it against your own traffic.

A TTL gives an upper bound on how long an entry may be returned, nothing more. For Redis, a key with an expiry is never returned after it has expired, and the server removes it by two routes: when someone touches it, and by periodically sampling keys that have an expiry, so an expired key can sit in memory for a short while without ever being served. Memcached documents the same promise for a non-zero expiry: it will not be retrievable afterwards, though its protocol notes that expiry is only accurate to about a second and that a TTL of 1 can expire at once. That is the whole guarantee. A TTL does not mean that the value is fresh before the deadline. If the price changes at second 10 of a 300 s entry, readers may see the old price for 290 more seconds, and nothing synchronises the two events. It does not mean that all caches agree, since replicas and local caches have their own timers. It does not stop the entry being evicted early: under memory pressure a key may disappear long before it expires, and in Memcached an expiry of zero means never expires but may still be deleted to make room. And a TTL does not protect against a stampede; it causes one when many entries share a deadline. In design terms TTL is a backstop for lost invalidations plus a staleness budget: for Meridian's price, event invalidation brings freshness within seconds normally, and a 300 s TTL with ±20% jitter means the worst case after a lost event is about 360 s.

• Should the TTL be shorter for volatile data? — Yes, within reason; TTL is a staleness budget per class. Thirty seconds for an availability badge, 300 s for a price with events, 15 minutes for descriptions. Very short TTLs raise the miss rate and database load.
• Do replicas expire keys themselves? — In Redis, replicas do not delete expired keys themselves: the primary expires the key and sends a DEL. A replica does, however, use its own clock to hide a logically expired key from reads while it waits for that DEL, and once promoted it expires keys independently (Redis replication documentation).

Lay it out as four steps with no locks involved. At t0 a reader on pod A misses on product:4821 and reads the database: price $99, version v5. The read is slow, maybe a garbage-collection pause or a busy connection. At t1 an editor changes the price to $109 (v6) and the commit succeeds. At t2 the invalidation subscriber deletes the cache key, which is already absent, so the delete does nothing useful. At t3 pod A finally finishes and fills the cache with $99, v5. The cache now holds the old price and no further event will correct it, because the invalidation already happened. Until the TTL ends, every reader sees $99. Deleting is not wrong; the problem is that the fill is an unconditional write based on a read taken before the commit. Remedies come in strengths. First, accept it: with a 300 s TTL ±20% the staleness is bounded at about 360 s, and checkout revalidates, which is acceptable for a display price. Second, make the fill conditional: store the record version in the entry, and have the fill succeed only if no newer version is present. That needs the atomic compare to run in one place (a small server-side script in Redis, or compare-and-set tokens in Memcached), and it needs something for the delete to leave behind, either a short-lived tombstone carrying v6 or a write of the new value, because an empty key carries no version to compare with. The version must come from the authority, the database's row version, not from a cache clock or an application timestamp. Third, delay a second delete a short time after the write, which narrows the window but does not close it.

• Why not update the cache on write instead of deleting? — Two concurrent writers can update the cache in the opposite order from their commits, leaving the older value last; updates also require writers to know every cached representation. Delete plus versioned fill is easier to reason about.
• Does a tombstone cost anything? — It occupies a key for a short TTL and adds a read-path branch. It buys a version to compare against; keep the TTL just longer than the longest plausible database read.

Separate what is displayed from what is decided, and treat each class differently. The product card price is display only; invalidation by an outbox event plus a 300 s TTL with jitter keeps it within seconds in normal operation and at most about 360 s if an event is lost, and checkout re-reads the price from the authoritative service, so a stale $99 on screen does not charge $99. The availability badge is a 30 s TTL entry for display. A cache entry for stock must never reserve or promise anything: the reservation happens against the inventory service and either succeeds or fails there. So the user may see In stock and then be told the item sold out at checkout, a UI cost rather than an oversell. If that experience is unacceptable for a particular launch, shorten the TTL or bypass the cache for the last step, at the price of more database traffic. For authorization the rule is stricter: permission decisions are never served from a TTL-only cache, since a revoked permission must fail safe. Check the authority, or compare a versioned revocation epoch with the entry. Stale-while-revalidate (serve the old value for up to 5 minutes while one request refreshes) is a fine choice for descriptions and a poor one for permissions or ownership of stock. Monitoring completes the picture: log the age of served entries and the rate of checkout price corrections, since a rising correction rate means the staleness budget is too wide.

• Can you make the badge exact? — Not from a cache. The best you can do is a short TTL plus event invalidation; exactness belongs to the inventory service at reservation.
• How would you measure staleness? — Compare record version and age in served envelopes with the database version on a sampled basis, and track checkout corrections.

Begin from the access pattern, since each policy encodes a guess about it. Eviction is what a cache does when memory is full; expiry is separate, and removes entries whose time has run out. LRU (least recently used) evicts the entry untouched for the longest time and suits traffic where something read recently will probably be read again. LFU (least frequently used) evicts the one used least often and suits skewed popularity, where a few products draw most reads and must not be pushed out by a one-off burst. Random is cheap and reasonable when accesses are roughly uniform. A TTL-first policy evicts whatever is closest to expiring, which only works if entries carry expiries. For Meridian the reads are skewed: the top product alone is 18% of eligible peak traffic. That points to LFU or an LRU variant, with a caveat about scans. Suppose a crawler requests 2 million distinct products once each. Under plain LRU it can push out the hot working set, because each crawled entry is the most recently used for a moment; LFU resists that, since a single touch earns a low frequency. LFU has its own weakness: an old favourite can stay long after interest fades, so engines add decay. In Redis, LRU and LFU are approximations; LRU samples a few keys (the docs mention maxmemory-samples, default 5) rather than keeping an exact list, and LFU uses a small probabilistic counter with decay. The docs describe allkeys-lru as a good default when there is no reason to prefer another, and volatile-* policies only consider keys with an expiry, behaving like noeviction if there are none. I would measure rather than guess: replay a sampled access log against candidate policies and compare hit rate, and watch the eviction rate in production, because evicting entries that are still hot shows up as a falling hit rate and more database load.

• What if memory fills with entries that never expire and noeviction is set? — Writes fail with an error. For a cache that is usually wrong: choose an allkeys policy, or make sure everything carries a TTL.
• Should you tune eviction for the hot key? — Probably not; a very hot key survives any decent policy. Tune for the long tail that determines the overall hit rate.

Think of it as pre-sorted shelves versus individually allocated items. Memcached divides memory into pages and gives each page to a slab class, a class of fixed-size chunks (the stats output reports a slab class id). An item goes into the smallest chunk size that fits, so some bytes in each chunk are wasted, which is internal fragmentation, and each class has its own LRU. The consequence is that if the size mix changes (a deploy makes values larger, say), pages are sitting in classes that are no longer useful while the new class has too few; Memcached has slab rebalancing for that (slabs reassign and slabs automove appear in the protocol documentation, which notes they may change), and an LRU crawler to reclaim expired items. Redis, by comparison, typically stores each key and value as separately allocated objects through a general-purpose allocator and enforces a total maxmemory limit with a global eviction policy across keys, so there is no per-size-class eviction. Its fragmentation shows as resident memory exceeding the bytes the dataset logically uses, and persistence or replication adds buffers and copy-on-write overhead, which is why the docs advise leaving headroom for them (INFO memory reports an amount not counted for eviction). The two also differ in data model: Memcached offers simple key-to-bytes items with cas tokens, and distribution is usually done by the client hashing keys across servers rather than built into the server; Redis adds structures, replication and a cluster mode. Details such as allocator choice, default item size (about 1 MiB by default and configurable) and automove behavior vary by version, so I would check the documentation for the version in use rather than quoting defaults.

• Which would you pick for a plain product cache? — Either can work. Choose Memcached-like simplicity when you need only key-value reads with client-side sharding, and Redis-like features when you want structures, replicas or server-side scripts. Operational familiarity often decides it.
• Why does value size matter in a slab design? — Chunk sizes round up, so many values just over a boundary waste memory; check the size histogram before setting growth parameters.

Payload arithmetic gives the floor and is almost never the answer. 8,000,000 objects × 1.5 KiB is 11.44 GiB (12.288 GB in decimal units; say which). Now add what the engine stores around each object. I would assume, and label as assumed, a 40-byte key and 64 bytes of metadata, hash-table and pointer overhead per entry: 1,536 + 40 + 64 = 1,640 bytes, so 8,000,000 entries come to 12.22 GiB. Allocators and slab classes waste memory too; with an assumed 1.25 fragmentation factor the total becomes 15.27 GiB. Spread across four shards, that is 3.82 GiB each on average, and if the busiest shard holds 15% more than the mean (an assumption about uneven distribution), 4.39 GiB, which is 54.9% of an 8 GiB budget. The nominal 4 × 8 = 32 GiB cluster is 47.7% utilised on average. One replica per primary doubles the bill: about 30.5 GiB across 8 nodes. Then reserve more. Replication and client buffers, and snapshot or fork behavior if persistence is on, need headroom the eviction logic does not count; I would not plan to run above something like 70–75% of the budget (a judgment, not a rule). Every overhead above is a placeholder: load a realistic sample into the actual engine and read its own memory report to replace them. It is equally important to size for the working set rather than the whole catalog: if 8,000,000 objects are hot but the long tail is cold, a smaller cache with a lower hit rate may be a better trade.

• What if hit rate falls when you shrink the cache? — That is the experiment: measure hit rate against cache size on real traffic and pick the knee of the curve.
• Does compression help? — It trades CPU and latency for memory. It pays for large compressible values and hurts tiny ones; store a compression flag in the envelope so mixed versions can coexist.

The three are different quantities and mixing them is the commonest accounting error. Hit rate is hits divided by cache lookups: of 48,000 eligible GETs per second at peak, 43,200 are hits, so 90%, and the miss rate is the remaining 10% (4,800). Backend request reduction compares database reads with and without the cache, across all traffic. Without a cache the database serves all 60,000 requests per second. With it, the database serves the 4,800 misses and the 12,000 non-eligible requests: 16,800. Reduction is 1 − 16,800 ÷ 60,000 = 72%, not 90%, because 20% of traffic never benefits. At the average load it is also 72% (3,360 against 12,000). Two further distinctions: latency and tails. With hits at 3 ms and misses at 85 ms (illustrative), the mean for eligible requests is 0.9 × 3 + 0.1 × 85 = 11.2 ms, but at a 10% miss rate the 95th and 99th percentiles are about the miss latency, 85 ms. A dashboard that shows only the average hides that. A reasonable dashboard computes hit rate per key class, per tenant and per shard (an overall figure can hide one collapsing class), counts a negative-cache hit separately, and excludes bypass or error paths from the denominator, saying how.

• Why report per class? — A healthy 90% overall can hide a price class at 40% hit rate that is doing the damage.
• Is a falling hit rate always bad? — Not always: a deploy that adds new key variants or a flash event of cold keys lowers it. It becomes an incident when database load approaches its ceiling.

Because the owner is hash(key) mod N, and changing N changes the answer for almost every key. Go from 3 nodes to 4: a key stays only when hash mod 3 equals hash mod 4. For a large uniform sample the fraction that changes owner is N ÷ (N+1), so 3 ÷ 4 = 75%. With ten nodes becoming eleven, 90.9% move. A tiny sample shows it: use illustrative hash values 17, 22, 35, 41, 58, 64, 70, 83, 96, 101, 112 and 120 (not a real hash function). Ten of the twelve land on a different node when N goes from 3 to 4, 83% in this small sample against the 75% expected. For a cache the movement is not a data migration problem but a cold-cache problem: those keys are not at their new owner, so reads miss, the misses go to the database, and the hit rate collapses just when you added capacity because load was already high. With 4,800 misses per second at peak and 75% of the keyspace suddenly cold, the database would face far more than its 19,200 per second planned ceiling. Modulo is fine for a fixed number of nodes that never changes, and unsuitable when the cluster must grow or lose nodes. The fix is a scheme in which a change relocates only a share of keys.

• Could you warm the new node before switching? — Yes, by replaying hot keys or shadow-reading, but the placement function still moves most keys, so you would warm most of the cache.
• What about removing a node? — The same: the remainder of the keyspace is rehashed unless you use a ring or slots.

Picture a circle of hash positions 0 to 99 (an illustrative space). Each node owns tokens on the circle, and a key belongs to the first token clockwise from its hash, at or after it. Let A hold tokens 10 and 60, B hold 25 and 80, and C hold 40 and 95; each token is a virtual node, one of several positions a physical node occupies. Ownership over the 100 positions works out to A 35, B 35, C 30. The key product:4821 hashes to 33; the next token clockwise is 40, so it lives on C. Now add node D with tokens 20 and 70. D takes the ranges (10, 20] and (60, 70], both of which previously belonged to B. That is 20 of 100 positions, and they all move from B to D; no key moves between A and C, and key 33 stays on C. Compared with modulo hashing, which would move about three quarters of the keys for a 3 to 4 change, the movement is limited to what the new node takes over. Notice also that the result is uneven (A 35, B 15, C 30, D 20) because we gave each node only two tokens; real systems place many virtual nodes per physical node so that shares average out and so that a removed node's load spreads over many survivors instead of dumping onto one neighbor. For balanced rings the expected movement when going from N to N+1 nodes is roughly 1/(N+1) of keys; I would not quote more specific percentages for a real ring without measuring it.

• What does a virtual node buy over a single token? — Smoother distribution and finer-grained rebalancing, at the cost of a larger routing table.
• Does the ring fix a hot key? — No. A single key has one hash and one owner however the ring is cut.

Redis Cluster does not place keys on a ring. It defines 16,384 hash slots and computes HASH_SLOT = CRC16(key) mod 16384 (a hash tag inside braces, when present and non-empty, is the only part hashed). Each primary serves a set of slots, kept in an explicit slot-to-node table, and resharding moves whole slots on purpose, using commands that put a slot into migrating and importing states. In my three-primary illustration A has 0–5460, B 5461–10922 and C 10923–16383; product:4821 hashes to slot 13107 and so lives on C, while tenant:cedar:product:4821 is slot 8298, on B. To add D, an operator or planner hands over 4,096 slots, a quarter (say 1,365 from A, 1,365 from B and 1,366 from C, in one illustration), and product:4821 moves because slot 13107 is in the range taken from C. Differences matter. On a ring ownership follows from positions and tokens implicitly; with slots the mapping is a table you choose and migrate explicitly. Slots are the unit of movement and make multi-key operations well defined: all keys in one slot are on one node, so a tag like {tenant:cedar} puts a tenant's keys together, at the price that a hot tenant becomes a hot slot on one node. The Redis spec also has no proxies and uses asynchronous replication with no merging of values; I would not describe it as consistent hashing in an interview, though both aim at limiting movement when nodes change.

• What happens to multi-key commands? — They require all keys to be in the same slot; otherwise the node rejects them with a cross-slot error. Hash tags are how you opt in.
• Why 16,384 and not more? — I would not give a design rationale I cannot cite. What the Redis Cluster specification does state is that the slot count fixes an upper limit of 16,384 primaries, with about 1,000 suggested as the practical maximum.

By treating redirection replies as normal protocol, bounded by a retry budget. Suppose the client believes slot 13107 belongs to C, but it has just moved to D. The client sends the command to C and receives a MOVED reply naming slot 13107 and the new address. MOVED is permanent: the client updates its slot map (by patching that slot or refreshing the table from any node) and re-sends to D. The other case is a slot that is mid-migration. The source node still answers for keys it has; for a key already transferred it replies ASK, a one-time redirect. The client then sends ASKING followed by the command to the target, which accepts it only because of that prefix, and does not update its map, since the slot has not finished moving. A well-behaved client therefore keeps a local slot table, handles both replies, caps the number of redirects per command (say 3–5), applies an overall deadline, and backs off if it keeps being redirected, because a loop usually means the cluster itself is unstable and hammering it makes things worse. Timeouts are the harder case: a request to a dead node says nothing about where the slot lives, so the client should refresh topology from another node, and it must remember that after a timeout the previous attempt may or may not have been applied, which is harmless for a read or an idempotent SET and dangerous for a retried increment or compare-and-set unless guarded by a token.

• Client library or proxy? — A smart client avoids an extra hop and has the map locally, but every language library must implement redirect handling correctly. A proxy hides topology from clients and centralises limits, adds a hop, and becomes another thing to run and scale.
• How often should the map be refreshed? — On MOVED, on connection errors, and periodically in the background, so a slow drift is caught even without errors.

Treat it as a staged rollout in which the old cache stays as the fallback and the database never sees a cliff. First change the key schema so that the new cluster is addressed by a new prefix or version (the v2: segment is there for this reason): old and new data can never be mistaken for each other. Second, build the new cluster beside the old one and run shadow traffic: the application still serves from the old cache but also fills and reads the new one, compares results and records hit rate, latency and mismatches without affecting users. Third, warm it. A cold 15 GiB cluster pointed at live traffic would send the miss path straight to the database, so replay hot keys or fill through shadow traffic until the hit rate approaches the old one's. Fourth, cut over gradually behind a flag: 1% of pods or tenants, then 10%, then everyone, watching hit rate, database QPS against the 19,200 per second ceiling, error rate and p99. Keep the old cache populated and writing during the window so rollback is a flag flip rather than a rebuild. Invalidation needs special care, since both caches must be invalidated until the old one is retired, or the old one will serve stale data after a rollback. Finally drain and remove the old cache once a full TTL cycle has passed with no reads. The usual mistakes are cutting over cold, forgetting dual invalidation, and having no rollback because the old cache was switched off at the same time.

• Can you migrate gradually per key range? — Yes: with slots or a prefix, route a subset of keys to the new cluster first. It is a finer-grained version of the same plan.
• What if the new cluster has different client semantics? — Verify with shadow reads before real traffic: key length limits, serialization, and multi-key operations are the common surprises.

Fewer than people assume. In Redis, replication is asynchronous by default: the primary applies a write, acknowledges the client, and streams the change to its replica, which reports the offset it has processed, also asynchronously. The primary does not wait. What you get is a copy that is usually close behind, a replica that can take over if the primary fails (with a partial resync using a replication ID and offset when possible, or a full resync from a snapshot when not), and the option of reading from replicas that may be stale. What you do not get is any promise that the replica has a given write at a given moment, so a read from a replica can return an older version, and no promise that a promoted replica contains everything the old primary acknowledged. The cluster documentation says it tries on a best-effort basis to retain writes from clients in contact with the majority of primaries, and that there are small windows where acknowledged writes can be lost, with larger windows in a minority partition. A WAIT command can block until a number of replicas acknowledge, which narrows the loss window but does not make the system strongly consistent. Memcached, by contrast, has no built-in replication; copies are a client or proxy concern. For a cache the usable summary is this: replicas improve read capacity and recovery time, they shorten the cold-start that follows a primary loss, and they never make the cache a safe place for the only copy of anything.

• Should reads go to replicas? — For data that tolerates a little extra staleness, yes: it spreads read load. Not for data that was just written and must be read back, nor for anything in the never-stale classes.
• What does replica lag look like in practice? — It can be anything from negligible to seconds under load or resync; the illustration in this guide uses 120 ms. Monitor the offset gap instead of assuming it.

Here is the timeline, with a hypothetical replica lag of 120 ms. At time t the cache write that reflects the price change for product 4821 (a $109 value, version v6, or the delete that removes the old entry) goes to primary C. C applies the write and replies OK, so the client believes it succeeded. The replication stream is still carrying the update; the replica C' has not applied it and holds $99, v5. Within that window, C crashes. The cluster elects C' as the new primary. It does so from the state it has, which does not include the write, and no process holds that write any longer; when the old primary eventually returns it is demoted and its extra data is discarded (the spec describes last failover wins and no merging of values). The cache is now serving $99 for a product whose price is $109 in the database. For a cache this is a bounded annoyance: the lost write is not replayed by anything, so only a later event for the same product or the TTL (at most about 360 s) repairs it, and checkout revalidates. A team that cannot wait that long can replay the last few minutes of invalidation events after every failover. If the cache had been the only copy of that price, the write would simply be gone, a business error rather than a cache blip. WAIT would shrink the window by waiting for the replica before replying, at the price of write latency, but it would still not guarantee survival in every failure. The defensible design is not to stake anything on the answer: keep the authority elsewhere, and make cache writes repairable.

• Could you use synchronous replication? — Waiting for acknowledgements trades latency and availability (the write stalls if the replica is unreachable) for a smaller window. It is rarely worth it for a cache; it is worth it for the system of record.
• How would you notice lost writes? — By sampling cache values against the database's record versions and alerting on mismatches after failovers.

Decide by data class before the incident, because during one there is no time. Descriptive content (C1) may be served stale for a defined period; stale-while-revalidate allows up to 5 minutes beyond the 15-minute TTL. A price on the card (C2) can usually be shown from a stale or last-known entry during a cache impairment, with the checkout path revalidating, since the displayed value is non-binding. Availability badges (C3) can go stale for their short 30-second budget or be hidden. Inventory reservation, payment instruments and permission decisions (C4) are never answered from the cache's stale copy: for these, call the authority directly or fail closed, so an unauthorised request is refused rather than allowed because of an old entry. The third option, calling the database, is limited by what the database can take. A cache outage at peak redirects 48,000 eligible requests per second to the database, but after the 12,000 that were never eligible, only 7,200 per second fit under the planned 19,200 ceiling. So the database is a choice for a thin slice, usually the classes where staleness is unacceptable or the cheapest-to-compute ones, and the rest is stale service, degraded responses or shedding. Use circuit breakers so repeated failures stop being attempted, and timeouts with retries that have jitter and budgets, because naive retries add load to a database that is already failing.

• Which class would you shed first? — The least valuable work: pages that can be degraded (hide recommendations, show cached descriptions) before anything that checkout needs.
• Who owns the policy? — The product and engineering together, written per class and tested, not improvised on call.

Start from the arithmetic, because it dictates the strategy. If the cache disappears at peak, up to 48,000 eligible requests per second go to the database in addition to 12,000 that never used the cache. A database with a hypothetical 24,000 reads per second capacity and a planned 19,200 ceiling can accept about 7,200 more: the eligible load is 6.7 times what is admissible, and 40,800 of every 48,000 requests, 85%, must be served some other way. Over one minute that is 2,880,000 extra requests, of which about 2,448,000 must be answered some other way. The defences work in layers. Admission control caps database concurrency to the 7,200 per second figure with a queue that has a deadline, rejecting the rest quickly rather than letting latency climb. Single-flight collapses duplicate reads of the same key, so the load is driven by distinct keys, not requests. Local L1 caches and stale-if-error serving absorb the hot keys and the permitted classes. Circuit breakers on the cache client avoid a long timeout on every request while the cache is down. Degraded modes drop expensive, non-essential features first. On recovery, do not let a cold cache and a thundering herd return together: warm the hottest keys first, ramp traffic back, and keep the admission cap until the hit rate has recovered. A replica per shard shortens the outage when one node fails; it does nothing when the whole cluster is unreachable, so the plan must exist anyway.

• Is a circuit breaker on the database also needed? — Yes: when the database slows, stop sending more work, return degraded responses, and probe gently for recovery.
• How would you test this? — In a game day: disable the cache at realistic load in a staging environment, confirm the admission cap holds, and measure how long recovery takes.

Spot it by looking at the distribution, not the average. Per-shard operations per second is the first signal: with 48,000 GETs per second at peak and four shards, a uniform spread gives 12,000 each. If product 4821 draws 18% of eligible reads, 8,640 per second land on one key, and its shard carries 8,640 + (48,000 − 8,640) ÷ 4 = 18,480 operations per second, 73.9% of a hypothetical 25,000 budget, while the others carry 9,840, a skew of 1.88. If a trending SKU took 45% (21,600 per second), its shard would need 28,200 and exceed the budget while the other three idle at 6,600. Detect with per-key counters sampled in the client or proxy, top-N reports, or the engine's own sampling tools; check the vendor documentation for what is available. Adding shards does not help, because the key still has one owner. What helps is serving the key from more places: a short-TTL local L1 in each pod (with 20 pods and a 2 s TTL, at most 20 ÷ 2 = 10 shard reads per second for that key), or replicated read-only copies under several derived keys with the reader picking one at random. Both reintroduce staleness, so invalidation has to reach the copies, and the TTL has to be the staleness the business accepts for that product (two seconds of old price on one item). Writes to a hot key are a different problem, since replicating copies does not apply; split the counter or buffer updates. Promotion can be automatic: a rolling counter promotes a key into the hot set and demotes it after the burst.

• Would you pre-split the hot key by default? — No; it adds invalidation cost to every key for a problem a few have. Detect and promote.
• What if the hot key is a write-heavy counter? — Shard the counter into N sub-keys and sum on read, or aggregate in memory and flush periodically, accepting delay.

Three layers, none of which a naming convention alone secures. Keys: the tenant belongs in the key (v2:cedar:product:4821), so that Cedar and Birch cannot collide, and the value should also record its tenant so a mix-up is detectable on read. But a prefix is not authorization. The service has to derive the tenant from the verified identity, never from a request field, and the cache should enforce access through its own accounts where it supports them; otherwise anything that can reach the cache can read every key. Placing a tenant's keys in one hash slot with a hash tag simplifies multi-key use but concentrates its load on one node, so a noisy tenant (Cedar, in the example) becomes a hot slot. Quotas: the cache is a shared pool, so cap memory per tenant (and entry counts for negative entries), or one tenant's large catalog evicts the others' hot data. Where one tenant is large enough to matter, a dedicated shard tier isolates it at the price of lower utilization; an alternative is weighted eviction or per-tenant limits in the application. Observability: break hit rate, memory, evictions and latency down by tenant, because a healthy global average can hide Birch's hit rate collapsing as Cedar floods the cache. Be careful that tenant identifiers in metrics and logs are not themselves sensitive. The classic incident is caching a response that depended on the caller's identity under a key that omitted it, which serves one tenant's data to another.

• Should each tenant get its own cache cluster? — Only where compliance or noise justifies the cost; the default is a shared pool with quotas and monitoring.
• How do you test isolation? — Automated tests that request a key as tenant A for an object owned by tenant B, expecting a miss or a denial, plus a canary that alerts on cross-tenant reads.

Each region runs its own cache close to its API pods, because a cross-ocean cache read would cost more than it saves. That choice makes invalidation a distribution problem. A write commits in the authoritative database, the outbox emits an event, and the event is delivered to a subscriber in each region (EU and US) that deletes or refreshes the key. Delivery takes time and can fail, so freshness is bounded, not instant: normal propagation is seconds, a lost event is bounded by the TTL, and a partitioned region keeps serving what it has. The design therefore states a staleness budget for cross-region visibility and does not claim that all regions agree at once. For residency, the question comes first: which data may leave which region at all? Data that must stay in the EU should never be written to a US cache, which means the key and the invalidation stream must be region-aware, replication of cache contents across regions should be off for that class, and each region fills from its own authoritative copy. Erasure requests are a second case: deleting a person's data must also delete or expire cached copies, so the deletion event must reach every regional cache and any L1, and the TTL gives an upper bound while the event is the mechanism; log that the purge happened. Legal specifics belong to your compliance team; the engineering contribution is knowing exactly where each copy can live and how it is removed.

• Why not one global cache? — A single region's cache adds cross-region latency to every hit and a cross-region dependency to every request; regional caches fail independently.
• What happens in a partition between regions? — Each region continues with its local cache and bounded staleness, and the invalidation backlog drains when the link recovers; the TTL protects against a lost backlog.

Think in three groups. Secrets and sensitive data: payment instruments, card numbers, passwords and tokens should not sit in a shared in-memory store, where a memory dump, a debug command or a misconfigured network exposes everything at once. If a token must be cached, store the minimum, encrypt it, set a short TTL and give it its own isolated instance. Authoritative decisions: inventory reservations (a cache entry cannot reserve stock), authorization or permission results served by TTL alone (a revoked permission would continue working until expiry), and anything whose wrong answer causes money or access to be given away. Identity-dependent responses cached under identity-independent keys: a page personalised for one user, a response containing another tenant's data, and a 403 or 401 stored as not-found for everybody. Negative caching needs its own restraint: a 30-second not-found entry is safe for an object that does not exist, but never cache an authorization failure across identities, and cap the number of negative entries per tenant so that an attacker enumerating random IDs cannot fill the cache or poison it. Finally, do not store anything you cannot rebuild: if losing the entry would be a business error, you have built a database, and it deserves one.

• Can you cache permissions at all? — Briefly and safely, with a versioned revocation epoch: a cached permission is honoured only if its epoch matches the authority's current one.
• Is encrypting values in the cache enough? — It limits exposure, but it does not fix a wrong key design or missing access control, and it adds CPU and key management.

I would test in four layers. At the bottom, unit and property tests on the logic that is easy to get wrong: the conditional fill must reject a lower version, single-flight must deliver one load and many results, TTL jitter must stay inside 240–360 s, and the key builder must include tenant and schema version. Next, integration tests with a real cache cluster in a container, exercising redirects: move a slot while a client is reading and check that MOVED and ASK are handled within the retry budget. Third, load tests with a production-shaped distribution rather than uniform keys. Replay a sampled access log, or generate a Zipf-like skew (a few keys take most reads, the rest a long tail) with a dominant key at 18% of reads, and look at per-shard load, not only the total: a uniform test would never expose the 18,480-versus-9,840 imbalance. Fourth, fault injection in a pre-production environment at realistic load: kill a primary and measure time to recovery and which writes were lost; partition a replica; add latency; fill memory; flush the whole cache and verify the admission cap protects the database; expire many keys at once; drop invalidation events and confirm the TTL bounds staleness. Each experiment needs a hypothesis and a pass condition, for example that the database stays under 19,200 reads per second during a full flush and the p99 for degraded classes stays within a stated limit. Record the findings and rerun them after major changes; a game day (a rehearsed, deliberate failure exercise) that runs once goes out of date.

• How would you test the race condition? — A deterministic test that pauses the loader between the database read and the fill, commits a write and an invalidation, then releases the loader and asserts that the old version is rejected.
• What would you monitor in production to catch what tests missed? — Hit rate by class and tenant, evictions, per-shard operations, replica lag, redirect rate, database QPS with an alarm below the ceiling, and a sampled version-mismatch counter.

At 1,000 requests per second, build very little. A single Redis or Memcached instance (plus a replica if restart time matters), cache-aside in the application, a TTL on every key with a little jitter, delete-on-write invalidation in the same code path that updates the database, and a size cap. In-process single-flight is a few lines and worth having. At 1,000 requests per second a database probably copes with the miss load, a cache outage degrades rather than destroys, and sharding, slots, regional caches and distributed leases would be cost without benefit. Watch hit rate and evictions and do not build the rest until the data demands it. At 100,000 requests per second, the same ideas hit limits. One node's operation budget (a hypothetical 25,000 per second in this guide) is exceeded, so shard, with slots and a client that follows redirects. The database cannot survive a cold cluster, so you need admission control, stale-if-error and a warm-up procedure. Skew becomes a certainty, so you need hot-key detection and L1 for a few keys. Replicas and failover matter because losing a shard is a regular event, and the invalidation path moves to an outbox and a stream so that it is reliable and observable. Multi-tenancy brings quotas and per-tenant metrics, and several regions bring a staleness budget. Between the two, add each piece when a measured symptom calls for it, in the order the symptoms appear: usually stampedes first, then a hot shard, then capacity.

• What is the first thing you would add after the minimum? — Per-class hit-rate and eviction dashboards; you cannot tune what you cannot see.
• At what size do you shard? — When one node's memory or operation budget is within reach, or when losing it would be unacceptable; the numbers in the capacity exercise are the method, not the answer.

Because everything that makes a cache effective is a reason it can lose data. It runs in memory with an eviction policy that deletes entries to stay under a limit, so a key you wrote can vanish tomorrow even before its TTL. Expiry deletes by design. Replication is asynchronous, so a failover can drop acknowledged writes, which Redis Cluster's own documentation describes. If persistence is turned on, snapshots and logs reduce the loss but do not turn it into a database with transactions across keys, backups, schemas, queries and audited recovery; atomicity is per command on a single node, so there is no cross-key or cross-shard transaction. Stale and missing entries are accepted by the design because the authority can rebuild them, and that is the property to protect: if the system would break when an entry is missing or older than the database, the cache has been made load-bearing for correctness. Write-behind is the usual path to that mistake: acknowledging users before the database has the data makes every cache failure a lost write. The test I would apply is whether you could flush the entire cache at 3 a.m. and have the product recover without data loss, only with higher latency and a controlled database load. If not, move that data into something durable and keep a cache in front of it.

• Is there a legitimate durable use of Redis? — Yes, as a database in its own right with its persistence and replication configured and tested for that purpose; that is a different deployment from a cache and needs a different contract.
• What about session storage? — Sessions are often kept in such stores; decide what losing them costs (users must log in again) and set persistence and replication accordingly, or keep them in a database.