URL Shortener — Failure & Scale Scenarios

Redis node/cluster crash, network partition between app servers and cache, or an operator-initiated failover with a gap in coverage.

All reads that would have been cache hits now query the database instead. Depending on the prior cache hit ratio, this can be a large multiplicative increase in database read load — if the cache was absorbing the large majority of traffic, the database could receive many times its steady-state read volume almost instantly. Do not assume this increase is automatically survivable; it depends on how much headroom the database and its replicas actually have.

If the database has headroom, redirects continue working with higher latency. If it does not, the database itself becomes saturated and redirect latency/availability degrades broadly — this is the dangerous cascading-failure case: Redis fails, all traffic falls through, the database overloads, and the overload can then affect requests that would otherwise have succeeded.

Cache client error/timeout rate spikes, database read QPS and connection-pool saturation climb sharply, redirect p99 latency degrades.

A circuit breaker on the cache client that fails fast (rather than waiting out a slow TCP timeout) prevents each request from stalling on a dead cache before falling through. Request coalescing limits duplicate database reads for the same code. A process-local cache, if present, continues serving its hottest entries without touching Redis at all. Bounding database concurrency (a request budget or queue limit) protects the database from being driven past its own failure point even while absorbing the extra load.

Redis restarts empty; it warms from real traffic as requests flow through it again. Uncontrolled instantaneous full fallback back onto a cold cache can itself cause a second stampede — a controlled warm-up/ramp (see Cold Cache / Cache Stampede below) is safer than assuming recovery is instantaneous.

Cache high-availability topology (replicated/clustered) where the added operational complexity is justified by the availability requirement; load shedding or rate limiting at the database boundary so a cache outage degrades gracefully rather than cascading; explicit, tested client timeout/circuit-breaker configuration rather than default library timeouts.

Primary node crash, storage failure, or an unplanned/planned failover event.

Cache-hit redirects continue working normally — they never reach the database. Cache-miss redirects fail, because there is nowhere authoritative left to read from. New URL creation fails completely (no primary to write to). Custom-alias creation fails for the same reason. Delete/disable operations fail similarly.

The fraction of redirects that continue working is exactly the fraction currently served from cache — this is workload-dependent, not a fixed percentage; a specific number (such as '92% of traffic') survives only if it reflects the actual, currently-measured cache hit ratio for this deployment. Any workload with a lower hit ratio, or a burst of first-time/expired lookups, sees proportionally more failed redirects.

Database connection errors on the write path and on cache-miss reads; primary health-check failures; replication-lag alerts on any surviving replica.

Serve an explicit, informative failure response (rather than a silent timeout) for operations that cannot succeed without the primary — including a Retry-After hint where appropriate. Cache-hit redirects require no special handling; they are already isolated from this failure.

Promote a replica to primary and repoint the application's write target. How long this takes and how automated it is depends entirely on the database topology and failover tooling in use — there is no universal recovery-time window; it can range from seconds with a well-tested automated topology to much longer with manual intervention.

Automated, tested failover tooling appropriate to the chosen database and topology; replication configured with an explicit, understood consistency guarantee (synchronous, semi-synchronous, or asynchronous) rather than an assumed one; monitoring that distinguishes 'primary down' from 'replica lagging' so operators know exactly what has failed.

A short link is shared to a very large audience simultaneously (viral social post, mass broadcast, high-traffic embed).

Failure order depends on the concrete deployment: cache warmth for that specific URL (a URL that has already received some traffic is likely already cached; a URL going viral within seconds of creation may not be), application-tier capacity and autoscaling responsiveness, load balancer/reverse-proxy connection and throughput limits, cache throughput and whether this URL's key is already hot before the spike, available network bandwidth, TLS termination capacity, how the analytics producer behaves under the additional event volume, and whether rate limiting is in place and where it is applied.

Whichever component saturates first determines the failure mode: elevated latency and eventual request queuing/rejection at that component, while components with headroom continue operating normally.

Per-component saturation metrics (app CPU/connection count, cache ops/sec for the specific key if observable, LB connection count, network throughput) read together, not any single metric in isolation.

Autoscaling the application tier with pre-warmed capacity where the traffic pattern is predictable; a process-local cache for the hottest keys removing load from the shared cache tier for exactly the requests that matter most during a spike; rate limiting at the edge/load balancer to protect the application tier from an inbound flood unrelated to the viral URL itself.

As traffic naturally decays or capacity catches up (autoscaling completes, cache fully warms), saturated components recover; no special data-repair step is typically needed since this is a load event, not a data-corruption event.

Autoscaling triggers based on measured resource pressure rather than a fixed threshold guess; a process-local cache tier for the hottest handful of mappings; edge/CDN caching for the specific case of an immutable, high-traffic redirect (see Complete Design → Production Evolution).

A single URL becomes extremely popular relative to all others, and every one of its reads lands on the same cache key.

The node/shard owning that key approaches its own throughput ceiling handling just this one key, even while the rest of the cache cluster has plenty of spare capacity — clustering/sharding spreads different keys across nodes, but does not automatically spread one key's traffic.

Elevated latency specifically for requests to this one code, potentially timeouts under extreme concentration, while requests for other codes remain unaffected.

Per-key or per-shard request-rate metrics (where available) showing one key or shard far above the rest; latency degradation isolated to one code rather than system-wide.

A process-local cache in each app server absorbs the majority of reads for the hot key without touching the shared cache at all. Read replicas, where the cache topology supports them, let multiple nodes each serve the same key. Edge/CDN caching of the redirect response removes the request from the origin entirely once warm, for genuinely immutable, cacheable redirects. Request coalescing addresses the case where the key has just gone cold and would otherwise stampede.

As traffic for the URL naturally declines, or once a mitigation (local cache, replica routing, edge caching) is in place, load on the single node/shard returns to normal.

Proactively identify likely-to-go-viral content where possible and pre-warm a process-local cache; maintain per-key/per-shard observability so a hot key is detected before it degrades latency broadly.

Consumer throughput falls below producer (click-event) throughput — under-provisioned workers, a traffic spike, or a downstream analytics-store slowdown that backs up the whole pipeline.

Backlog age grows. The redirect path is unaffected as long as the producer-side policy (bounded timeout, non-blocking publish, drop/sample on failure) is respected — see Architecture Deep Dive → Analytics Delivery Mechanics.

Analytics dashboards show stale click data. Redirects themselves are not affected.

Consumer group lag (age and/or event-count backlog), not just raw queue depth — backlog age matters more than raw event count for deciding urgency, since a large backlog of very recent events is a different situation from a smaller backlog that is very old relative to retention.

If retention/storage capacity comfortably covers the backlog, this is simply delayed analytics. If the backlog risks exceeding retention before being drained, older events are at risk of being dropped by the transport itself — at that point the system must choose explicitly whether to prioritize draining old events (preserve completeness) or fresh events (preserve currency), depending on product requirements.

Scale out consumers to exceed the current producer rate plus enough headroom to drain the existing backlog in a reasonable window. Replaying a large backlog can itself add load to the downstream analytics store — pace the drain if the store cannot absorb a sudden full-speed replay.

Autoscale consumers based on the lag metric itself (add capacity as lag grows, remove it once lag is low and stable); a fixed retention window (such as '24 hours by default') is not an architecture fact — retention is a configuration choice for the specific transport in use, and it should be set deliberately based on the acceptable backlog-drain window.

A cache restart, a newly deployed region with an empty cache, a popular key expiring, or a mass eviction under memory pressure — any event that clears cache state for one or more hot keys while traffic for them continues arriving.

Without coalescing, every concurrent request for the same now-cold code independently queries the database for the identical row, multiplying database load far beyond what the actual read volume would otherwise require — a database thundering herd.

Elevated latency concentrated on the specific codes that just went cold, and — if the herd is large enough — potential database saturation that spills over into unrelated requests.

A spike in database query volume for a small set of distinct keys, correlated with a recent cache restart/deploy/eviction event, rather than a broad, uniform increase in traffic.

Request coalescing/single-flight per key so only one database lookup happens per cold code regardless of concurrent request volume; a process-local cache continuing to serve any keys it already holds; bounding database concurrency so even an imperfectly-coalesced herd cannot fully saturate the database; a controlled recovery ramp (gradually reintroducing traffic to a freshly restarted cache) rather than assuming instant full-capacity fallback.

As coalesced lookups complete and repopulate the cache, subsequent requests for the same keys become hits again — recovery is typically fast (seconds) for a small number of hot keys, longer for a fully cold cache warming from real traffic.

Proactive cache warming for known-hot mappings ahead of a planned restart/deploy; jitter on TTLs so a large set of entries does not expire in the same instant, which would otherwise manufacture a synchronized stampede out of ordinary expiry.

Cache population after creation fails, is skipped, or the entry is evicted before an app server needs to read it — and the resulting read is routed to an asynchronously-replicating replica that has not yet caught up to the primary's most recent commit.

The read replica correctly reports 'no such row' from its own current state, even though the mapping exists on the primary — the replica is not wrong given what it has replayed so far, it is simply behind.

A just-created short link intermittently returns 404 for a brief window immediately after creation, which is a particularly bad experience since new links are often shared and clicked immediately.

Replication lag metric on the replica; a pattern of 404s correlated with very recent created_at timestamps once investigated.

Populate the cache immediately after the primary commit succeeds, so most reads never reach a replica at all during the risk window. Route reads to the primary for a short window immediately following a write for that specific mapping ('read-after-write' routing). Use synchronous or semi-synchronous replication if the product genuinely cannot tolerate any staleness window. A short, bounded retry on an unexpected 404 for a very recently created mapping can also paper over a brief lag window, if acceptable.

The replica catches up automatically as it continues replaying the write-ahead stream; no manual intervention is normally required for an isolated lag event.

Monitor replication lag as a first-class metric; do not silently assume replication is synchronous — verify and document the actual guarantee the chosen topology provides. See Architecture Deep Dive → Database Access & Indexing and Trade-offs → Read from Primary vs Read Replica on Cache Miss.

Namespace occupancy (the fraction of the keyspace already in use) rises over time as more mappings are created, increasing the per-attempt probability that a newly generated random code is already taken.

The database UNIQUE constraint prevents any actual data corruption — a collision on insert simply fails to insert and returns zero rows affected. The application detects this and retries with a new random code within a bounded retry budget.

Under normal, low-occupancy conditions, negligible — most creations succeed on the first attempt. As occupancy rises, average retries-per-creation rises too, adding a small, growing amount of latency to the write path.

A collision-rate/retry-count metric on the creation path trending upward over time is the leading indicator — track this explicitly rather than discovering the problem only when retries start failing outright.

The bounded retry loop absorbs a reasonable number of collisions gracefully. If retries are repeatedly exhausted (which would only happen at very high occupancy relative to the keyspace), the creation request should fail explicitly rather than looping indefinitely, and the increasing exhaustion rate itself is the signal to act.

Increase code length before occupancy becomes operationally expensive — a longer code multiplies the keyspace and drives the per-attempt collision probability back down. This is a proactive capacity decision, not a reactive one; see Architecture Deep Dive → Short-Code Generation.

Two concurrent creation requests specify the identical custom alias, arriving close enough in time that a naive check-then-act sequence would race.

A pre-check such as 'SELECT to see if the alias is already taken, then INSERT if not' has a race condition: both requests can pass the SELECT before either has inserted, and both proceed to attempt the INSERT.

The database's UNIQUE constraint on the short-code/alias column is the final, atomic arbiter. Both requests attempt the INSERT; the database guarantees exactly one succeeds, and the other receives a constraint-violation response that the application translates into an explicit conflict response to that caller.

One caller receives their requested alias successfully; the other receives a clear conflict (e.g. 409) rather than silently failing, silently overwriting, or nondeterministically 'winning' based on request timing.

Constraint-violation rate on the creation path for alias requests specifically — a healthy signal, not an error to eliminate, as long as it correctly results in an explicit conflict response rather than a corrupted or duplicated mapping.

Rely on the database's atomic uniqueness guarantee as the correctness mechanism, not application-level locking or pre-checks — this scales correctly under concurrency without additional coordination infrastructure.

A cache entry was populated with a fixed TTL that was not bounded by the mapping's own expires_at, so the cache entry outlives the mapping's authoritative validity.

Requests for the expired short code continue to be served directly from cache — the authoritative store would correctly report it as expired, but the cache is never consulted because the entry is still 'valid' by the cache's own (incorrect) TTL.

An expired link keeps working past its intended expiry, which can range from a minor inconsistency to a meaningful security/product problem depending on why the link was set to expire.

A discrepancy between database-reported expired mappings and continued redirect traffic for those same codes, if monitored; otherwise this tends to surface only when specifically tested or reported by a user.

Bound cache TTL by remaining mapping lifetime (cache_ttl = min(configured_cache_ttl, expires_at - now)), and where expiry metadata is cached alongside the mapping, verify it at serving time as a defensive guard rather than trusting cache presence alone. Background deletion of expired rows is storage hygiene — it is not, and must not be relied upon as, the mechanism that enforces expiry; the authoritative read path itself must treat an expired row as expired. See Architecture Deep Dive → Cache Expiry Correctness.

Once the TTL bound is corrected, existing over-long cache entries eventually expire naturally; if immediate correction is required, explicitly invalidate cached entries for affected mappings rather than waiting out their (incorrect) TTL.

Broker outage, network partition between app servers and the broker, or the broker itself becoming overloaded and slow to accept publishes.

Every publish attempt on the redirect path either blocks, times out, or fails — depending entirely on the producer configuration in place.

If the producer blocks or retries unboundedly waiting for the broker, redirect latency and eventually redirect availability degrade along with it — this must never be allowed to happen, since analytics is not supposed to be able to take down redirect serving.

Redirect anyway and drop or sample analytics events for the duration of the outage; buffer events locally within a strict, bounded size/time limit and flush once the broker recovers; or, if analytics loss is genuinely unacceptable for the product, accept a durable-retry policy with its associated stronger infrastructure and cost requirements. Whichever is chosen, it must be a bounded, tested behavior — not an emergent one.

None on the redirect path, if the producer policy is correctly bounded and non-blocking. Analytics dashboards show a gap or reduced granularity for the outage window.

Publish error/timeout rate on the producer side; broker health metrics; analytics event-volume drop correlated with the outage window.

Bounded publish timeouts and a non-blocking producer configuration verified under test, not just assumed from the client library's defaults. See Architecture Deep Dive → Analytics Delivery Mechanics.

Automated scanning of the short-code namespace looking for valid/interesting mappings, or bulk automated creation of short URLs for spam/phishing purposes.

A large volume of lookups for codes that do not exist bypasses any positive cache (nothing to cache-hit) and hits the database directly for every single request, unless negative caching is in place. Bulk creation similarly stresses the write path and consumes namespace.

Degraded latency for legitimate traffic if the scan/abuse volume is large enough to compete for database or application capacity.

A high 404/not-found rate concentrated from a small number of sources or IP ranges; creation-rate anomalies per IP/account; negative-cache hit-rate spikes (if instrumented) indicating heavy scanning.

Rate limiting per IP/account on both lookups and creation; negative caching of not-found results so repeated scans for nonexistent codes are absorbed by the cache rather than reaching the database each time; a creation quota to bound how many URLs a single actor can produce; WAF/abuse-control rules for clearly automated traffic patterns; a malware/phishing-disable flow so abusive destinations can be disabled after detection independent of the scanning defense.

Once rate limiting/blocking takes effect, database and application load returns to baseline; disabled/flagged malicious mappings stop resolving via the disable flow rather than requiring deletion.

Treat negative caching and rate limiting as standard parts of the architecture, not optional add-ons — see Architecture Deep Dive → Cache Hierarchy.