Cascading Failure

A cascading failure occurs when a problem in one component causes the components that depend on it to degrade, which then causes the components that depend on THOSE to degrade — turning a small, localized problem into a broad, system-wide failure.

The chain is often as ordinary as it gets: Service A calls Service B, which calls a database. If Service B becomes slow, Service A waits longer on every call it makes. Those slower calls keep A's worker threads, outbound connections, or concurrency slots occupied for longer than planned. Once enough of that shared capacity is tied up, Service A starts queueing or timing out its own requests — and now its callers start waiting on IT.

The critical thing to understand is that the original failure does not need to be a complete outage. A dependency going from 40 milliseconds to 2 seconds is enough. It is still "up." It is still returning correct answers. It is simply asking everyone who depends on it to wait much longer than they budgeted for — and that wait is what spreads.

A cascading failure needs a slow dependency, a caller that shares finite resources across all of its work, and nothing in place to bound the wait or isolate the damage.

Put together, this is a resource-sharing problem, not a single-point failure. Service B didn't take Service A down directly — Service A ran out of its own worker, connection, or queue capacity because too much of that capacity was tied up waiting on Service B. Aggressive, uncontrolled retries make it worse: they add traffic onto a dependency at the exact moment it can least absorb it.

Press play to watch a cascading failure unfold in a live request chain — from a healthy system, through a gradual downstream slowdown, resource exhaustion, upstream propagation, mitigation, and full recovery.

A cascading failure can turn one degraded dependency into a system-wide outage — and the symptom users see is often several hops away from the actual cause.

Because every hop in the chain shares finite resources with every other hop, the damage doesn't stay contained to where it started. Threads and connections held open on one service are threads and connections that service can't use for anything else — including requests that have nothing to do with the failing dependency.

The same causes recur in many published cascading-failure postmortems:

• No timeout on a downstream call — or one so long it fails to release resources before they pile up.
• A shared resource pool with no isolation between dependencies.
• Unbounded queues that let requests pile up instead of getting rejected.
• Aggressive retries that add more traffic onto an already-struggling dependency.
• No circuit breaker — every caller keeps attempting calls to a service that clearly cannot serve them.
• No backpressure — upstream callers are not told to slow down, so work keeps arriving faster than the system can drain it.
• No prioritization — critical and non-critical traffic compete for the same shrinking capacity.

How to find the real cause

Start from the edge errors, then walk down the call chain and look for the deepest hop that went bad first. Four signals, checked per hop, usually point to it:

1. Per-hop latency (from traces or per-dependency client metrics) — the culprit is the lowest hop whose own latency rose; every hop above it only inherits that wait.
2. Thread or connection-pool saturation — in-use at its maximum and callers waiting for a slot, first at the hop next to the culprit, then one hop up at a time.
3. Queue depth — a backlog that grows without draining means arrivals exceed what the hop can complete.
4. Circuit-breaker state — a breaker that never opened (or doesn't exist) explains why callers kept waiting; one that opened tells you which dependency was judged unhealthy.

Five techniques address different parts of the mechanism. Resilient systems almost always combine several of them.

Containing a cascade protects the system. Graceful degradation protects the response — the goal usually isn't "never fail," it's "fail in a controlled way."

When a non-critical dependency is unavailable, the entire request does not necessarily need to fail with it. A product page can render without a recommendations widget. A dashboard can load without analytics. An API response can go out without profile enrichment. Deciding in advance what "good enough" looks like, without a given dependency, is what makes this possible.

• Recommendation service unavailable → show the page without recommendations.
• Analytics unavailable → continue serving the core request.
• Profile enrichment unavailable → return a basic response instead of failing entirely.