Retry Storm

A retry storm occurs when retries — meant to recover from transient failures — add enough extra traffic to an already-struggling dependency that they make it worse instead of better.

Retries exist for a good reason. Networks drop packets. Services have momentary blips. A single failed request, retried once, quietly recovers and the caller never notices. This is healthy retry behavior — rare, cheap, mostly invisible. The general mechanics live in Retry & Backoff; this page is about what happens when that pattern is applied without enough restraint.

The problem starts when a dependency doesn't just blip — it degrades under load. Latency climbs. Requests start timing out. Callers retry those timeouts. But the retries don't land on a healthy service recovering from a hiccup — they land on a service that is already struggling, adding more load at the worst possible moment.

That's the difference between healthy retries and retry amplification: healthy retries recover from rare, independent failures. Retry amplification happens when the failures are caused by load, and the retries add more load — closing a feedback loop that feeds on itself.

A retry storm needs three ingredients: a dependency that degrades gradually under load, a retry policy that reacts without regard for that degradation, and enough concurrent callers for the extra load to matter.

The critical insight is that this is a feedback loop problem, not a retry problem. Retrying is the right instinct for a transient failure. The danger is a retry policy that can't tell the difference between "this failed because of bad luck" and "this failed because the dependency is drowning" — and reacts to both the same way: immediately, and without limit.

Where does "280 calls" come from? The numbers are illustrative. Suppose a policy allows at most 3 attempts per request (1 original + 2 retries). All 100 requests time out on the first attempt, so they send 100 first retries; 80 of those time out again and send 80 second retries. Total calls = 100 + 100 + 80 = 280, an amplification of 280 ÷ 100 = 2.8 attempts per request. The worst case for this policy is 3 × 100 = 300. If two layers each retry up to 3 times (say the gateway and Service A), a single user request can become up to 3 × 3 = 9 calls on the dependency, which is why retries must be budgeted across layers, not per layer.

Press play to watch a retry storm unfold in a simulated system — from a healthy call chain, through gradual degradation, timeouts, retry amplification, mitigation, and full recovery.

A retry storm can turn a minor slowdown into a full outage — and it gets worse exactly when engineers most need the system to stabilize.

Because the loop is self-reinforcing, a retry storm often keeps getting worse until something changes the traffic pattern. Every retry consumes threads, connections, and CPU on both sides of the call — the caller blocks waiting for a response it may never use, and the callee spends resources on requests that have often already been abandoned or retried elsewhere.

The same causes recur in many retry storm postmortems:

• No retry limit — a request can be retried indefinitely.
• Aggressive retry policy — retrying on almost any failure, including ones that will not resolve by retrying.
• Very short (or zero) delay between retries.
• Many clients retrying simultaneously, in near-perfect sync.
• No exponential backoff — every retry adds the same load as the last.
• No jitter — synchronized clients stay synchronized, wave after wave.
• Retrying non-idempotent operations, risking duplicate side effects.
• Retrying into a dependency that is already visibly overloaded, instead of backing off.

When a caller blocks on a dependency this way, its own callers start timing out too — the pressure spreads upstream through the whole call chain instead of staying isolated to one slow dependency. See Cascading Failure for how that propagation happens in detail.

How to spot it

The tell-tale sign is that the dependency's load stops tracking user demand. Watch these signals:

1. Retry rate — client-side retries per second, or the share of attempts that are retries, rising from near zero.
2. Attempts per request (the amplification ratio) = total attempts ÷ original requests. Healthy is close to 1.0; a sustained value of 2 or more means the retry policy is doing a large share of the work.
3. Dependency request rate versus edge request rate — callee RPS climbs while user-facing RPS stays flat, and callee saturation (CPU, pool, queue depth) follows.
4. Timeouts and latency — timeouts at the caller while the callee's own error log shows many requests it completed after the caller had given up.

Five techniques address different parts of the loop. Production systems almost always combine several of them. Timeouts bound a single attempt; retry limits or deadlines bound how much total time and traffic a caller spends across all attempts.

The lesson is not "retries are bad." It's "uncontrolled retries can amplify an outage" — and not every operation should be retried the same way.

A read — fetching inventory, looking up a user — is usually safe to retry when the failure looks transient, like a timeout, a connection reset, or a 503. A validation 400 or an auth failure usually is not. But a non-idempotent write — charging a card, sending a notification, incrementing a counter — is a different story. If the original request actually succeeded but the response was lost, a naive retry can perform the same side effect twice.

The standard fix is an idempotency key: the caller generates a unique key per logical operation and sends it with every attempt, including retries. The receiving service uses that key to recognize "I've already done this" and returns the original result instead of repeating the side effect.