Retry & Backoff

A retry is a new attempt at an operation after an earlier attempt failed or left its outcome unknown. Exponential backoff spaces those attempts further and further apart, with delays that grow geometrically up to a configured cap. Jitter randomizes each delay so that many clients do not come back at the same instant. Used together they let a system absorb brief failures while giving a struggling dependency room to recover.

Retries exist because a large share of failures in a distributed system are brief: a connection reset, a replica being replaced during a rollout, a DNS lookup that took 200 ms instead of 2, a load balancer draining a node. Without retries each of these reaches the user as an error. With them most disappear. The cost is that a retry is not free recovery. It is additional work sent to a component that may already be failing, so the benefit and the risk come from the same mechanism.

The mental model: a retry spends capacity to buy availability. Every decision in a retry policy answers one of three questions. Is this failure likely to go away (classification)? Is it safe to do the operation again (side effects)? Can the system afford the extra load, and can the caller still wait (budget and deadline)? A policy that answers only the first, "retry three times," is incomplete in ways that turn small incidents into large ones.

Consider a checkout service calling inventory. If a connection is reset before the server ever saw the request, one delayed retry will probably succeed. If instead the call times out after inventory committed the reservation but before the response arrived, the caller cannot tell from the timeout whether the reservation exists. The same retry mechanism now risks reserving stock twice unless the operation carries a stable identifier. The rest of this lesson is about telling those two cases apart and bounding what the retries are allowed to cost.

A transient failure may disappear without changing the request: a dropped connection, a replica that is briefly unavailable, a 503 Service Unavailable during a rollout. A permanent failure will not be repaired by repeating the same input: bad credentials, a schema violation, a business rule such as insufficient balance. An ambiguous failure is one where the client does not know whether the server performed the action. Timeouts, and connection loss after the request was sent, usually fall here. Ambiguity is the dangerous category, because the correct response depends on the operation, not on the error.

Protocol codes are signals, not a policy. They have to be read together with what the operation does and what the service promises.

Prefer an explicit allowlist of retryable conditions over a blocklist. A blocklist silently retries every new error type someone adds later. HTTP itself draws the line the same way: a client may automatically repeat a request after a connection failure only when the method is idempotent, or when it knows the request was not processed. A POST that failed after being written is neither.

Database deadlocks and serialization failures are a good example of a genuinely transient application-level error. The database aborted the transaction and says so explicitly, and the work is safe to repeat in full because nothing was committed. A short delay with jitter, since the cause is contention between concurrent transactions, is usually enough, with a small attempt limit.

The common capped schedule is:

delay(n) = min(maxDelay, baseDelay × 2^n)        n = 0 for the first retry

With a 200 ms base and a 3,000 ms cap the delays for successive retries are 200, 400, 800, 1,600 ms, and then 3,200 ms capped to 3,000 ms. These are waits between attempts, not the duration of each request. The cap matters as much as the base: without it the delay would reach minutes after a handful of failures, far beyond any caller's deadline. Try other values below. Note that the worst-case total wait is the sum of the individual delays, which is dominated by the last ones.

Why grow at all? A fixed delay treats a one-second blip and a ten-minute outage identically: it hammers the dependency at the same rate for as long as the failure lasts. Linear backoff (1 s, 2 s, 3 s, 4 s) reduces the pressure slowly. Exponential backoff reduces the retry rate by half with each attempt, so the load from a persistently failing client decays quickly, while a client that fails only once still retries almost immediately. That asymmetry is the point: brief failures recover fast, long ones stop consuming capacity.

Backoff alone does not break synchronization

A deterministic schedule is the same for every client. Suppose 10,000 mobile devices lose connectivity during a regional network interruption and all see the error at the same moment. With the schedule above they retry at 200 ms, then 600 ms (200 + 400), then 1,400 ms (200 + 400 + 800) after the failure, ignoring request duration. The retries are spaced out in time, but they are still aligned across clients, so the dependency sees three sharp bursts of 10,000 requests, each potentially bigger than the load that caused the problem. Exponential growth reduces how often clients retry; it does nothing about when they do. That is the thundering herd, and it is what jitter is for.

Full jitter draws the wait uniformly between zero and the capped exponential delay:

cap  = min(maxDelay, baseDelay * 2^retryIndex)
wait = random(0, cap)            // mean wait is cap / 2

It is the widest spread: a fleet that would all have retried at 200 ms now retries uniformly across the 0 to 200 ms window, so the peak arrival rate at the dependency drops by roughly the width of the window divided by the width of one measurement slice. The trade-off is that some clients retry almost immediately, and the average wait is half the nominal delay.

Two common alternatives. Equal jitter keeps a floor under the delay: half = cap / 2; wait = half + random(0, half). No client retries sooner than half the nominal delay, at the cost of a narrower spread. Decorrelated jitter derives each delay from the previous one instead of from the attempt number: wait = min(maxDelay, random(baseDelay, prevWait × 3)). It needs per-call state but produces delays that keep growing on average while remaining unsynchronized. Their behavior under heavy contention differs, and there is no universal winner. What matters is that some randomization is applied once many independent clients can fail together.

The figure idealizes the situation: every retry fails, so the three waves stay visible, and the arrival counts are expected values rather than a random sample. Even so, the picture matches what happens in production. Without jitter the dependency sees the whole fleet land in a single 25 ms slice, three times. With jitter the first wave is spread over 200 ms, and later waves smear into each other because each client's total delay is a sum of random draws. The load profile approaches a smooth plateau instead of a series of spikes.

Jitter helps in other places too. A scheduled job that all instances run at the top of the minute, a cache entry with a fixed TTL that expires everywhere at once, a reconnect loop after a deploy: any time many clients share a trigger, a small random offset removes the alignment. The randomized wait must still respect the caller's remaining deadline, which the next section covers.

Two clocks matter. A per-attempt timeout bounds how long one call may take. A deadline bounds the whole operation as the caller experiences it, including waits between attempts. Retries must be budgeted against the remaining deadline, not a fresh one for each attempt.

Suppose a user-facing API has a total 1,200 ms deadline and calls its dependency with a 250 ms per-attempt timeout. The first call times out after 250 ms. The client waits 120 ms and tries again. The second attempt returns 503 after 100 ms, ending at 470 ms. The client waits 260 ms and starts a third attempt at 730 ms. At that point 470 ms of the deadline remains, so the third attempt may use at most min(250, 470) = 250 ms and the call still has headroom for processing the response.

This is why a policy written as "three retries" is incomplete. Four total attempts at 250 ms each, plus waits and connection setup, can exceed a user's latency budget by itself. State explicitly whether a setting counts total attempts or additional retries; libraries differ, and a setting of 3 means four calls in some and three in others. The same applies when a wait is longer than what is left: if a drawn delay is greater than the remaining deadline, fail now. Sleeping past the deadline and then failing wastes the caller's time and holds resources for nothing.

Choosing the per-attempt timeout is a trade-off. Too short and healthy but slow responses are cut off and retried, adding load exactly when latency is rising. Too long and a single hung attempt consumes the whole budget, leaving no room for a retry. A common starting point is a little above the dependency's observed p99 latency on a healthy day, then adjusted against the deadline. Treat that as a heuristic to tune, not a rule.

async def call_with_retry(operation, deadline, max_attempts=3,
                          base=0.1, cap=1.0, per_attempt=0.25):
    for attempt in range(max_attempts):
        remaining = deadline - monotonic_now()
        if remaining <= 0:
            raise DeadlineExceeded()
        try:
            return await operation(timeout=min(per_attempt, remaining))
        except Exception as exc:
            last = attempt == max_attempts - 1
            if last or not is_retryable(exc) or not retry_budget.try_spend():
                raise
            delay = uniform(0, min(cap, base * 2 ** attempt))   # full jitter
            delay = max(delay, retry_after_seconds(exc))        # honor server hint
            if delay >= deadline - monotonic_now():
                raise DeadlineExceeded()
            await sleep(delay)

The loop uses a monotonic clock so that wall-clock adjustments cannot stretch the deadline. It is conceptual: a real implementation needs correct exception classification, cancellation propagation, connection pooling behavior, and metrics. For side-effecting operations it must also reuse one idempotency key across the loop, covered below.

Timeouts do not cancel work

When a client gives up on an attempt, the server often keeps working on it. The caller now has a second attempt in flight while the first still consumes CPU, locks, and connections downstream. Propagate cancellation or the remaining deadline where the protocol supports it (gRPC deadlines propagate automatically; HTTP needs a header convention or connection close). Do not assume cancellation rolls back a committed side effect: a cancelled request that had already written to the database is still written.

Retry-After is defined by HTTP and may be either a number of seconds or an HTTP date. It commonly accompanies 429 and 503 and tells the client when the server expects to have capacity again. A client should wait at least that long. Using the larger of the server's hint and the local backoff delay is the safe combination, as the loop above does.

Distinguish the two codes. 429 Too Many Requests says this caller has exceeded an allowance, so retrying sooner just consumes more of the allowance. 503 says the server cannot serve right now regardless of who is asking. The first is a quota conversation covered in Rate Limiting; the second is overload, where the right response is often to shed load on the server side and back off on the client side.

Two limits on trusting the hint. First, do not blindly obey a very long delay that exceeds the operation's business deadline; if the server says 300 seconds and the user is waiting, fail fast or hand the work off. Second, if thousands of clients receive the same absolute retry time, honoring it exactly recreates a herd, so add bounded jitter on top where the service contract allows it. Some APIs also publish rate-limit state in response headers such as RateLimit-Remaining; those header names come from an IETF draft that has changed between revisions, so rely on each provider's documentation rather than assuming a common format.

On every failure the client runs the same short decision sequence: classify, check the budget, check the deadline, wait, then attempt again with the same identity. A failed check at any step returns the last error to the caller.

The parameters of the policy follow from that flow:

• Maximum attempts. Usually small. Three to five total attempts is a typical range; each additional attempt adds load while raising the chance of success only if failures are independent and brief. Count total attempts or retries, and say which.
• Base delay and cap. The base sets how soon the first retry happens, often 100 to 500 ms for service calls. The cap bounds any single wait and must be consistent with the deadline.
• Jitter. Full jitter is a sound default. Apply it to every wait, including those derived from Retry-After when the contract allows.
• Retryable conditions. An explicit allowlist, such as connect failures, 429 with a hint, 503, and UNAVAILABLE, narrowed by whether the operation is safe to repeat.
• Per-attempt timeout and total deadline. Both, with the deadline winning.
• Retry budget. A cap on additional attempts relative to original traffic, described next.

Express the policy in one place per call site or per dependency, not scattered across code. Policies that live only as library defaults are the ones nobody can account for during an incident.

Consider a request path Gateway → Orders → Inventory → Database. If each of the three upstream layers independently performs up to three total attempts against the next, one gateway request can lead to as many as 27 database attempts in the worst case, before counting any other fan-out. The calculation assumes every attempt fails and reaches the next layer, so it is an upper bound, not a typical value.

The multiplier depends on how often attempts fail. If each attempt at the bottom fails independently with probability p and a layer makes up to N attempts, the expected number of attempts per call is 1 + p + p² + … + p^(N−1) = (1 − p^N) / (1 − p). At p = 0.5 and N = 3 that is 1.75 attempts for one layer, and because the next layer up only retries when the layer below ultimately failed (probability p³ = 0.125), the total over three layers is only about 2×. At p = 1 every layer exhausts its attempts and the multiplier is N per layer, so N³ = 27. Amplification is therefore mild during scattered errors and severe during a sustained outage, which is precisely when the dependency can least afford it. Failures in practice are correlated, not independent, so the independent-failure rows below understate real load.

Little's law shows why the extra requests hurt beyond throughput. In-flight work is L = λ × W: arrival rate times time in the system. A dependency serving 1,000 requests per second at 50 ms holds about 50 requests in flight. If an incident triples the arrival rate to 3,000 attempts per second and responses stall until the 250 ms client timeout, in-flight work becomes roughly 3,000 × 0.25 = 750, fifteen times higher, which can exhaust thread pools, connection pools, and queues. (Illustrative figures.) Retries raise both factors at once.

Choose a retry boundary

Prefer a deliberate retry boundary over retries everywhere. The service closest to the failure often knows best whether a retry is safe and whether the error is transient. The top-level caller owns the end-to-end deadline and the user context. The right answer depends on architecture, but the wrong one is common: enabling automatic retries at the application, the SDK, the service mesh sidecar, and the gateway at once, each unaware of the others. Audit retry defaults at every layer and decide which one retries. Inner layers can then fail fast and report that they did so.

Retry budgets

A retry budget limits additional attempts relative to original requests or available capacity. If a client sends 1,000 original requests per minute and allows at most 100 additional attempts per minute, retries cannot silently double downstream load, however badly the dependency is failing. During a small blip the budget is not binding and every retry goes through; during a major outage it caps the amplification at 1.1×, and the remaining requests fail fast.

The budget is a policy, not a particular algorithm. A token bucket is a common implementation: successful or original requests deposit a fraction of a token, each retry withdraws a whole one, and an empty bucket means no retry. gRPC's retry throttling works like this: a failed RPC removes one token, a success adds tokenRatio, and retries stop while the bucket is at or below half of maxTokens. Envoy expresses a similar idea as a retry budget relative to active requests. The exact size must reflect traffic, SLOs, and the dependency's headroom.

A retry budget and a rate limit are different controls. A rate limit protects a server from one caller's total demand. A budget protects a dependency from a caller's own retries, by making retries a bounded fraction of that demand. The two compose but neither replaces the other: compliant callers each within their limits can still overload a dependency collectively, and a budget does nothing about first attempts.

An operation is idempotent when applying it multiple times has the same intended effect as applying it once. GET is safe; PUT and DELETE have idempotent HTTP semantics, even if side effects like audit entries accumulate. POST is not idempotent by default. For a payment or a reservation, a timeout leaves the caller in the ambiguous case: the work may have completed and only the response was lost.

The standard solution is an idempotency key: a unique identifier generated by the client before the first attempt and reused on every retry of that logical operation. The server records the key with the result. A repeat of the same key returns the stored result instead of performing the work again. Four details decide whether this actually works:

• Atomic claim. Concurrent requests with the same key need a uniqueness constraint or atomic compare-and-set. A check-then-insert sequence lets two simultaneous retries both pass the check.
• Scope and payload binding. Scope keys to a caller and an operation, store a fingerprint of the request, and reject reuse of the same key with a different payload.
• Commit point. Define when the result is recorded relative to the side effect, what a retry sees while the first attempt is still in progress, and what happens if the worker crashes in between.
• Retention. Keys are kept for a bounded period. A retry after that window is a new request, so the client's retry horizon must be shorter.

INSERT INTO idempotency_keys (caller_id, key, request_hash, status)
VALUES (:caller, :key, :hash, 'in_progress')
ON CONFLICT (caller_id, key) DO NOTHING;

-- rows_affected = 1  → we own it: do the work, then UPDATE status='completed', store response
-- rows_affected = 0  → read the row:
--     request_hash differs → reject (key reused with a different payload)
--     status = completed   → return the stored response
--     status = in_progress → tell the caller to retry later (for example a 409 response)

A key reduces duplicate business effects; it does not make an external payment provider or a multi-step downstream transaction atomic. The implementation must still define how an unknown outcome is reconciled, for example by looking up the operation by its key after the retry window. Where an API offers no key and no natural deduplication, do not automatically retry a side-effecting call after an ambiguous failure. Surface the uncertainty or reconcile instead. See Idempotency for how the key store works in detail.

Retry when the failure is plausibly transient, the operation is safe to repeat, the deadline permits another attempt, and the dependency has capacity to recover. Fail fast when validation fails, a circuit breaker is open, the deadline has nearly expired, or the retry budget is exhausted. Hand off to a durable queue when the work is not user-interactive and can complete asynchronously. That moves retries out of the request path into workers, with their own delivery and deduplication concerns (see Queues & Workers, and Dead-Letter Queues for work that keeps failing).

Retry storm after recovery. A database recovers, then synchronized callers immediately overwhelm it again, and it falls over a second time. A recovered dependency is not healthy until the returning traffic is manageable. Jitter, per-client concurrency limits, a retry budget, and admission control on the server all help, and a breaker in a half-open state that admits a few probes at a time is the controlled version of this.

No retry limit, or an unbounded queue of retries. A dependency is down for four hours and clients, or a retry queue, keep trying. Traffic never stops, memory grows with the backlog, and the dependency cannot recover. Every retry loop needs a maximum attempt count and a maximum age. Work that exhausts them goes to a dead-letter destination or is reported as failed. A circuit breaker stops the flood earlier by refusing calls once failures cross a threshold.

Timeout without cancellation. A caller times out and retries while the first server operation continues. Both attempts now consume resources, and if the work has side effects both may run to completion.

Nested retry loops. The application, the SDK, and the proxy each retry independently, turning three attempts into dozens as shown earlier. Whoever tunes one layer is unaware of the multiplication.

Duplicate side effects. A POST /payments is retried after a timeout without an idempotency key and the customer is charged twice. The key must be generated before the first attempt and reused, not regenerated per attempt, or the protection disappears.

Retrying deterministic errors. Repeated 500 responses come from a serialization bug, or a 400 is retried automatically. Every retry returns the same error, adding latency and load without a single additional success. Track retry success by cause, not just retry counts.

No jitter. Every client uses the same backoff configuration and all retry at the same instant after a shared failure. The result is synchronized spikes that look like a periodic attack on the dependency.

How real systems apply the pattern

AWS SDKs document retry modes (standard and adaptive) that apply exponential backoff with jitter and a client-side limit on retry volume. Many Google Cloud client libraries retry transient errors automatically and avoid retrying deterministic client errors. Stripe accepts client-supplied idempotency keys on mutating requests so a lost response cannot produce a duplicate charge, and prunes keys after a retention window. Kubernetes applies the same principle to restarts: CrashLoopBackOff doubles the delay between container restarts up to a cap of five minutes, so a crashing pod does not restart in a tight loop. Check each provider's documentation for current defaults before relying on them.

A checkout service authorizes a card through a payment API. Policy: 3 total attempts, base 200 ms, cap 3 s, full jitter, per-attempt timeout 800 ms, overall deadline 3 s, and a single idempotency key K-7f3 created before the first call. All timings are illustrative.

• t = 0 ms. Attempt 1 sent. The API returns 503 after 120 ms (t = 120). Transient, budget available, deadline fine.
• Wait 1. Cap = min(3,000, 200 × 2⁰) = 200 ms. Drawn wait 140 ms, so attempt 2 starts at t = 260.
• t = 260 ms. Attempt 2 is sent with the same key. It times out at 800 ms (t = 1,060). The outcome is now ambiguous, and this is exactly where the key matters.
• Wait 2. Cap = 400 ms. Drawn wait 310 ms, so attempt 3 starts at t = 1,370.
• t = 1,370 ms. Attempt 3 reaches the API, which finds K-7f3 already completed by attempt 2 and returns the stored 201 in 90 ms. The call finishes at t = 1,460 ms with one charge.

Compare the instant-retry version: three attempts within a few milliseconds all hit the same struggling API, all fail, and the user sees an error after almost no time, with the dependency having received triple load in the worst moment. If 1,000 clients behave this way together the API sees about 3,000 requests where the baseline was 1,000, and nothing was gained because the wait was too short for the failure to clear. The backoff version took 1.46 s of the 3 s deadline, spread its load, and, because of the key, was safe even though attempt 2's result was unknown. Had attempt 3 also failed, the 3-attempt limit would have ended the call there and returned the last error, with the key still available for a later reconciliation.

Measure attempts per original request (total attempts divided by first attempts), retry rate by error class, first-attempt success rate, eventual success after retry, added latency from retries, deadline-exhaustion count, retry-budget rejections, and saturation of the downstream dependency.

The most important trap is a healthy-looking aggregate. High eventual success can conceal a dependency that is getting worse if first-attempt success is collapsing: users see success, but each success costs more load and more latency, until the budget runs out and everything fails at once. Alert on first-attempt success and on attempts per request, not only on final error rate. A rise in attempts per request from 1.02 to 1.4 is an early signal long before errors reach users.

Break metrics down by dependency and operation, and by error cause, so a deterministic bug is separable from transient noise. Avoid unbounded labels such as raw user IDs or full URLs; metric cardinality is its own outage. For retried mutations also track idempotency-key hits (how often a retry found a completed result): a spike means responses are being lost or timeouts are too aggressive.

Retry is the only pattern here that acts on one request after it fails. The others act on a population of requests, and a retry policy is only safe when the others are present. The circuit breaker answers whether a dependency should receive calls at all, and retries consult it before every attempt. Rate limiting and load shedding give the server a way to refuse excess load, and a well-behaved client treats those refusals as instructions to slow down. Backpressure carries the same message inside a pipeline. Bulkheads stop one caller's retries from consuming the capacity of unrelated work.

The retry and circuit breaker pairing deserves one more sentence. Retry handles the brief blip; the breaker handles the sustained outage. Without a breaker, retries keep flooding a dependency that is down for hours. Without retries, a breaker trips on failures that a single repeat would have cleared. Which failures count toward the breaker is a design decision of its own: a failure that was retried successfully is a different signal from one that exhausted its attempts.

• Decide whether to retry before deciding how: retry transient failures, never deterministic client errors, and treat ambiguous outcomes (timeouts) as possibly already applied.
• Use capped exponential backoff with jitter; backoff alone keeps independent clients synchronized, and full jitter spreads them.
• Fit the whole policy inside the caller's deadline and honour Retry-After within limits. A timeout does not cancel the work already started.
• Retries multiply across layers: three attempts at each of three layers can mean up to 27 database attempts. Choose one retry boundary and cap extra load with a retry budget (for example at most 100 extra attempts per 1,000 requests).
• Retrying a non-idempotent write needs an idempotency key and a server-side claim, or a lost response becomes a duplicate charge.
• Monitor retry rate, retry success rate and amplification at the dependency, since retries that rarely succeed are only adding load.

The natural next lesson is the Circuit Breaker, which decides when to stop retrying altogether. From the server's side, Rate Limiting and Load Shedding explain what produces the 429 and 503 responses clients retry on, and Backpressure and Bulkhead cover keeping pressure and failures contained. Retries are only safe for side-effecting calls with Idempotency, and the API Design, REST, and gRPC lessons cover the status codes and deadline semantics a retry policy depends on. For moving retries out of the request path, see Queues & Workers and Job Processing.