Circuit Breaker

A circuit breaker is a small state machine wrapped around calls to a dependency, typically a remote service in a microservice or other distributed system. It records recent outcomes, and when they suggest the dependency is unhealthy it stops forwarding calls and lets new requests fail immediately (or take a designed alternative path) instead of each waiting out a timeout. After a cooldown it admits a few trial calls to find out whether the dependency has recovered. It does not repair anything; it protects the caller's latency and resources and takes pressure off the dependency while it recovers.

The reason it exists is that a slow dependency is usually more dangerous than a dead one. A dead dependency refuses connections and returns errors in microseconds. A slow one holds your threads, connections, memory, and request deadlines for seconds at a time, and because those resources are shared with every other code path in the process, the damage spreads to features that have nothing to do with the sick dependency. The original fault was one slow service; the outage is your own exhausted pool. A breaker cuts that chain by refusing to wait once the evidence is strong.

The mental model is the electrical one, with one correction. A household breaker trips on a measured overload and stays tripped until a person resets it. A software breaker trips on measured failure and then tests itself: it periodically lets a small amount of traffic through, and closes again if that traffic goes well. It is also a caller-side mechanism. Each process keeps its own opinion based on its own recent calls; it is not a health oracle for the dependency, and two replicas of the same service may disagree about whether the circuit is open.

Take a checkout service that calls a recommendation service before rendering the confirmation page. Recommendations are optional. When that service starts timing out, checkout should not make every order wait several seconds. A breaker that opens on a meaningful pattern of failures lets checkout skip the optional call and stay available. Whether bypassing is acceptable is a business decision for each dependency: a failed payment authorization must never be quietly treated as success.

Little's law says the average number of requests in flight equals the arrival rate times the average time each spends in the system: L = λ × W. Applied to a thread pool (or a connection pool, or an async concurrency limit), it tells you how many workers a dependency call occupies: busy = calls per second × seconds per call.

Take a service with a pool of 200 worker threads that sends 1,000 calls per second to a dependency answering in 50 ms. Busy workers = 1,000 × 0.05 = 50, a quarter of the pool, which is comfortable. The ceiling is 200 ÷ 0.05 = 4,000 calls per second. Now the dependency slows to 5 seconds. The same arrival rate would need 1,000 × 5 = 5,000 workers. The pool has 200, so it fills in roughly 200 ÷ 1,000 = 0.2 seconds, and its ceiling has dropped to 200 ÷ 5 = 40 calls per second. A 100× latency increase is a 100× loss of capacity, and the loss is not confined to the slow call path: the health check, the login endpoint, and the cache warmer that share the pool all queue behind it.

Try the numbers yourself below. A timeout helps but does not by itself solve this: with a 200 ms timeout and every call hanging until the timeout, demand is still 1,000 × 0.2 = 200 workers, which exactly saturates the pool. The timeout bounds the worst case; the breaker removes the demand. With the circuit open, calls are rejected in microseconds, occupy almost no worker time, and the occupancy attributable to that dependency falls to near zero apart from the occasional probe.

The cascade then propagates upstream. When the service's pool is full, its callers see timeouts, their pools fill, and so on. Retries make the climb steeper: if each caller tries three times, a struggling dependency receives up to three times the traffic exactly when it can least afford it. This is why the breaker sits next to the retry policy rather than replacing it, a relationship covered in its own section.

Closed is the normal state: calls are forwarded and every outcome is recorded into a window. The name does not mean the dependency is healthy, only that the opening condition has not been met. The breaker evaluates a policy over the window, typically a failure rate, a slow-call rate, or both, and normally refuses to evaluate until it has seen a minimum number of calls.

Open means ordinary calls are not forwarded. They fail fast or take a fallback path you designed in advance. The breaker stays open for a configured wait interval (a fixed value in most libraries; some implementations lengthen it adaptively after repeated reopenings). While open, the dependency sees essentially no traffic from this caller, which is the breathing room it needs if the cause was overload.

Half-open admits a limited number of probe calls to test recovery. If they meet the success criteria the breaker closes and the window is reset; if they fail or are too slow, it reopens and the wait starts again. The crucial constraint is that probe concurrency is bounded. If every request queued behind the cooldown is let through at once, the recovering dependency is hit by the full backlog and the protection has been defeated.

A minimal implementation shows how small the mechanism is. The sketch below is deliberately incomplete (no thread safety, no slow-call tracking) and exists to make the transitions concrete:

class Breaker {
  state = 'CLOSED'; openedAt = 0; probes = 0; probeOk = 0;
  window = new RingBuffer(20);               // last 20 outcomes: true = failure

  async call(fn, fallback) {
    if (this.state === 'OPEN') {
      if (now() - this.openedAt < WAIT_MS) return fallback('REJECTED_OPEN');
      this.state = 'HALF_OPEN'; this.probes = 0; this.probeOk = 0;
    }
    if (this.state === 'HALF_OPEN') {
      if (this.probes >= PERMITTED_PROBES) return fallback('REJECTED_HALF_OPEN');
      this.probes++;
    }
    try {
      const result = await withTimeout(fn, TIMEOUT_MS);
      this.record(false);
      return result;
    } catch (err) {
      if (isFailure(err)) this.record(true);   // 4xx business errors are not failures
      throw err;
    }
  }

  record(failed) {
    if (this.state === 'HALF_OPEN') {
      if (failed) return this.trip();
      if (++this.probeOk === PERMITTED_PROBES) { this.state = 'CLOSED'; this.window.clear(); }
      return;
    }
    this.window.push(failed);
    if (this.window.size >= MIN_CALLS && this.window.failureRate() >= THRESHOLD) this.trip();
  }

  trip() { this.state = 'OPEN'; this.openedAt = now(); }
}

The scope of a breaker is a design choice with real consequences. The usual unit is one breaker per dependency, sometimes per operation or endpoint when operations fail independently. A single global breaker lets a slow analytics endpoint take down a healthy critical one. At the other extreme, a breaker per user or per raw URL multiplies state and hides the aggregate picture, because no individual breaker sees enough calls to decide anything. Pick the boundary that matches a real failure domain: usually one service, or one replica behind a load balancer when a single bad host is the realistic failure. Very fine granularity starves each breaker of samples, which brings back the minimum-call problem described next.

The opening rule is a design decision, not a constant. Suppose a library evaluates the most recent 20 calls, requires at least 10 before it will evaluate anything, and opens at a failure rate of 50% or more. If 12 of the last 20 calls failed, the rate is 12/20 = 60%, so the breaker opens. If only 2 calls have happened and both failed, the rate is 100%, but the minimum-call rule keeps the breaker closed because two samples say almost nothing. Without that floor, one failed request on a quiet night trips the circuit.

Windows come in two forms. A count-based window tracks the last N calls; a time-based window tracks calls in the last T seconds. They behave differently at the edges. A count-based window at low traffic can hold failures from many minutes ago, so a stale burst keeps influencing decisions; at very high traffic it covers only a fraction of a second and reacts to noise. A time-based window ages old outcomes out by the clock, but at low traffic it may contain too few calls to evaluate. The 50% figure is likewise not a universal recommendation. The right value depends on the dependency's normal error rate, the cost of one timeout, how fast your pool saturates (see the arithmetic above), and what a false trip costs the business.

Slow calls deserve their own threshold. Many dependencies degrade by getting slow long before they start erroring, and a call that returns 200 after four seconds has still consumed four seconds of a worker. Libraries such as Resilience4j can count calls slower than a configured duration toward a separate slow-call rate that can open the breaker independently of errors. The slow-call duration should sit comfortably above the normal p99 and below the per-attempt timeout.

Failure classification matters as much as the threshold. A client-caused 400 Bad Request or 422 says nothing about dependency health and should not count. A 404 may be a normal business answer. Connection refusals, timeouts, and 502, 503, 504 are the signals that matter. Whether 500 counts depends on whether your dependency uses it for transient faults or for deterministic bugs in specific requests. A breaker that counts validation errors can be tripped by a single misbehaving client, and then healthy requests from everyone else are rejected. Decide the failure predicate explicitly rather than relying on a default of "any exception". In the same spirit, the exception the breaker itself raises when it rejects a call must not be classified as a failure of the dependency, or the open state would feed itself.

Finally, be clear about what the breaker observes. If the breaker wraps the retry layer, one logical call that retries three times records a single outcome. If the retry layer wraps the breaker, every attempt passes through the breaker and is recorded individually, so a burst of retried failures fills the window three times as fast. Libraries differ, and the choice changes what a 50% threshold means. Write the choice down.

An orders service calls an inventory service with a 200 ms per-attempt timeout. Its breaker uses a count-based window of 20 calls, a minimum of 20 calls, a 50% failure threshold, a 30-second wait interval, and two permitted probes. The numbers here are illustrative. Traffic is five inventory calls per second per instance.

For ten seconds everything is healthy. Then inventory starts failing about 70% of calls, mostly 200 ms timeouts. At five calls per second the window sees roughly 3.5 failures per second, and the failure count reaches the 10 out of 20 that equals the threshold after about three seconds. Around 13 s the breaker opens. From then on every inventory-dependent request gets an immediate controlled failure or enters a business-specific alternate flow. Over the 30-second open interval about 150 calls never reach inventory; under the assumed 70% failure rate roughly 105 of them would have failed anyway, each after a 200 ms wait, and the rest would have succeeded slowly while holding workers.

At 43 s the wait elapses. The breaker admits two probes, which both succeed within the slow-call threshold, so it closes. If either had failed under a 50% half-open policy, the breaker would have reopened for another 30 seconds. The probe count and success rule are configuration, not guarantees of the pattern. A breaker also does not ramp traffic: at 44 s the full load returns at once. If the dependency has just come back from an overload, the returning traffic can knock it over again, so concurrency limits or rate controls on the dependency side may still be needed. Both are covered in the next section.

Implementations differ in how half-open works, and the differences are important. Netflix's Hystrix allowed a single trial request after its sleep window and used that one outcome to decide. Resilience4j admits a configurable number of permitted calls, waits for all of them to complete, and then evaluates the failure rate (and slow-call rate) of that small sample against the same thresholds. With two permitted probes and a 50% threshold, one failure out of two is 50% and reopens the breaker, which matches the conservative behavior described in the flow above. Other implementations require all probes to succeed. Read your library's rule rather than assuming one.

Several details decide whether recovery is graceful. First, the wait interval starts when the breaker opens, and on many libraries the transition to half-open happens lazily when the next call arrives after the wait has elapsed, not by a timer. If you want it to happen on a schedule (Resilience4j calls it automatic transition), configure it. Second, synchronized recovery is a real risk. Fifty replicas that all opened at the same moment all go half-open at the same moment, and fifty probe sets hit the dependency together. Adding jitter to the wait interval spreads that out. Third, probes should be real traffic or a cheap synthetic check that exercises the same failure mode as the real call; a health endpoint that returns 200 while the real query path is still broken closes the breaker prematurely and costs you a second round of failures.

Fourth, closing the breaker is not the same as being recovered. A dependency that failed from overload is often fragile for minutes, and a breaker that goes from zero traffic to full traffic in one step can trip again immediately. Gradual ramp-up is not a built-in property of the pattern. You get it from a concurrency limit or bulkhead on the dependency, a rate limiter on the call path, or load balancer slow-start, and a service mesh may apply it for you. See Rate Limiting and Bulkhead for those controls. And in the other direction, a breaker that reopens repeatedly has found something real: it is the signal to look at the dependency, not to tune the breaker until it stops flapping.

A timeout bounds how long one attempt can wait. A retry recovers from transient failure by trying again. A circuit breaker stops attempts altogether when recent evidence says they are likely to fail. A bulkhead caps how much of the caller's capacity one dependency can occupy, so that even before the breaker has gathered enough evidence, the damage is contained. They complement each other, but stacked without a policy they can make an outage worse.

The timeout comes first, because a breaker cannot count a call that has not finished. If the timeout is 10 seconds, the breaker needs outcomes that arrive 10 seconds apart per worker, and by the time it has 20 failures the pool is already gone. Tighten the timeout to something consistent with the dependency's real latency distribution (a few multiples of p99, not a round number picked in a meeting), and the breaker gets its evidence in time.

Retries and breakers interact in two ways. In load terms, three attempts per logical call multiplies traffic to the dependency by up to three, and that is the traffic the breaker is trying to remove. In counting terms, as discussed above, whether the breaker sees attempts or logical calls changes the effective threshold. A common arrangement is retry outside, breaker inside: each attempt asks the breaker for permission, and a breaker-rejected attempt should not be retried immediately, because the rejection was exactly the signal to stop. The retry layer should treat the "circuit open" error as non-retryable (or retry after the breaker's own interval) and give the caller the fallback. See Retry & Backoff for backoff, jitter, and retry budgets that bound the amplification, and Idempotency for why retrying a write is only safe when the operation tolerates duplicates.

Bulkheads and breakers answer different questions. The breaker says "stop calling, the evidence is bad"; the bulkhead says "never use more than N concurrent slots on this dependency, regardless of the evidence". Used together, the bulkhead keeps the pool from emptying during the seconds it takes the breaker to collect failures, and the breaker keeps the bulkhead from being full of doomed calls for the duration of the incident.

An open breaker does not require a fallback. It requires a deliberate decision about what the caller does when the dependency is unavailable, and failing fast with a clear error is a legitimate answer when nothing safe exists to degrade to. The danger is the opposite: a blanket catch that returns success-shaped data. Distinguish degraded service, which is reduced but true, from incorrect service, which is false.

Recommendations can fall back to an empty list. A shipping-estimate service can show a clearly labelled estimate, or hide the estimate and let the user check out. A cached response is reasonable when the product accepts bounded staleness: cache the last good value with a time-to-live, and label or bound it so that stale data is never presented as current. A notification can be queued and sent later, which is a deferred operation rather than a lost one, provided the queue is durable and the consumer tolerates duplicates (see Queues & Workers). Payment authorization has no honest fallback: it can fail fast with a clear message, or accept the order in a "payment pending" state if the business explicitly supports that and the later charge uses an idempotency key, but it must not report "approved".

Fraud scoring shows why this is a business call. If the scorer is down, fail-open (accept the order and flag it for manual review) preserves revenue and accepts fraud risk; fail-closed (reject or hold) preserves safety and loses sales. Neither is right everywhere, and the decision should be written down by someone who owns the risk, not discovered during an incident.

Fallbacks can fail too. A fallback that calls a second overloaded service produces a second cascade, so it deserves its own timeout and often its own breaker. For synchronous HTTP, a fast-fail rejection should usually surface as 503 Service Unavailable with a Retry-After hint when the wait interval is known, and the caller needs to know the rejection came from the breaker and not from the dependency itself. And because a fallback makes failures invisible to the user, it must also be visible to you: count degraded outcomes separately from successes, or the dashboard will report green for a feature that is not working.

Application libraries put the breaker in your code. Netflix Hystrix popularized the pattern in microservices but has been in maintenance mode since 2018; Resilience4j is the common Java choice today, with count- or time-based sliding windows, failure-rate and slow-call-rate thresholds, a configurable wait interval, and bounded half-open probes. Other ecosystems have equivalents, for example Polly in .NET and gobreaker in Go. Library breakers know the semantics of the call: they can classify exceptions, include fallbacks, and tag metrics with business context.

Service meshes and proxies implement related behavior at the infrastructure layer, but the terminology deserves care. In Envoy, and therefore in Istio, "circuit breaking" mostly means resource limits: maximum connections, pending requests, active requests, and retries toward an upstream cluster, after which excess calls are rejected immediately. The behavior closest to a classic breaker is outlier detection, which ejects individual hosts from the load-balancing pool after consecutive errors and returns them after a backoff period. These are valuable and language-agnostic, and they need no application changes, but they operate on HTTP or connection-level signals; they do not know that a 200 containing an error payload is a failure, and they cannot choose a business fallback.

State is local. Every replica keeps its own breaker, which avoids a central coordination dependency (a shared breaker store would itself need a breaker) and works acceptably in practice, because each replica observes the same sick dependency and opens at about the same time. But replicas will not open and recover at identical times, load balancers can send a replica more or less traffic, and a low-traffic replica may take far longer to collect a window than a busy one. Do not assume a single global circuit state unless you built one, and know that building one adds a failure mode of its own.

The configuration below reproduces the worked example. Property names are Resilience4j's; other libraries expose the same ideas under different names, and the values are illustrative starting points, not recommendations.

slidingWindowType: COUNT_BASED
slidingWindowSize: 20                  # evaluate the last 20 calls
minimumNumberOfCalls: 20               # no decision on smaller samples
failureRateThreshold: 50               # open at >= 50% failures
slowCallDurationThreshold: 150ms       # below the 200 ms per-attempt timeout
slowCallRateThreshold: 80              # open if >= 80% of calls are slow
waitDurationInOpenState: 30s
permittedNumberOfCallsInHalfOpenState: 2
automaticTransitionFromOpenToHalfOpenEnabled: true
recordExceptions: [java.io.IOException, java.util.concurrent.TimeoutException]
ignoreExceptions: [com.shop.InvalidRequestException]   # 4xx-style business errors

The two mistakes people make with a block like this are copying it between dependencies with different latency profiles, and setting the slow-call duration above the timeout so that it can never fire. Derive the numbers from measured p99 latency, normal error rate, and traffic volume of each dependency.

No breaker on a synchronous remote call. The classic story: a payment service slows, the order service's threads pile up waiting, the connection pool exhausts, and an application that had nothing wrong with it goes down. The postmortem names the payment service; the cause was the unbounded wait. Every synchronous call to a remote dependency whose slowness can reach shared resources deserves a timeout, and usually a breaker or bulkhead.

Threshold too sensitive. A rule such as "open after 3 failures" with no minimum sample or rate is noise at scale: three failures among thousands of calls is an ordinary background error rate, and the circuit flaps. Use a rate over a window with a minimum volume, and tune against the dependency's normal error rate.

Threshold too relaxed. The opposite error is a breaker that needs 90% failures over 200 calls, or a 10-second timeout, and opens only after the pool is already exhausted. Tune the thresholds against real p99 latency and the pool size using the occupancy arithmetic above, and use a bulkhead to cap damage in the meantime.

Undecided open-state behavior. Nobody chose what the open state returns, so it returns whatever the framework throws, often a generic 500 that triggers the caller's own alerts and retries. Pick the response in advance: fail fast with a clear error, serve cached data, use a safe default, or queue.

One breaker for everything. A slow analytics endpoint trips the breaker that also guards checkout. Partition by failure domain. Counting client errors. Validation failures inflate the rate and block healthy requests; set the failure predicate deliberately.

Half-open stampede. The wait expires and hundreds of queued requests rush through, or a fleet of replicas all probe at the same instant, which knocks the recovering dependency over again. Cap probes, jitter the wait, and control the traffic that follows.

Silent fallback. Metrics report success while a critical feature is not working. Track degraded outcomes separately. An open circuit nobody owns. The breaker opens, no alert fires, and the dependency stays unexamined for days. Alert on sustained open state for critical dependencies, and route the alert to the team that owns the dependency relationship, not on every brief transition.

Track state transitions and time spent open per breaker, the number of calls rejected while open, probe outcomes, the failure rate and slow-call rate the breaker is computing, downstream latency percentiles, and fallback usage. Emit the state as a gauge (0 closed, 1 half-open, 2 open) and the transitions as counters, so you can see flapping.

Alert on a critical dependency remaining open beyond a few wait intervals, or on a high transition rate (flapping), not on every brief open. Correlate with pool occupancy, timeout counts, and retry rates: if the breaker keeps opening while retries stay high, a retry layer may be hammering the dependency; if a breaker never opens during an incident, check the failure predicate and the minimum-call setting. A breaker that is open for long periods is either protecting you from a real outage or masking a persistent one, and only the correlation tells you which.

The pattern fits any call that crosses a process boundary and can be slow or unavailable in a way that holds your resources: third-party APIs, microservice calls, payment gateways, and shared databases or caches when their slowness can block unrelated request paths. It matters most for synchronous paths with user-facing deadlines. It adds nothing to an in-process call, which has no network and no timeout risk, and little to a call that is already isolated by a queue.

The trade-offs are mostly about tuning. An aggressive threshold protects quickly and risks false trips on ordinary variance, so users see errors that were avoidable. A conservative threshold keeps availability on noisy dependencies and may let the pool exhaust before the breaker reacts. A breaker also adds state that has to be configured, monitored, and tested; the failure mode where the breaker itself opens wrongly is real, and it is best rehearsed with fault injection in a staging environment. Finally, the breaker is not the right tool for every overload problem. If a client is exceeding an allowance, that is rate limiting; if producers outpace consumers, it is backpressure; if you are overloaded from your own demand, it is load shedding. The breaker addresses the specific case of a downstream that is failing or slow from your side of the call.

• The opening rule is a design decision: a failure-rate threshold needs a minimum-call floor so a single failure on a quiet night does not trip the circuit (for example 12 of 20 calls is 60%, so it opens at a 50% threshold).
• Set the timeout first, from the dependency's real latency distribution (a few multiples of p99), because a breaker can only count calls that have finished.
• Half-open semantics differ by library (Hystrix used one trial call, Resilience4j evaluates a configured number of permitted calls), so read your library's rule and probe with a few calls, not full traffic.
• A fallback must preserve business correctness: serve stale or degraded data where it is honest, and reject clearly where it is not (for example a payment gateway).
• Retries multiply the traffic a breaker is trying to remove, so define which layer retries and what an open breaker does to queued work and consumers.
• Breaker state is per instance in library implementations, and Envoy/Istio circuit breaking is mostly resource limits plus outlier detection. Monitor state transitions and rejected-call counts so an open breaker does not hide an incident.