'%20stroke-width='24'%20stroke-linecap='round'%20stroke-linejoin='round'%3e%3cpath%20d='M196%2058%20C196%2042%20184%2032%20168%2032%20H112%20C84%2032%2062%2054%2062%2082%20C62%20108%2082%20126%20110%20126%20H148%20C176%20126%20194%20144%20194%20170%20C194%20198%20172%20224%20144%20224%20H88'/%3e%3c/g%3e%3ccircle%20cx='196'%20cy='58'%20r='14'%20fill='%236366F1'%20stroke='%23FFFFFF'%20stroke-width='6'/%3e%3ccircle%20cx='88'%20cy='224'%20r='14'%20fill='%23EC4899'%20stroke='%23FFFFFF'%20stroke-width='6'/%3e%3ccircle%20cx='62'%20cy='82'%20r='14'%20fill='%238B5CF6'%20stroke='%23FFFFFF'%20stroke-width='6'/%3e%3cpath%20d='M121%20102%20L121%20150%20L160%20126%20Z'%20fill='%23FFFFFF'%20opacity='0.98'/%3e%3c/svg%3e){kind=small}
Architecture
Reliability & Resilience
Reliability is delivering the promised outcome consistently; resilience is containing failures and recovering well. Architects often design under the assumption that dependencies will eventually fail, then choose patterns, SLOs, and recovery paths that keep the system useful.
Mangesh MandlikLast updated:
What Is Reliability & Resilience?
Reliability is how consistently a system delivers the intended result. Availability is whether the system can be used at all. Durability is whether committed data survives failures. Resilience is the ability to contain faults, degrade safely, and recover. Fault tolerance is the ability to keep operating through a specific fault without full service loss. Recoverability is how quickly and safely the system can restore normal behavior after damage occurs.
In architecture work, teams often treat failure as a design assumption rather than a surprising exception: not because every dependency is constantly broken, but because partial failure, timeouts, bad deploys, slow disks, and regional incidents are realistic enough that the recovery path must be designed up front.
Related but different reliability terms
Reliability
Does the system keep delivering the promised result over time?
Availability
Can users reach and use the system right now?
Durability
Does acknowledged data survive node, disk, or process failure?
Resilience
Can the system contain faults, degrade safely, and recover?
Fault Tolerance
Can it continue operating through a defined fault without full outage?
Recoverability
How quickly and safely can normal service be restored?
Visual Story
Dependency failure → retry storm → circuit breaker saves the system
Reliability & Resilience
Reliability & Resilience
System Healthy
0 / 0
⤢ Expand for synchronized narration timeline
Without a circuit breaker (Scenes 2–3), a slow dependency cascades into a full outage. With one (Scene 4), the system degrades gracefully and recovers automatically.
Problem It Solves
Why distributed systems need explicit failure isolation
In a distributed system, every service depends on other services, databases, queues, and third-party APIs. When one dependency slows down, it can block threads in the caller, exhaust shared pools, and take down unrelated functionality.
A failure-model-first approach makes the reasoning explicit: define what can fail, the failure mode you expect, the blast radius if it fails, how you detect it, how you recover, and what degraded behavior is still acceptable while recovery is happening.
What fails?
A database, queue, region, dependency API, deployment, or control plane.
How does it fail?
Timeout, throttling, stale replica, partial write, retry storm, bad data, or total outage.
What is the blast radius?
Which users, journeys, teams, or regions lose functionality when it breaks?
How is it detected?
SLIs, alerts, synthetic probes, queue lag, timeout rates, or customer-visible symptoms.
How is it recovered?
Retry, failover, rollback, replay, operator action, or automated repair.
What degradation is acceptable?
Read-only mode, stale data, queued work, limited features, or deliberate load shedding.
SLI, SLO, and SLA
Reliability targets drive architecture decisions
SLI vs SLO vs SLA
An SLI is the measured indicator: success rate, latency percentile, freshness, recovery time, or another observable signal. An SLO is the target for that indicator: for example, a checkout path might target a much tighter success and latency goal than a nightly analytics export because the user and business impact are different. An SLA is the external commitment, often legal or commercial, that may attach penalties if the promised level is not met.
Good SLOs are chosen from user harm and business impact, not from round numbers that feel impressive. A tighter SLO usually pushes the architecture toward more redundancy, better automation, safer deploy paths, stronger observability, and more operational investment. A looser SLO may justify a simpler design if the blast radius and recovery time are acceptable.
Resilience Patterns
Core patterns for containing failures and recovering safely
Timeouts
Every external call needs a deadline so a slow dependency cannot hold your capacity forever.
Retries + backoff + jitter
Retry transient failures carefully, spread attempts over time, and avoid synchronized retry storms.
Circuit breakers
Stop calling an unhealthy dependency for a cooldown window so your service can fail fast instead of piling on.
Bulkheads
Separate thread pools, queues, or connection pools so one bad dependency cannot starve everything else.
Load shedding
Deliberately reject low-priority work to protect the critical path when the system is overloaded.
Redundancy and failover
Replicas, standby paths, or alternative regions reduce single points of failure, but only if failover is tested.
Graceful degradation
Return a reduced or stale result when that is safer than a total outage.
Idempotency
Make retries safe for mutating operations so recovery does not create duplicate side effects.
For deeper pattern mechanics, see Retry & Backoff, Circuit Breaker, Bulkhead, Load Shedding, and Idempotency. To see how these failures unfold and how to spot them, read the Production Engineering diagnoses for Retry Storm and Cascading Failure.
How Architects Think
Resilience is about acceptable degradation and recoverability
Resilience is not the same as trying to eliminate every failure. It is designing the system so a realistic fault becomes a bounded, observable problem with a recovery path and an acceptable degraded mode.
Chaos engineering and failover drills are valuable because untested recovery is still a guess. If your SLO says checkout must keep working during a dependency outage, the architecture has to fund that promise with redundancy, isolation, automation, and operator practice.
For incident follow-through, continue to Incident & Postmortem Analysis. For deeper partition and operation-level consistency trade-offs, see Distributed Architecture.
Illustrative Scenario
A payment timeout that takes down checkout (constructed for teaching, not a real incident)
Illustrative incident: at 14:32, a payment provider's P99 latency jumped from 120ms to 8s. The checkout service had no timeout on its payment HTTP client. Within 90 seconds, checkout threads were blocked waiting for payment responses — including threads serving users who were not paying yet.
Why it cascaded: the payment and non-payment code paths shared the same thread pool. Once payment requests filled it, no new requests could be served. A non-critical dependency brought down the entire checkout flow.
What a resilient design changes: an explicit timeout bounds waiting, retries use backoff and jitter, and a circuit breaker opens after repeated failures so the service can fail fast. If payment can be retried later, queued work and idempotency keys make that retry safe. Checkout may degrade, but browsing and cart management stay available. The Incident & Postmortem page walks through the same scenario as a timeline and postmortem.
Architectural lesson: the payment provider being slow is a plausible fault model. The outage came from missing isolation and recovery behavior: no timeout, no breaker, shared thread pool, and no graceful path for degradation.
Trade-offs
What reliability patterns cost
Redundancy vs cost
Higher availability targets often cost disproportionately more, but the curve depends on workload, failure model, and operational maturity. Match redundancy to the business impact of downtime rather than pursuing extra nines by default.
Fail fast vs retry
Retrying hides transient blips but, uncontrolled, turns a slow dependency into a self-inflicted DDoS (retry storm). Failing fast with a circuit breaker protects the system but surfaces errors sooner.
Strong consistency vs availability
During a network partition you may reject work to preserve a stronger consistency guarantee or accept work and reconcile later. The right choice depends on which failure mode is more harmful for that operation.
Graceful degradation vs correctness
Serving a stale cache or a cut-down response keeps the system useful under stress, but returns data that may be wrong. Some domains (payments) forbid this; others (feeds) depend on it.
The consistency-versus-availability trade-off during partitions is subtle and can vary by operation, not just by product label. See Distributed Architecture for the precise partition-time reasoning.
What Can Go Wrong
How resilience architectures fail in practice
⏱️Missing timeouts
One slow dependency blocks all threads. No timeout means no upper bound on blast radius.
🌪️Retry storm
Clients retry together without backoff or jitter and overwhelm an already struggling service.
🧵Shared thread pool
One dependency exhausts the shared pool. Unrelated flows are starved even though they are healthy.
♻️Non-idempotent retries
Recovery works technically, but each retry can duplicate a charge, email, or order mutation.
⚡Untested failover
Standby path exists on paper but behaves differently under real traffic or stale data.
🔴No graceful degradation
Service returns 500 instead of a reduced response. Users lose unrelated features because one dependency is down.
When To Use / When Not To Use
Calibrating resilience investment to actual risk
✓ Apply resilience patterns when
- → Calling external or failure-prone dependencies
- → A component fault could cascade into critical user journeys
- → Operating against meaningful SLOs or contractual SLAs
- → Degraded service is better than total unavailability
⚠️ Balance investment when
- → The system is low-risk and manual recovery is acceptable
- → Simplicity matters more than shaving rare edge-case downtime
- → The team cannot yet operate the extra redundancy safely
- → A graceful fallback would violate correctness requirements
Architect's Challenge
A third-party payment API starts responding slowly (P99: 8 seconds). Your checkout service has no timeout configured. What is the HIGHEST PRIORITY architectural fix?
Remember· 30-second mental model
Design as if important dependencies will eventually fail, then make the failure bounded, visible, and recoverable.
- 1Reliability, availability, durability, resilience, fault tolerance, and recoverability are related but different design concerns.
- 2Choose SLIs and SLOs from user and business impact; the tighter the promise, the more redundancy and automation you usually need.
- 3Bound waiting with timeouts, then use retries with backoff and jitter only where the operation is safe to retry.
- 4Bulkheads, circuit breakers, graceful degradation, and load shedding contain blast radius instead of pretending faults will not happen.
- 5Untested failover is still a guess. Exercise recovery paths before the first real incident.
Key lesson
The dependency being slow is often the trigger, not the architectural failure. The deeper question is whether your design contained the fault and recovered on purpose.