Failure & Scale Scenarios

One of N stateless application servers crashes. This is the easiest failure to handle — and the one your architecture should be designed to shrug off without anyone noticing.

What happens

A single application server — handling ride requests, driver updates, or API traffic — crashes due to an OOM kill, a segfault, a bad deploy, or a hardware fault. The process dies, and all in-flight requests on that server fail immediately. TCP connections from clients and downstream services are reset.

Detection

The load balancer runs health checks against every backend server, typically every 5–10 seconds. When a server fails two consecutive health checks, the load balancer marks it as unhealthy and removes it from the rotation. Most cloud load balancers detect a dead backend within 10–30 seconds. Meanwhile, clients whose requests were in-flight on the dead server receive connection-reset errors and retry automatically.

Impact

Requests that were mid-processing on the crashed server fail. If a rider was in the middle of requesting a ride, that request fails and the client retries. If a driver was sending a location update, that single update is lost — but the next heartbeat (sent every 3–6 seconds) will correct the state. Because the servers are stateless, there is no persistent data loss. All durable state lives in PostgreSQL, all ephemeral state lives in Redis, and all events live in Kafka. The dead server held none of it.

Mitigation and recovery

The load balancer redistributes the dead server's traffic share across the remaining N-1 servers. Each surviving server sees a small increase in load — roughly 1/(N-1) of the dead server's traffic. If you were running at 60% capacity with 10 servers, losing one server pushes you to about 67%. A replacement server (or the same server restarted) spins up, passes health checks, and is added back to the rotation. In a Kubernetes environment, this is entirely automatic — the pod restarts and the readiness probe gates traffic until the server is ready.

Why this is the easiest failure

Stateless servers are cattle, not pets. You don't repair them, you replace them. There is no recovery procedure beyond "start another one." The entire point of separating compute from state is that you can lose any compute node at any time without data loss or prolonged degradation. If losing a single application server causes an outage, your architecture has a deeper problem — either you're running too few servers (no headroom) or you've accidentally put state on the server (session affinity, local caches that aren't backed by a shared store).

Redis goes down. Driver locations vanish from the geospatial index, cache misses spike, and the matching service can't find nearby drivers. But no trip data is lost — because critical state should never exist only in Redis.

What happens

A Redis instance or cluster node becomes unavailable — process crash, network partition, memory exhaustion, or a failed failover in Redis Sentinel or Cluster mode. All commands sent to that node return errors or time out. If you're using Redis Cluster, only the slots owned by the failed node are affected; the rest of the cluster continues serving. If you're using a single Redis instance or Sentinel with a failed failover, everything is down.

Immediate impact

Driver location lookups fail. The GEOSEARCH (or older GEORADIUS) queries that the matching service uses to find nearby drivers return errors instead of results. Cache misses spike across all services that use Redis for caching — user profiles, pricing data, route estimates. Every cache miss falls through to PostgreSQL, which suddenly sees a massive increase in read traffic (a "thundering herd" or "cache stampede").

What is NOT lost

Trip state is in PostgreSQL. Every active trip — its status, rider, driver, pickup location, destination, fare estimate — is persisted in the relational database. User accounts, payment methods, ride history: all in PostgreSQL. Event history (trip-started, trip-completed, payment-processed): all in Kafka. Redis held only ephemeral, reconstructable data. The source-of-truth systems are unaffected.

What IS affected

Real-time driver positions are gone. The geospatial index that tracks where every active driver is right now existed only in Redis. Nearby-driver discovery is broken — the matching service cannot answer "which drivers are within 5 km of this rider?" Cached data (surge pricing multipliers, ETAs, user preferences) must be re-fetched from the database, causing load spikes on PostgreSQL and slower response times for riders.

Degraded mode

The matching service detects that Redis is unavailable and enters degraded mode. It returns "no drivers available" to ride requests rather than crashing. Riders see a "no drivers nearby" message or stale driver positions on the map (if the client cached the last-known positions). Existing trips that were already matched and in progress continue normally — the driver and rider are already connected, and trip state is in PostgreSQL. Only new ride matching is impaired. If a replica or surviving shard still holds recent positions, matching can keep using that slightly stale data, as the Complete Design's fault-tolerance goal intends; "no drivers available" is the fallback when the index is empty or too stale to trust.

Recovery

Redis restarts (or a replica is promoted via Sentinel/Cluster failover). The geospatial index is empty. But drivers are continuously sending location updates via their heartbeat — every 3–6 seconds, each active driver sends a GPS coordinate. Within roughly 30 seconds of Redis coming back (a few heartbeat cycles; timings here are illustrative), the geospatial index has been rebuilt organically from driver heartbeats. No backfill job is needed. Caches rebuild themselves as requests flow through: each cache miss populates the cache for subsequent requests. Within a few minutes, the system is fully recovered.

Kafka brokers go down. The critical synchronous path — ride creation, driver matching, trip updates — continues. But every asynchronous process stops: notifications, analytics, billing event processing. The question is how long your producers can buffer, and what happens when consumers resume.

What happens

One or more Kafka brokers become unavailable due to disk failure, network partition, or a botched rolling upgrade. If a broker that was the leader for certain partitions goes down, those partitions are temporarily leaderless. Producers writing to affected partitions receive errors. If the entire Kafka cluster is down, all event production and consumption stops.

What continues working

The synchronous ride creation flow does not depend on Kafka. When a rider requests a ride, the Ride Service writes to PostgreSQL, calls the Matching Service, which queries Redis for nearby drivers, selects a driver, and updates PostgreSQL. This entire path is synchronous HTTP/gRPC — Kafka is not in the loop. Trip state updates (driver accepted, rider picked up, trip completed) are also synchronous writes to PostgreSQL. The core ride experience is unaffected.

What stops

Asynchronous event processing halts. Push notifications to riders and drivers ("your driver is arriving") are delayed or lost. Analytics pipelines stop receiving events — real-time dashboards go stale. Billing events (trip-completed events that trigger payment processing) stop flowing. ETA recalculation services that consume location events stop updating. Surge pricing models that depend on supply/demand event streams stop adjusting multipliers.

Producer behavior during outage

Kafka producers have an in-memory buffer (buffer.memory, typically 64 MB). When brokers are unreachable, producers buffer events locally and retry with exponential backoff. For a short outage (seconds to a few minutes), the buffer absorbs the gap and all events are delivered once brokers return. For a prolonged outage, the buffer fills, send() calls block for up to max.block.ms (default 60s), and then throw exceptions. At this point, the producing service must decide: drop the event, write it to a dead-letter store, or fail the request.

Consumer behavior and lag

Consumers simply stop receiving data. Their poll() calls return empty or throw errors. When brokers recover, consumers resume from their last committed offset and begin processing the backlog. Consumer lag — the gap between the latest produced offset and the consumer's position — spikes. The consumers now process old events at maximum speed until they catch up. For a 10-minute outage on a topic producing 10,000 events/second, that's 6 million events in the backlog.

The idempotency requirement

After recovery, producers may have retried events that were actually persisted by the broker before it died (the ack was lost, not the write). Consumers may see duplicate events in the backlog. Every consumer must be idempotent — processing the same event twice must produce the same result. For billing, this means using the trip ID as an idempotency key: "charge for trip X" is safe to process twice because the payment system checks whether trip X was already charged.

The primary database goes down. This is the most severe failure in the system — PostgreSQL is the source of truth for every trip, every user, every payment. All writes stop. Recovery speed depends entirely on decisions you made weeks ago about replication and failover.

What happens

The PostgreSQL primary becomes unavailable — hardware failure, kernel panic, disk corruption, network partition from the application tier. All write operations fail immediately. Every INSERT, UPDATE, and DELETE returns an error. Trip state cannot be updated: a driver who just picked up a rider cannot transition the trip to "in-progress." New rides cannot be created. Payment records cannot be written.

Why this is the most severe failure

PostgreSQL is the authoritative source of truth for the entire system. Redis can be rebuilt from heartbeats. Kafka events can be replayed. Application servers are stateless. But PostgreSQL holds the actual state: which trips exist, who is on them, what the fare is, whether payment was processed. Losing the ability to write to PostgreSQL means the system cannot make forward progress on any business operation. It's not degraded — it's stuck.

Read replicas during the outage

If you have synchronous or asynchronous read replicas, read traffic can continue. Riders can still see their trip history. The app can still display driver profiles. But the replicas may have stale data — depending on replication lag, the last few seconds of writes may not have reached them. And critically, no new writes are possible. You cannot serve reads forever without writes; eventually the data becomes too stale to be useful ("your trip shows as 'in progress' but the driver completed it 5 minutes ago").

Automated failover

The standard approach is automated failover using a tool like Patroni (for self-managed PostgreSQL) or the managed failover provided by cloud databases (RDS Multi-AZ, Cloud SQL HA). Patroni uses a distributed consensus store (etcd, ZooKeeper, or Consul) to elect a new primary from the available replicas. The healthiest replica — the one with the least replication lag — is promoted to primary. The other replicas re-point to the new primary. Application connection strings update via DNS or a virtual IP.

RPO and RTO

RPO (Recovery Point Objective) answers: how much data can we afford to lose? With synchronous replication, RPO is zero — every committed transaction was confirmed by at least one replica before the primary acknowledged it. With asynchronous replication, RPO equals the replication lag at the moment of failure — typically sub-second, but under load it could be several seconds. Those "several seconds" might contain hundreds of trip state updates.

RTO (Recovery Time Objective) answers: how quickly must we recover? With automated failover (Patroni, RDS Multi-AZ), RTO is typically tens of seconds to a couple of minutes, depending on the product and its settings — the time to detect the failure, elect a new primary, and update routing. Manual failover can take minutes to hours, depending on how fast an on-call engineer responds. For a ride-hailing platform during peak hours, every second of write unavailability means failed ride requests, stuck trips, and angry users.

WAL replication trade-offs

PostgreSQL replicates via Write-Ahead Log (WAL) streaming. Synchronous replication means the primary waits for the replica to confirm it wrote the WAL entry before committing the transaction. Zero data loss, but every write pays the network round-trip to the replica — adding 1–5ms of latency. Asynchronous replication means the primary commits immediately and ships WAL in the background. Lower write latency, but if the primary dies before the replica received the latest WAL, those transactions are lost.

Most production systems use synchronous replication to one replica (guaranteeing at least one copy) and asynchronous to additional replicas (for read scaling). This gives you zero-RPO failover to the synchronous replica with acceptable write latency.

The network doesn't fail cleanly — it degrades. Service A can reach Service B, but Service B can't reach Service C. Some requests succeed, some fail, and the failure mode is maddeningly inconsistent. This is where timeouts, retries, circuit breakers, and idempotency keys earn their keep.

What happens

A partial network failure occurs between services. Maybe a switch fails in one rack, maybe a firewall rule change blocks traffic between two subnets, maybe an overloaded NAT gateway starts dropping packets. The result is not a clean "service is down" — it's "some requests from some services to some other services sometimes fail." This is the most confusing failure mode because it's inconsistent. Health checks might pass (the health endpoint is reachable) while actual business requests fail (they hit a different code path that crosses the partition).

Timeouts: the first line of defense

Without timeouts, a request to an unreachable service blocks a thread indefinitely. The calling service's thread pool fills up with blocked threads, and it stops being able to serve any requests — even ones that don't involve the broken network path. Every inter-service call must have a timeout. For a ride-hailing platform, typical timeouts are: matching service → location service: 500ms; ride service → matching service: 2s; API gateway → ride service: 5s. These timeouts should be set based on P99 latency under normal conditions plus a small buffer — not on how long you're willing to wait.

Retries: helpful until they aren't

Retries handle transient failures — a single dropped packet, a momentary GC pause on the server. But during a sustained network partition, retries amplify load. If every failed request is retried 3 times, the broken service receives 3x its normal traffic at exactly the moment it's least able to handle it. Retries must use exponential backoff with jitter: wait 100ms, then 200ms, then 400ms — plus a random offset so all callers don't retry at the same instant.

Circuit breaker pattern

After N consecutive failures (or a failure rate above a threshold within a time window), the circuit breaker "opens" and stops sending requests to the failing service entirely. Instead, it fails fast — returning an error immediately without waiting for a timeout. After a cooldown period (e.g., 30 seconds), the circuit breaker enters a "half-open" state: it sends one test request. If it succeeds, the circuit closes and normal traffic resumes. If it fails, the circuit stays open for another cooldown period.

Partial failure: the confusing middle ground

During a network partition, some requests succeed and some fail — depending on which network path they take, which server handles them, and timing. A rider in one part of the city can request rides normally; a rider three blocks away cannot. The matching service can reach Redis but not the location service. The ride service can create trips but can't send notifications. This partial failure is far harder to reason about than a total outage, and it's what actually happens in production.

Duplicate requests from retries

When the network is flaky, the server might successfully process a request but the response never reaches the client. The client, seeing a timeout, retries — and the server processes the same request again. Without idempotency keys, the rider now has two rides created for the same trip. The driver receives two dispatch notifications. The billing system charges twice. Idempotency keys are not optional in a distributed system — they are a correctness requirement.

Redis gets slow — not down, just slow. 5ms responses become 2000ms. Location Service queues up. Matching Service queues up waiting on Location. Ride Service queues up waiting on Matching. Within 60 seconds, one slow component has taken down the entire request path. This is how most large-scale outages actually happen.

The setup

The call chain: Ride Service → Matching Service → Location Service → Redis. Under normal conditions, the entire chain completes in under 100ms. Redis responds in 1–2ms, Location Service adds 5ms of processing, Matching Service adds 10ms of logic, and the Ride Service returns to the rider in well under a second. Each service has a thread pool of, say, 200 threads.

The trigger: Redis gets slow

Redis doesn't crash — that would be easy. Instead, it starts responding slowly. Maybe it's running a KEYS * command from a debugging session. Maybe it's doing an RDB save on a large dataset. Maybe the underlying VM is experiencing noisy-neighbor I/O contention. Response times go from 2ms to 2000ms. Redis is still responding, just 1000x slower. Health checks pass — the PING command still returns PONG within the health check timeout.

Location Service: the first victim

Every GEOSEARCH call now takes 2 seconds instead of 2 milliseconds. Location Service's 200 threads are all blocked waiting on Redis responses. New requests arrive and find no available threads. The request queue fills. Location Service's own response time goes from 5ms to 2000ms+ (waiting for a thread) to outright rejecting requests when the queue overflows. From the outside, Location Service looks like it's failing — but the root cause is Redis.

Matching Service: the second victim

Matching Service calls Location Service and waits. Its requests, which normally complete in 15ms, now take 2000ms+ each. Matching Service's 200 threads fill up with requests blocked on Location Service. Its own callers start timing out. The Matching Service hasn't done anything wrong — it's perfectly healthy code waiting on a perfectly healthy dependency that is waiting on a slow dependency.

Ride Service: the final domino

Ride Service calls Matching and waits. Its thread pool fills. The API gateway calls Ride Service and waits. The gateway's connection pool fills. Riders see timeouts or 503 errors. The entire ride-request path is now down. No new rides can be created. And the root cause? A single Redis instance responding slowly to geospatial queries.

Retry amplification makes it worse

At each layer, retry logic kicks in. The Ride Service retries its call to Matching. Matching retries its call to Location. Location retries its call to Redis. If each layer retries 3 times, a single user request generates 3 calls to Matching, each generating 3 calls to Location, each generating 3 calls to Redis: 27 Redis calls for one user request. Redis, which was already slow, is now receiving 27x its normal load. The retries that were supposed to help are accelerating the collapse.

Mitigation: breaking the chain

Timeouts at every boundary: Location Service should timeout its Redis calls at 50ms. If Redis can't respond in 50ms, it's not going to respond usefully. Circuit breakers: after 10 consecutive timeouts, stop calling Redis entirely for 30 seconds. Bulkheads: use separate thread pools for Redis calls and for serving requests to Matching Service. If the Redis thread pool is exhausted, the health-check and metrics threads still work. Load shedding: when the request queue exceeds a threshold, start rejecting new requests immediately with 503 rather than queueing them (they'll timeout anyway).

A major concert ends at 11 PM and 50,000 people in a single city all request rides within 5 minutes. Normal traffic for that city is 5,000 rides per hour. This is not a gradual increase — it's a step function. Autoscaling won't save you because it's 3 minutes too slow.

The scenario

A sold-out stadium event ends. Fifty thousand attendees open the app simultaneously. Ride requests spike from a baseline of ~80/minute to ~10,000/minute in a single geographic area — a 125x increase in one city while global traffic is normal. This is a localized, sudden, extreme traffic spike. It's predictable (you know the event is happening) but the magnitude is hard to provision for exactly.

What breaks first: the matching service

The matching service receives 10,000 concurrent match requests. Each match request queries Redis for nearby drivers, scores candidates, and selects the best one. Under normal load, matching takes 20ms. Under 10,000 concurrent requests, thread pools saturate, Redis connections are exhausted, and response times climb. The matching service becomes the first bottleneck because it does the most computation per request and has the most contention on shared resources (Redis connections, scoring logic).

What breaks second: Redis geospatial queries

Redis is single-threaded for command execution. Under normal load, GEOSEARCH commands complete in microseconds. Under 10,000 concurrent geospatial queries hitting the same geographic area, commands queue behind each other. Redis response times climb from 1ms to 50ms to 200ms. This isn't a Redis bug — it's the fundamental consequence of single-threaded execution under extreme concurrency. The mitigation is Redis Cluster with geographic sharding, but even then the hotspot is in one geographic area, which maps to one shard.

What breaks third: Kafka consumer lag

The event volume spikes: 10,000 ride-requested events, 10,000 notification events, 10,000 pricing events — all within minutes. Kafka handles the write side fine (it's built for throughput), but consumers can't keep up. Consumer lag increases. Notifications are delayed — riders request a ride and don't get the "searching for drivers" push notification for 30 seconds. Analytics dashboards show stale data. Surge pricing calculations lag behind the actual supply/demand ratio.

Autoscaling: necessary but insufficient

Horizontal autoscaling detects the CPU/request-rate spike and begins provisioning new application server instances. But provisioning takes time: 1–2 minutes to launch a new VM, 30 seconds to pull the container image, 15 seconds to pass health checks. Total: 2–5 minutes before new capacity is serving traffic. The spike started 30 seconds ago and you're already dropping requests. Autoscaling handles gradual growth beautifully. It doesn't handle step-function spikes.

Queue-based buffering

Instead of rejecting ride requests when the matching service is overwhelmed, queue them. A ride request enters a queue, the rider sees "finding your driver..." with an estimated wait time, and the matching service processes requests at its maximum sustainable rate. This trades latency for availability — the rider waits 60 seconds instead of 5 seconds, but they get a ride instead of an error. This is the pattern behind "surge wait times" that riders see during high-demand periods.

For how the surge pricing engine itself works, see our Architecture Deep Dive →

Rate limiting and load shedding

Rate limiting protects core services from total overload: no single user can submit more than 1 ride request per 10 seconds. Load shedding goes further: when the system detects it's beyond capacity, it deliberately rejects lower-priority requests to preserve resources for higher-priority ones. An active trip's status update is higher priority than a new ride request. A payment confirmation is higher priority than an ETA update. Shedding ETA updates to keep payments flowing is the correct trade-off.

The geographic hotspot problem

Global capacity doesn't solve a localized problem. You might have 1,000 application servers worldwide with 40% spare capacity. But the matching service instances handling this city are saturated. The Redis shard holding this geographic area is overwhelmed. Adding capacity in another region doesn't help. The fix requires geographic-aware scaling: pre-provisioning extra capacity in cities where you know large events are happening, or dynamically routing matching requests to less-loaded instances that can query the same Redis data.

An entire data center goes dark. Every server, every database, every cache instance in that region is gone. If you're single-region, you're completely down. If you're multi-region, you're about to find out whether your cross-region replication actually works.

What happens

An entire region — a data center, an availability zone, or a cloud region — becomes unavailable. Power failure, fiber cut, cooling system failure, or a cloud provider's regional outage. Every service instance, every database replica, every Redis node, every Kafka broker in that region is unreachable. This is not a gradual degradation — it's a cliff. One moment everything works, the next moment nothing in that region responds.

Single-region architecture: complete outage

If all your infrastructure is in one region, a regional failure means a complete outage for all users globally. No rides can be requested, no trips can be updated, no payments can be processed. The outage lasts until the region recovers — which could be minutes (a brief power interruption) or hours (a major infrastructure failure). There is nothing you can do except wait. This is the simplest architecture and the most fragile.

Multi-region architecture: failover

In a multi-region setup, your services run in at least two geographically separated regions. Each region has its own set of application servers, Redis instances, and Kafka clusters. The database layer is replicated across regions — a primary in one region with replicas in others, or a multi-primary setup. When one region fails, traffic is rerouted to the surviving regions.

DNS-based failover

The simplest failover mechanism: update DNS records to remove the failed region's IP addresses and point all traffic to the healthy region. DNS propagation takes time — typically 30 seconds to 5 minutes depending on TTL settings and client caching behavior. During this propagation window, some clients still try to reach the dead region and fail. Lower DNS TTLs (e.g., 30 seconds) reduce failover time but increase DNS query load during normal operation.

Global load balancer: faster failover

A global load balancer (AWS Global Accelerator, GCP's global LB, Cloudflare) performs real-time health checking of each region and routes traffic to healthy regions within seconds. No DNS propagation delay — the load balancer's anycast IP stays the same, and the routing decision happens at the network edge. Failover can happen in under 30 seconds. This is the production standard for any service that cannot tolerate minutes of downtime.

Data: the hard part

Routing traffic to another region is the easy part. The hard part is data. The surviving region needs a copy of the failed region's data — and that copy must be reasonably up-to-date. Cross-region PostgreSQL replication (streaming WAL to a replica in another region) introduces latency equal to the network round-trip between regions (typically 20–100ms for cross-region, depending on distance). If replication is asynchronous, some recently committed data may be lost. If synchronous, every write in normal operation pays that 20–100ms latency penalty.

Cross-region replication lag

Asynchronous cross-region replication typically lags by 100ms–1s under normal conditions. Under heavy write load, lag can grow to several seconds. When the primary region fails, all data written since the last replicated WAL position is lost. For a ride-hailing platform doing 10,000 writes/second with 1 second of replication lag, that's up to 10,000 transactions — trip updates, ride creations, payment records — permanently lost. This is the RPO trade-off at regional scale.

The cold start problem

When Region A fails and traffic routes to Region B, Region B must suddenly handle its own traffic plus Region A's traffic. If Region B was provisioned for its local traffic only, it's now at 2x capacity and may itself become overloaded. Autoscaling helps, but the 2–5 minute provisioning delay means Region B is overwhelmed during the critical first minutes of the failover. The mitigation is to run each region at no more than 50% capacity during normal operation — expensive, but it means either region can absorb the other's traffic without scaling. Alternatively, shed non-critical traffic during failover (disable analytics, reduce notification frequency) to buy time for autoscaling.