Failure Scenarios & Scaling

Fanout workers crash. Posts are still created. Feed updates stop propagating.

Because fanout is queue-backed, the failure is non-destructive. The Post Service continues accepting posts and writing events to the queue. The queue retains them. The key health signal during the outage is queue lag — the age of the oldest unprocessed event — which grows the longer workers stay down.

Feed staleness increases during the outage. Users may see feeds from minutes or hours ago. Posts still exist and are discoverable via profile. Core functionality degrades gracefully — posting works, feed freshness lags.

Draining the backlog on recovery

When workers recover, retries and at-least-once delivery mean the same event may be reprocessed; idempotent feed writes (keyed on (feed_owner_id, post_id)) make this safe. The backlog should drain at a controlled rate rather than all at once, and downstream feed-store write capacity should bound how fast workers scale up — recovering by overwhelming the feed store just converts one failure into another.

CDN edge nodes become unreachable. Media requests fall through toward origin.

CDN failures are typically partial — individual PoPs or regions fail, not the entire network. Affected users experience degraded media load times. Text-based feed metadata remains available and fast, since it does not depend on the CDN.

Origin object storage sees a spike in direct requests. Origin should be sized to absorb a reasonable percentage of CDN-bypass traffic as a burst — not the full scale of normal CDN traffic, which would be prohibitively expensive to provision for. Rate limiting at the origin protects it from complete saturation at the cost of some requests being rejected during the worst of the degradation.

The pre-computed feed store becomes unavailable. The Feed Service must degrade rather than simply falling back to unrestricted read-time composition for every request.

Naively falling back to full fanout-on-read for every single request during an outage simply recreates the original database-load problem that the feed store existed to solve — at exactly the moment the system is already degraded. A better approach is bounded degradation: serve cached or slightly stale feeds where available, reduce the candidate count considered per request, simplify or skip expensive ranking, allow controlled (rate-limited) read-time composition rather than unlimited fallback, and apply load shedding to protect the database from being overwhelmed.

A circuit breaker around the feed store prevents repeated failed calls from adding latency once the failure is detected, and helps the system shift cleanly into the degraded mode described above rather than retrying the same failing path per request.

A high-follower account posts viral content. Reads, likes, and comments all surge at once.

Because the account is excluded from fanout-on-write, the fanout queue and worker fleet are largely unaffected. Hybrid fanout converts extreme write amplification into read-time work instead — but that read-time pressure lands on several places at once, not just the Feed Service: the object/post cache absorbs a surge of hydration lookups, the CDN and origin see a spike in media requests, the ranking service processes more candidate sets per second, and the likes/comments path and general API traffic all increase simultaneously.

Database read replicas absorb the post-fetch load. Sharded like counters absorb the interaction write load rather than a single relational row. Rate limiting at the API layer prevents any single surge from overwhelming the system end to end.

A major event causes uploads to spike well beyond normal processing capacity.

The processing queue's lag grows, and posts sit in PROCESSING for longer than usual. Worker autoscaling can help, but it is ultimately bounded by downstream CPU/GPU and storage constraints — transcoding is genuinely expensive work, and there is a real ceiling on how fast it can scale.

Users see their own posts remain visibly in a PROCESSING state for longer than usual rather than a hard failure. Prioritization and fairness matter here just as they do for fanout: a handful of large uploads should not be allowed to starve the processing of many smaller, ordinary ones. A backlog-age SLO (e.g., a target maximum time in PROCESSING) gives the system something concrete to recover toward, and recovery should ramp back to normal capacity gradually rather than causing a second spike.

An uploaded file is corrupt, malformed, or otherwise unprocessable.

Validation at completion time (file size/type checks, integrity checksum) catches many cases immediately. For cases that only fail during actual processing, transient issues should be retried a bounded number of times — not indefinitely. A file that keeps failing after those bounded retries is moved to a dead-letter or quarantine path and the post is marked FAILED, rather than being retried forever and silently consuming worker capacity.

Observability matters here: quarantined media should be inspectable so engineers can distinguish a genuinely corrupt file from a processing bug affecting many uploads at once.

A post goes viral before its media has been widely cached at the edge — different from a CDN outage, and arguably harder to handle.

A sudden surge of requests arrives at many edge locations simultaneously, each of which may never have served this asset before — producing simultaneous cache misses across the network rather than one origin request. Without protection, this can look like an origin request storm proportional to reader count, even though the CDN itself is healthy.

A cache hierarchy with a shielding layer between edges and origin absorbs much of this: multiple edges miss against the shield, but the shield itself only misses once. Request coalescing collapses many concurrent misses for the same object into a single in-flight origin fetch, so N simultaneous requests for a hot object produce one origin fetch rather than N. Hot-object protection and general overload controls at origin guard against any remaining spike.

A large fanout job is still in progress when a reader opens their feed.

Because fanout batches followers and checkpoints progress, some followers may have already received the new feed entry while others have not — temporary partial visibility, resolved by eventual consistency as the remaining batches complete.

Retries against a partially completed job are safe because feed writes are idempotent: reprocessing an already-completed batch produces no duplicate entries. Fairness policies ensure this catch-up work does not indefinitely delay fanout for other, unrelated posts queued behind it.

The Social Graph service becomes unavailable.

Posting and media processing can continue as long as their own storage is healthy — they do not depend on the social graph directly. Fanout, however, cannot enumerate a poster's followers and must queue the fanout event and retry rather than fail outright, resuming once the graph service recovers.

Existing, already-fanned-out feed entries in the feed store can continue serving candidates normally — that data does not require a live graph lookup to read. Pull-source discovery for high-fanout accounts, however, needs a following-lookup at read time; this can fall back to cached graph state if available, or degrade to serving only precomputed candidates while the graph service is down.

The ranking service is either fully down or responding much slower than usual — two distinct situations worth separating.

An outright outage and elevated latency call for different handling. Give ranking a strict latency budget as part of the overall feed-request timeout; if that budget is exceeded, fall back rather than let one slow dependency stall the whole request.

Reasonable fallbacks, in order of preference depending on what's available: cached ranked results from a recent request, a lightweight heuristic score computed inline, or a chronological/recency ordering as the simplest possible fallback that still returns a valid feed.

The feed store's underlying data is lost outright — not just temporarily unavailable, but gone.

Because the feed store is derived state, posts themselves remain completely safe in the durable post store — nothing about this loses actual content. Recovery means rebuilding recent candidates from source data: for each affected user, reconstruct a recent feed by querying followed accounts' recent posts, rather than treating the loss as unrecoverable.

Rebuilding for every user at once would recreate the exact database-load problem fanout was built to avoid. Prioritize active users first, serve those users via temporary, rate-limited read-time composition while their feed store entry is being rebuilt, and rate-limit the rebuild work itself so it does not overwhelm the same database it's reading from.

A post already fanned out to millions of feed stores is deleted — or the author blocks the reader, goes private, or is suspended after distribution.

This is a correctness question, not a performance failure. Correctness comes from read-time visibility validation during hydration — checking post availability and the current relationship/visibility state before the post is ever shown — not from synchronously rewriting every feed store entry that referenced it.

For media specifically, application-level unavailability can take effect immediately while CDN cache invalidation follows shortly after; either way, the removed content should not be served through the feed, even if a stale reference to it lingers in a feed store until asynchronous cleanup removes it.

A viral post receives millions of likes in a short window.

If every like synchronously increments one relational counter row, that row becomes a write hotspot — every interaction serializes against the same row, and throughput is capped by how fast a single row can be updated, regardless of how much other capacity exists elsewhere in the system.

Partitioned/sharded counters split the increment load across multiple physical counters instead of one, removing the single-row bottleneck. A fast counter layer absorbs immediate writes and gives users instant feedback, while asynchronous aggregation periodically reconciles the sharded counters into a durable total. During this window, the displayed count may be a temporary, eventually-consistent approximation rather than an exact real-time figure — an acceptable trade-off for a like count.