Twitter/X Timeline: Failure Scenarios

Fanout workers go down. New posts are created but not delivered to follower inboxes.

Posts continue to be accepted and stored in the Posts DB. The fanout queue grows. Follower timelines become stale — new posts from followed users don't appear.

Queue depth alone isn't the most useful health signal here — backlog age is. A queue of a million jobs that's draining steadily is a very different situation from a queue of a hundred thousand jobs that hasn't moved in an hour. Recovery itself can become a second failure: workers coming back online and draining the backlog at full speed can overload inbox storage with a sudden burst of writes, so recovery may need a controlled drain rate rather than an all-at-once catch-up. Autoscaling more workers helps only up to the point where a downstream dependency (Redis/inbox storage capacity) becomes the real limit — see Architecture Deep Dive → Fanout Queue & Worker Architecture for the full chunking and backpressure model.

The timeline cache (Redis cluster) becomes unavailable. Every timeline read falls through to full composition.

The Timeline Composer and Ranking Service absorb significantly more load. Timeline response times increase — from milliseconds (cache hit) to hundreds of milliseconds (full composition). But timelines are still served and they're correct.

The ranking service goes down. What does the timeline look like?

Fall back to reverse-chronological order. The Timeline Composer skips the ranking step and returns candidates sorted by timestamp. The user's feed is less personalized — but it's still a valid, reasonable timeline.

Because ranking is decoupled from candidate collection, this fallback is clean. The Composer doesn't need to know the ranking algorithm — it just stops calling the ranking service and applies a simple sort instead.

A complete outage and high latency/timeouts are different failure shapes and deserve different handling. A complete outage is straightforward: stop calling the service, fall back immediately. High latency is trickier — the Composer shouldn't wait indefinitely hoping the ranking call eventually returns. Instead, it should enforce a timeout budget: if ranking hasn't responded within an acceptable window, fall back to the simpler chronological ordering rather than blocking the whole timeline request on a slow dependency.

A genuinely skewed key or shard workload — not just "a lot of traffic" in general — drives enormous, concentrated load onto one storage partition.

The first line of defense — the timeline cache — absorbs most reads. If the cache is warm (typical for active timelines), the partition sees only 5–10% of total traffic. For a viral event that causes many cache misses (new users checking a trending topic), cache miss rate spikes.

Second defense: read replicas. Each partition has N read replicas. Reads can round-robin across them, approaching N times the read throughput for read-only, lag-tolerant traffic (subject to replication lag, and a single hot key can still hit a per-node cache or CPU limit that replicas of the shard do not remove). Third: explicit load shedding or rate limiting on requests to the hot partition, combined with degraded-mode serving (serve a slightly stale cached version).

A celebrity with 50 million followers posts during a major live event. Millions of followers open the app simultaneously.

With hybrid fanout: the celebrity's post creates zero inbox writes. When followers open their timeline, the Composer fetches the post from the Posts DB alongside their pre-built inbox. The Posts DB handles a surge of reads — but it's a single-key read, highly cacheable, and served from replica.

The timeline cache absorbs most of the read surge. For followers whose timeline cache is warm, the celebrity post appears the next time the cache refreshes. For those with cold caches, it appears immediately on composition.

Fanout isn't completely down — workers are just behind. Some followers have already received a post; others haven't yet. This partial state, not a clean up/down binary, is the realistic day-to-day picture of fanout health.

This is exactly what eventual consistency predicts and accepts: temporary partial visibility of a new post across different followers is expected behavior, not a bug. Queue checkpointing tracks how far processing has gotten so a worker restart resumes rather than reprocessing everything. Retries rely on the same idempotency guarantee used elsewhere in fanout — reprocessing a follower who already received the post must be a no-op.

The Social Graph service — the source of truth for who follows whom — becomes unavailable. The effect is different depending on which part of the system needs it.

Post Creation

Unaffected. A post can still be durably written to the Posts DB without touching the Social Graph at all.

Fanout

Directly affected. Fanout needs the author's follower list to know where to push, so follower enumeration cannot proceed while the service is down. Affected fanout work queues and retries once the Social Graph recovers, rather than being lost.

Timeline Reads

Less affected than it might seem. Precomputed inboxes — already built from earlier fanout — continue serving recent candidates without needing a live Social Graph call. Pull-source discovery (fetching posts from accounts a reader follows directly) degrades gracefully depending on how much graph state is already cached; a reader's followed-account list, if cached, can still serve reads even while the live service is unavailable.

How Do You Rebuild a Lost Timeline? The Redis cluster holding timeline inboxes is lost entirely.

Because inbox state is derived rather than source of truth, posts themselves remain completely safe in the Posts DB — nothing about this incident touches durable content. What's actually lost is the precomputed candidate structure for every affected user.

A post ID remains sitting in derived inboxes and caches after the post is deleted or its visibility changes. This is a correctness failure, not merely a performance one — it's about whether the right content gets served, not how fast.

Read-time visibility validation is what actually prevents this from being served: the hydration step performs an object lookup for each candidate, and a deleted or now-unavailable post simply returns as deleted/unavailable at that point, regardless of how many stale references to it still exist upstream. Faster cache invalidation shrinks the window during which stale references might otherwise be served from a cache rather than freshly validated, but read-time validation is the actual correctness guarantee — invalidation speed only affects how quickly staleness shrinks, not whether correctness holds.

Asynchronous cleanup removes the stale reference from the inbox itself at some point later, which is purely a storage-efficiency concern — by the time cleanup runs, the reference was already harmless because read-time filtering had been preventing it from being served all along.