Failure & Scale Scenarios

A gateway crashes and thousands of active sockets disappear at once.

Failure

One or more gateway processes terminate or become unreachable. Existing TCP or WebSocket sessions may fail via reset, timeout, or heartbeat loss rather than by a graceful close frame.

User impact

Affected users lose realtime connectivity briefly. Messages already durably accepted remain safe, but realtime delivery to those sessions pauses until reconnect or sync completes.

Detection

Gateway health checks, heartbeat gaps, sudden disconnect spikes, and route lookup failures reveal the incident. Blast radius scales with the number of connections each gateway holds.

Safe behavior

Clients reconnect with exponential backoff and jitter. The backend treats stale routes as ephemeral and relies on durable message state plus later sync rather than assuming the dead gateway delivered anything.

Recovery

Healthy gateways admit reconnecting clients, new routes replace stale ones, and reconnecting devices catch up from their sync cursor. Recovery time depends on workload and fleet size.

Prevention

Use draining for planned maintenance, keep per-gateway headroom, and design reconnect storm protection so one gateway failure does not cascade into fleet saturation.

A message-processing instance crashes mid-request.

Failure

The process can die before persistence, after a failed persistence attempt, after a successful durable write but before ACK, or after acceptance but before realtime routing.

User impact

Alice may see a timeout or ambiguous send result. Bob may get the message later even if Alice did not receive the first ACK. Durable correctness depends on where the crash happened relative to the write and ACK boundary.

Detection

Timeouts, instance health loss, elevated retry rates, and write or ACK mismatches reveal the failure mode.

Safe behavior

Alice retries with the same client_message_id. If no durable write happened, the retry creates the message. If the durable write already happened, the retry replays the original acceptance result. If routing failed, later delivery or sync still completes from durable state.

Recovery

Stateless replacement capacity resumes request handling quickly, but only idempotency resolves the ambiguity around before-vs-after durable write.

Prevention

Keep the service stateless, define the ACK boundary precisely, and test the canonical “persist succeeded, ACK lost” path explicitly.

The connection registry becomes unavailable or partially unavailable.

Failure

Realtime route lookup degrades or fails, but message durability should remain intact because routing state is not the source of truth for accepted messages.

User impact

Users may still send messages successfully, but recipients receive fewer instant pushes and instead see messages after reconnect or controlled sync fallback.

Detection

Route lookup errors, stale-entry NACKs, or sudden drop in successful live deliveries identify the incident. Rebuild time depends on heartbeat interval and fleet size.

Safe behavior

The safe posture is to preserve accepted messages durably, degrade to reconnect or controlled pull-based catch-up, and avoid triggering a massive periodic polling tax across the whole population.

Recovery

As sessions reconnect or re-heartbeat, ephemeral routes repopulate. Once the registry recovers, live delivery becomes fast again without replaying already durable messages into duplicates.

Prevention

Use leases or TTLs, replica-aware registry design, and reconnection-based rebuild paths; a global polling fallback would add its own load across the whole user base.

The durable message store loses replicas, latency spikes, or cannot satisfy the chosen write consistency.

Failure

If the configured write contract cannot be satisfied for a conversation partition, new sends for that partition may fail or pause. Already acknowledged writes remain as durable as the storage contract actually guaranteed under that failure model.

User impact

Senders may see failed or delayed sends for affected conversations while the write contract cannot be met. Messages that were already acknowledged are still readable according to the storage guarantees; recipients are mainly affected by delayed new messages and slower history loads.

Detection

Replica-health alarms, consistency-level failures, write latency spikes, and replica-set imbalance expose the issue. What RF=3 or QUORUM actually guarantees depends on the real topology and which replicas remain reachable.

Safe behavior

New messages are not acknowledged if the durable contract was not met. Fail clearly or retry safely rather than returning a misleading “sent”.

Recovery

Once enough healthy replicas or replacement capacity return, writes resume. Backlog drains and devices can continue delivery or sync from the restored durable history.

Prevention

Use bounded partitions, tested repair procedures, and explicit RPO and RTO targets that match the replication and consistency settings actually configured.

The event stream or Kafka path is unavailable.

Failure

Only the features that actually depend on Kafka degrade. For example, async group fan-out, push triggering, or analytics may stall while synchronous 1:1 durable acceptance continues if Kafka is not on that send path.

User impact

Most users can still send and receive 1:1 messages if the send path does not depend on the stream. Users may notice delayed group fan-out, late push notifications, or stale derived data such as analytics and counters until the backlog drains.

Detection

Producer failures, consumer lag explosions, unavailable partitions, or blocked outbox drains expose the incident.

Safe behavior

The safe behavior is to keep accepted message correctness independent of transient in-memory producer buffers. Durable async work should live in an outbox or equivalent recoverable state, not only in process memory.

Recovery

When Kafka recovers, consumers resume from their offsets or the outbox-backed replay source, drain backlog, and must process duplicate-safe events.

Prevention

Define exactly which workflows depend on Kafka, keep consumers idempotent, and document which features depend on Kafka — if core group delivery relies on it, treat it as a critical dependency rather than analytics-only infrastructure.

Some parts of the system can talk to each other while others cannot.

Failure

A request may succeed on one side of the split and appear to fail on the other. Gateways, sequencers, the message store, and the registry may each hold a different view of what has happened.

User impact

Users see inconsistent behavior: some sends time out, some arrive late, some devices stay connected but stop receiving. A partition can also create ownership ambiguity for sequencers or regional write authority.

Detection

Watch for asymmetric timeout patterns, split metrics by dependency path, and unexpected divergence between successful writes and successful live deliveries.

Safe behavior

Timeouts, circuit breakers, and idempotent retries limit blast radius. Safe behavior is “durable messages can be retried and synchronized after recovery subject to retention and product policy,” not an absolute eventual-delivery claim for every situation forever.

Recovery

When connectivity returns, retries, outbox replay, and reconnect sync close the gap. Conversation ordering authorities may need lease or fencing validation before accepting new writes again.

Prevention

Derive timeouts from measured latency percentiles and per-dependency budgets, and design explicitly for ambiguous success when the response path is partitioned but the write path was not.

Volume jumps suddenly because of a predictable or unpredictable event.

Failure

Message volume rises sharply, whether from a predictable event such as a holiday or an unpredictable one such as breaking news. The load is not spread evenly: it concentrates on whichever component runs out of headroom first.

User impact

Users see slower sends and delayed delivery, and some clients may fail to connect or reconnect while the saturated layer recovers. Which symptom appears depends on the layer that saturates first: gateways, sequencers, message store, fan-out workers, push providers, or network egress. Messages already acknowledged stay durable.

Detection

Measure queue depth, route latency, storage latency, per-conversation heat, consumer lag, and reconnect pressure. There is no universal first bottleneck, so measure rather than assume.

Safe behavior

Preserve accepted user messages first. Defer or drop typing, some presence updates, analytics, and maybe some receipt granularity before compromising accepted message durability.

Recovery

Autoscaling, pre-provisioned capacity, and workload prioritization restore headroom. Large events may also require staged recovery to avoid a second spike during catch-up.

Prevention

Base capacity planning on measured load tests and production metrics, and classify shedding tiers before the incident happens.

One conversation or current storage bucket becomes disproportionately active.

Failure

A single conversation, or the storage bucket currently receiving its writes, takes far more traffic than its neighbours. Because writes for one partition land on the same replica set, adding more nodes elsewhere does not relieve it.

User impact

That conversation experiences rising latency, and unrelated traffic may suffer if it shares replica sets, coordinators, or fan-out workers with the hot key.

Detection

Per-partition write rate, bucket size, replica load skew, and sequencer ownership metrics detect the hotspot. Time bucketing helps over the long term but does not automatically spread all concurrent writes in the current bucket.

Safe behavior

Rate-limit the conversation if needed and preserve ordering semantics. Do not attempt an emergency partition-key change mid-incident: it requires a compatible data migration and cannot be rolled out instantly.

Recovery

Longer-term mitigation may involve new bucketing, sharded broadcast treatment, dedicated worker lanes, or product-level rate limits.

Prevention

Plan bucket strategy ahead of time and distinguish “hot current writes” from “large historical dataset” because the mitigations differ.

An entire region loses service capacity or network reachability.

Failure

A whole region becomes unavailable or unreachable, taking its gateways, services, and local storage replicas with it. Traffic must be served by the surviving regions, which then carry extra load.

User impact

Users routed there disconnect, reconnect elsewhere, and may temporarily lose access to the freshest state if replication lag or write authority handoff is involved.

Detection

Regional health loss, widespread disconnects, replication lag dashboards, and control-plane failover alarms identify the event. Failover time depends on DNS TTLs, Anycast routing convergence, and health-check settings.

Safe behavior

Use explicit RPO and RTO thinking. Preserve single-conversation write authority, prevent dual writers, and degrade non-critical features while surviving regions absorb load.

Recovery

Recovery means rerouting users, replaying or reconciling lagging state, and restoring a safe write-authority model before resuming normal regional placement.

Prevention

Pre-provision N+1 or N+2 capacity, test regional failover, and do not lean on generic timestamp last-write-wins as a substitute for conversation ordering semantics.

Alice’s send times out even though the server already persisted M.

Failure

The server persists message M and the acknowledgement is lost on the way back, for example through a dropped connection or a timeout on the response path. The write succeeded, but the sender never learns that.

User impact

Alice sees an ambiguous state and may press retry. Bob's laptop might already have M while Alice still only sees “sending”.

Detection

Look for accepted message rows with repeated client IDs, timeout spikes on the response path, and duplicate retry attempts from the sender device.

Safe behavior

The safe behavior is one logical message: the retry replays the original message_id, sequence, and accepted status for that same client_message_id.

Recovery

Once the retry succeeds or sync catches up, Alice's client resolves to the canonical accepted message. No second logical message should appear.

Prevention

Make idempotency atomic and test this path continuously because it is the canonical messaging correctness failure.

Bob's device receives the same canonical message twice because of retry, replay, or reconnect.

Failure

A retry, a replay after reconnect, or a redelivery by a worker causes the same canonical message to be pushed to a device more than once. The duplicates carry the same message_id.

User impact

Without deduplication, Bob sees duplicate bubbles or duplicate notifications. With proper IDs, the duplicate is mostly invisible beyond extra transport work.

Detection

Device logs, duplicate delivery counts by message_id, and post-reconnect replay metrics reveal the issue.

Safe behavior

The safe behavior is at-least-once transport with exactly-once logical display: Bob deduplicates on stable message_id before rendering.

Recovery

Late duplicate deliveries become harmless no-ops. If a device ack was also duplicated, receipt handling remains idempotent.

Prevention

Assume delivery can duplicate anywhere a worker, gateway, or reconnect flow retries; design client and server processing around that reality.

Bob's laptop receives sequence 101 and then 103, but 102 is missing.

Failure

A device receives conversation sequence numbers out of contiguous order, for example 101 then 103, because a delivery was dropped, delayed, or reordered on the realtime path. The missing message exists in durable storage.

User impact

Bob may temporarily see a loading gap or a held-back message depending on client UX. Alice's send correctness is unaffected, but conversation readability is degraded until repair succeeds.

Detection

Clients can detect the gap immediately by comparing expected next sequence with received sequence. Server-side telemetry can also observe elevated gap-repair requests.

Safe behavior

Buffer within bounds, request the missing range, deduplicate late arrivals, and escalate to conversation sync if the targeted repair does not close the gap quickly.

Recovery

Once sequence 102 arrives or history sync backfills it, the client advances its contiguous pointer and continues normally.

Prevention

Keep sequence metadata explicit and make gap repair a first-class protocol, not an afterthought.

The current ordering authority for a conversation fails or loses authority mid-stream.

Failure

The process that assigns sequence numbers for a conversation crashes, stalls, or loses its lease. A replacement must take over, while the old owner may still be running and may not yet know it lost authority.

User impact

Sends to that conversation may pause briefly or retry, but the worst-case danger is duplicate ownership leading to conflicting sequence assignment.

Detection

Lease expiry, stale-owner rejection counts, and fencing-token conflicts reveal the problem.

Safe behavior

A new owner must obtain a higher epoch or fencing token before writing. Any stale owner attempts are rejected even if that process is still alive enough to try.

Recovery

Clients retry safely, and temporary gaps are repaired once the new owner resumes accepted writes.

Prevention

Use explicit ownership transfer rules; “consistent hashing routed it there” is not enough when instances fail or rebalance.

A wide outage or route flap causes huge numbers of clients to reconnect at once.

Failure

After an outage or network flap, a large share of clients detect the disconnect at about the same time and retry together. The synchronized attempts hit gateways, authentication, and the connection registry in one burst.

User impact

Users see slow reconnects, repeated connection failures, or delayed catch-up even after the original failure is gone because the recovery load itself becomes the next incident.

Detection

Connection-attempt rate, auth QPS, registry write spikes, and gateway accept-queue pressure show the storm quickly.

Safe behavior

Gate reconnects with exponential backoff and jitter, apply connection-rate limits or admission control, and stage recovery rather than allowing a synchronized wall of immediate retries.

Recovery

As capacity stabilizes, clients reconnect gradually and catch up through cursor-based sync. Some low-priority features may stay degraded until the storm subsides.

Prevention

Provision reconnect headroom, practice controlled draining, and keep auth and registry dependencies sized for bursty recovery—not just steady state.

Traced example (illustrative numbers)

Illustrative assumptions, not measurements: a route flap drops 10% of the 200M connections from the scale estimate, so 20M clients must reconnect, and the authentication and registry path can admit 200,000 new connections per second. If every client retries within about one second, the offered load is roughly 20M attempts/s, 100 times what the path can admit; most attempts fail, clients retry again, and the storm continues. Admitting 20M at 200,000/s takes at least 100 seconds even with perfect pacing. Spreading retries with exponential backoff and full jitter across a 120-second window offers about 167,000 attempts/s (20M ÷ 120), which fits under the limit; a 60-second window would offer about 333,000/s and still needs admission control to shed the excess. Each admitted client then catches up through cursor-based sync instead of a full history fetch.

A connected client stops reading or processing fast enough.

Failure

A device keeps its connection open but reads from the socket more slowly than messages are produced, for example on a poor network or a suspended app. Outbound data for that connection accumulates on the gateway.

User impact

That one user may see a disconnect or delayed messages; the broader risk is gateway memory growth if the server buffers indefinitely.

Detection

Per-connection queue depth, write timeout counts, and socket backpressure metrics detect the issue.

Safe behavior

Bound the queue, enforce write deadlines, and disconnect when necessary. Rely on durable sync after reconnect rather than infinite in-memory buffering.

Recovery

After reconnect, the device catches up from durable state. No acknowledged message is dropped just because the live socket was too slow; the client can detect the gap through its sync cursor and retry.

Prevention

Treat slow-consumer logic as part of gateway design from day one, not as an optimization to add after memory incidents.

A device returns after being offline long enough to accumulate a huge backlog.

Failure

A device stays offline long enough that many conversations have new messages, receipts, and membership changes waiting. When it reconnects, its sync cursor is far behind the current state.

User impact

The user may not see every old conversation instantly. If the sync API is naive, the device or server can be overwhelmed by an enormous single response.

Detection

Large cursor lag, unusually long sync sessions, and high per-device backlog counts identify the scenario.

Safe behavior

Use paginated sync, bounded batch size, checkpoints, and recency-based prioritization so the user sees their most relevant threads first. If incremental retention expired, switch explicitly to full reconciliation.

Recovery

The device advances its cursor in batches and eventually becomes current without exhausting memory or network budgets.

Prevention

Retention policy, cursor expiry behavior, and background prioritization rules should all be designed before the first “offline for weeks” user shows up.

A remove, leave, or join happens concurrently with message send or fan-out.

Failure

A member is added, removed, or leaves at nearly the same moment that a message is accepted or fanned out. The membership list used at send time and the one used at fan-out time can differ.

User impact

Whether a recipient sees the disputed message can vary if the design lacks explicit snapshot semantics, which creates confusing and irreproducible behavior.

Detection

Sequence-of-events traces comparing membership version at acceptance versus membership version at fan-out reveal the race.

Safe behavior

Bind the accepted message to a membership version or snapshot and fan out according to that explicit rule. Worker timing alone should not decide who receives the message.

Recovery

Once the system consistently applies the chosen snapshot semantics, replay and sync produce the same answer everywhere.

Prevention

Keep membership versioning durable and make send-time authorization and fan-out-time membership lookup speak the same version language.

A large group or channel generates a flood of delivery or read receipts.

Failure

Delivery and read receipts from many members of a large group or channel arrive in a short window. Each message can produce a receipt per recipient, so receipt volume can exceed message volume.

User impact

Users may see stale receipt state, delayed fan-out of status updates, or intentionally reduced granularity while the system protects core messaging throughput.

Detection

Receipt queue growth, per-conversation receipt cardinality, and worker lag expose the storm.

Safe behavior

Aggregate aggressively with delivery semantics by user or device, use read high-watermarks, batch asynchronously, and suppress overly expensive detail for huge audiences when the product allows it.

Recovery

Once backlog drains, visible receipt state converges. Some omitted detail may never be reconstructed if the product intentionally caps it for scale.

Prevention

Define receipt semantics by conversation type ahead of time so the system is not forced into a per-recipient explosion for every product surface.

APNs, FCM, or the push integration path is unavailable.

Failure

The mobile push provider (APNs, FCM, or the integration around them) rejects, delays, or drops notification requests. Push is a wake-up hint; the messages themselves are already stored.

User impact

Offline recipients are woken later or not at all until they manually open the app, but already connected recipients and durable message state remain correct.

Detection

Push send failures, provider-side incident feeds, and widening gap between durable acceptance and mobile wake-up events show the outage.

Safe behavior

Keep messages durable and treat push as a hint. Sender ACK meaning stays the same even when the wake-up path is unhealthy.

Recovery

When the provider recovers, newly enqueued push hints resume. Previously durable messages remain discoverable through reconnect sync regardless of push success.

Prevention

Design push payloads and retry policies so push outages affect latency, not source-of-truth correctness.

Presence or last-seen derivation is degraded or unavailable.

Failure

The service that ingests heartbeats and derives online, last-seen, or typing state degrades or becomes unreachable. These signals are ephemeral and separate from durable message state.

User impact

Users may see stale online indicators or no typing state, but sending, durable storage, and message sync should keep working.

Detection

Heartbeat ingestion loss, presence query failures, and broad staleness in displayed online indicators reveal the incident.

Safe behavior

Fail soft: keep messaging operational and let presence become stale or temporarily hidden instead of making it a dependency for core delivery.

Recovery

As heartbeat ingestion and query serving recover, presence naturally refreshes from active sessions without special replay of old typing noise.

Prevention

Architect ephemeral presence separately enough that its failure does not bleed into the durable message pipeline.

Media upload, object storage, or CDN delivery is degraded while core text messaging is still healthy.

Failure

Object storage, the upload path, or the CDN serving media bytes becomes slow or returns errors, so media bytes cannot be uploaded or fetched reliably.

User impact

Text and control messages keep working; media upload, preview, or download may fail or retry. Users might see placeholders where media content should be.

Detection

Upload error rates, object-store latency, and CDN miss or failure patterns expose the problem.

Safe behavior

Keep the media byte path decoupled so text chat remains healthy. Failed uploads should fail clearly rather than silently enqueueing unbounded bytes through gateways.

Recovery

When media storage or CDN service recovers, upload retries or later fetches complete. Existing message references remain meaningful if the object lifecycle stayed intact.

Prevention

Separate media infrastructure and treat upload and download recovery independently from message-send correctness.

Two regions or owners both believe they can assign order for the same conversation.

Failure

A network partition or a faulty failover leaves two regions or owners each believing they hold write authority for the same conversation. Both may accept sends and assign sequence numbers.

User impact

This is one of the few failures that can truly corrupt conversation order: duplicated or conflicting sequence ranges and divergent recipient histories that may be hard or lossy to reconcile afterward.

Detection

Conflicting fencing tokens, duplicate sequence ranges, or simultaneous accepted writes from multiple authorities reveal the split-brain.

Safe behavior

Freeze or reject writes until one authority is clearly fenced in. Safe behavior is reduced availability for that conversation, not silent dual acceptance.

Recovery

Recovery may require authoritative replay, conflict repair, and explicit reconciliation for the affected conversation before normal writes resume.

Prevention

Use leases, consensus, or another unambiguous write-authority mechanism wherever regional or owner failover exists.