Collaborative Editor — Failure & Scale Scenarios

A portion of the WebSocket Gateway fleet crashes or becomes unreachable.

Failure

Connected clients lose their persistent channel for edits, acks, presence, and subscription updates. The authoritative Document Server may still be healthy, but many clients are suddenly detached from it.

User impact

Users see disconnect banners, paused remote updates, and stale collaborator presence. Local typing may continue temporarily if the Editor Client falls back to its Offline Queue.

Detection

Sharp drops in active connection count, gateway heartbeat failures, reconnect spikes, and missing broadcast acknowledgements reveal the outage quickly.

Safe behavior

Any operation the client submitted but never received a committed ack for must be treated as unacknowledged, not lost — the client keeps it pending rather than assuming either success or failure. Receiving nothing back from a dropped gateway is not evidence the edit was rejected, and a gateway receipt the client did see before the drop is not evidence the edit was durably committed (see Operation Durability & Acknowledgements in Architecture Deep Dive).

Recovery

Shift clients to healthy gateways, resume sessions using the last committed/acknowledged version and pending operation IDs, and prioritize edit-path recovery before nonessential presence refreshes. Reconnect may land on a completely different gateway instance than the original connection — that is expected, not a special case.

Prevention

Keep gateways stateless enough to replace quickly, use jittered reconnects, and make reconnect resume lightweight so the fleet can absorb failover without a second outage.

Some clients or regions become partially isolated, so edits are created against stale remote views for a period of time.

Failure

Two distinct partition shapes matter here. A client-to-platform partition just isolates one user's connection — they queue edits locally like any other offline client. An internal owner/coordinator partition is far more dangerous: it can make the fleet believe a healthy document owner is dead, creating real split-brain risk if a second owner is promoted while the first is still reachable by some clients.

User impact

Users may keep typing locally but stop seeing teammates update live. When the partition heals, many operations may replay at once.

Detection

Watch for rising offline-queue depth, missing heartbeats from one network segment, version gaps, clusters of reconnects from the same ISP or region, and — for the internal case — conflicting ownership signals between coordinator and document servers.

Safe behavior

For a client-side partition, keep queueing locally — there is nothing unsafe about that. For an internal owner/coordinator partition, the durable write path must rely on fencing/epochs rather than trusting either side's belief about liveness, and may need to temporarily refuse writes for the affected document until a single fenced owner is confirmed (see Document Owner Split-Brain below).

Recovery

Let clients reconnect, fetch the newest authoritative version, and replay queued edits through the normal synchronization path so Conflict Resolution still applies.

Prevention

Design Reconnection & Resync from day one, preserve stable operation IDs, keep the client capable of local-first editing when the network is unreliable, and make sure internal ownership decisions always go through fencing rather than best-effort liveness guesses.

Many users edit the same region of a document at once — this is normal collaborative load, not a failure by itself.

Failure

A high volume of concurrent edits is not the failure. The failure is a transformation/merge bug, an invalid version sequence, replica divergence, or a lost accepted operation caused by the Synchronization Engine mishandling that concurrency.

User impact

Users see garbled sentences, cursor jumps, or silent lost edits — the most damaging failure mode for collaborative trust.

Detection

Transform errors, impossible version sequences, checksum/integrity mismatches, and user reports of disappearing text are strong signals. A high count of concurrent edits or transforms on one document is normal telemetry, not itself an alarm.

Safe behavior

If a transform/merge bug is suspected, stop applying further writes on the affected document rather than continuing to compound a corrupted state — a paused document is recoverable; a document that keeps transforming against already-wrong state gets harder to repair with every additional operation.

Recovery

Pause new writes on the damaged document if necessary, restore from the last healthy snapshot, replay safe operations, and inspect the transform or merge path before reopening full traffic.

Prevention

Keep authoritative ordering explicit, test transform logic thoroughly (deterministic simulation and property testing catch far more than manual QA), store revision history, and isolate per-document ownership so concurrency reasoning stays local.

An operation is acknowledged in some sense, but never becomes part of the durable authoritative document history.

Failure

The dangerous sequence: (1) the owner applies the operation in memory, (2) the client receives a "saved" acknowledgement, (3) the owner crashes before the durable write completes, (4) the edit disappears entirely. This is only possible if the ack was sent before durability was actually achieved — exactly the ambiguity Operation Durability & Acknowledgements is designed to eliminate.

User impact

A user believes text was saved, but it later disappears or never shows up on peer clients. This is worse than a visible error because trust is broken.

Detection

Mismatch between client-submitted opIds and durable history, ack-without-persist anomalies, and gaps in per-session operation sequences reveal the problem.

Safe behavior

The design must not let "committed" acks be sent before the durability policy's boundary is actually satisfied. If durability lags, the honest behavior is to delay the ack, not to send it early and risk exactly this scenario.

Recovery

Reconcile from the client's pending queue if possible, replay missing operations idempotently, and use the Version Store to audit the exact version boundary where loss occurred.

Prevention

Use stable operation IDs, idempotent retries, an explicit and honest durability acknowledgement boundary, and strong observability between gateway receipt, authoritative apply, and persistence.

The Presence Service becomes unavailable or starts serving stale participant state.

Failure

Join/leave events, online-user lists, cursors, and selections stop updating accurately even though the edit path may still be healthy.

User impact

Users see ghost collaborators, missing cursors, or incorrect online counts, which feels confusing but should not corrupt the document itself.

Detection

Stale heartbeat age, growing presence TTL expiries, divergence between gateway sessions and presence sessions, and support reports about ghost cursors expose the issue.

Safe behavior

Prefer hiding or clearly degrading presence UI over continuing to show presence the system already knows is stale — showing confidently wrong collaborator state is worse than showing nothing.

Recovery

Rebuild presence from active gateway sessions where possible, expire stale entries aggressively, and preserve text editing even if cursor rendering is temporarily degraded.

Prevention

Keep presence decoupled from the authoritative edit path, use heartbeat-based TTL cleanup, and allow clients to hide degraded presence gracefully instead of blocking editing.

After a regional gateway event, a huge number of clients try to reconnect, resubscribe, and replay offline edits at the same time.

Failure

Healthy gateways, the Shard Router, and Document Server shards are flooded with resume requests, history fetches, and queued-operation replays.

User impact

Recovery feels slow or flaky. Some users reconnect quickly, while others remain stuck waiting for backoff windows or replay slots.

Detection

Connection-attempt spikes, elevated auth and subscribe latency, surge in offline replay depth, and sharp load concentration on a few shards reveal the storm.

Safe behavior

Apply admission control rather than accepting every reconnect attempt at once — protect the authoritative edit path and durable storage first, and let some clients wait in a bounded queue rather than let the whole fleet degrade.

Recovery

Use exponential backoff with jitter, queue resume work, prioritize latest-version fetch before full presence hydration, batch replay rather than replaying everything at once, degrade presence first, and apply per-tenant/per-user fairness so one large workspace cannot starve everyone else's reconnect.

Prevention

Engineer lightweight resume semantics, regionally distribute gateways, protect snapshot/cache capacity with admission control, and rehearse reconnect load so incident recovery does not create a second outage.

A small number of documents become far hotter than the average, such as company announcements, classrooms, or launch review docs.

Failure

"Hot" is not one failure mode. It can mean many concurrent writers (write-serialization pressure), many passive viewers (read/fan-out pressure), or a cursor/presence flood — and each of these stresses a different part of the stack.

User impact

Users on the hot document see slower remote updates, delayed cursor movement, or brief queuing, while unrelated documents should ideally remain unaffected.

Detection

Per-document operations/sec (write pressure), subscribers per doc and broadcast lag (fan-out pressure), and presence message rate (cursor-flood pressure) let operators tell these apart rather than treating every hotspot the same.

Safe behavior

Shed nonessential presence frequency and coalesce broadcast before slowing or rejecting authoritative edit application — protect the ability to keep the document's own writers productive even if viewer freshness degrades.

Recovery

Isolate the hot document onto dedicated capacity, apply hierarchical fan-out for the read/broadcast bottleneck, coalesce presence for the cursor-flood bottleneck, and add owner capacity or bounded queues for the write-serialization bottleneck — matching the fix to the actual bottleneck identified in detection.

Prevention

Use hotspot-aware routing, per-document load metrics broken out by bottleneck type, and shard-rebalancing workflows that can move or isolate high-traffic docs without breaking ordering guarantees.

A document grows very large, making client rendering, transform cost, snapshot size, or history replay slower than normal.

Failure

The Editor Client may struggle to render or rebase efficiently, while the server may pay more for transformations, snapshot serialization/deserialization, and reconnect replay on that document — and network bootstrap size grows with it.

User impact

Typing can feel heavier, reconnect takes longer, and remote changes may appear in bursts rather than smoothly.

Detection

Document-size percentiles, transform latency by size bucket, snapshot generation/parse time, client-side frame or input lag, and bootstrap payload size reveal the issue.

Safe behavior

Prefer serving a usable, possibly partial view (lazy-loaded sections) over blocking the client entirely on a full multi-megabyte bootstrap or a full-lifetime replay.

Recovery

Serve fresh snapshots on reconnect, compact history, reduce replay depth, and if necessary encourage segmentation of extremely large documents into smaller logical units (see Large-Document Partitioning in Architecture Deep Dive).

Prevention

Use snapshot compaction, bounded operation windows, efficient data structures in the client, block-level/lazy-loading options for very large documents, and observability that treats document size as a first-class performance dimension.

The single server currently owning a document's authoritative write path crashes outright.

Failure

In-flight operations at the moment of crash split into two categories: those that were durably committed before the crash, and those that were only accepted in memory and never reached durable storage. The latter are lost by definition — the system's job is to make sure they were never falsely acknowledged as committed (see Lost Operations above).

User impact

Connected editors briefly stop receiving updates and cannot get new writes committed until a new owner takes over. Anything actually committed before the crash survives; anything only in-flight and unacknowledged should simply retry once a new owner is live.

Detection

Owner heartbeat loss, a sudden drop in acks for a specific document/shard, and health-check failures on the owning server trigger owner election.

Safe behavior

Stop routing new writes to the crashed owner immediately, and do not let a new owner start accepting writes until it has a strictly higher ownership epoch than the crashed one — this is the same fencing guarantee that prevents split-brain (see Document Owner Lease & Fencing).

Recovery

Assign a new owner with a higher epoch, restore from the latest durable snapshot plus the log tail after it, resume the version sequence safely from that recovered state, reroute connected sessions to the new owner, and rely on client-side idempotent retries (opId) to safely resubmit anything that was never acknowledged.

Prevention

Keep failover fast and automatic, keep the durable log/snapshot recovery path exercised regularly (not just in theory), and make sure clients always retry unacknowledged operations idempotently rather than assuming success or giving up.

Two servers simultaneously believe they own the same document's authoritative write path — a critical scenario, since without a fix it produces two divergent authoritative histories.

Failure

A coordinator mistakenly promotes a new owner while the previous owner is still alive and still accepting writes from clients that have not yet been rerouted (the exact sequence described in Document Owner Lease & Fencing).

User impact

If unmitigated, different clients could see different, diverging versions of the same document. With fencing in place, the user-visible impact is instead a brief rejection/resync for clients attached to the stale owner — inconvenient, but safe.

Detection

Stale-epoch write rejections at the durable layer, telemetry showing two servers both reporting ownership of the same document, and divergent version-assignment attempts for the same document ID are the signals.

Safe behavior

The durable write path validates the fencing epoch on every write and rejects anything tagged with a stale epoch — this is what actually prevents split-brain, not the ownership record alone. The old owner must stop accepting writes once it observes it has been superseded.

Recovery

The router and affected client sessions refresh their ownership information and reroute to the current, correctly-fenced owner; clients resync their version state as part of that reroute.

Prevention

Never treat "one owner per document" as self-enforcing — always back it with a monotonically increasing epoch/fencing token validated at the durable write path, and treat overlapping-ownership telemetry as a page-worthy signal, not background noise.

The owner accepts and validates an operation, but the durable log or replicated write path is temporarily unavailable when it tries to persist it.

Failure

The durable operation log (or its replicated write path) cannot accept the write — due to a storage outage, replication lag exceeding safety bounds, or a dependent service being down.

User impact

The user's edit stays visible locally (optimistic apply), but should not be reported as committed. If the outage is brief, the user may notice nothing beyond a slightly delayed "saved" indicator.

Detection

Durable-write error rates, rising time-to-commit latency, and a growing count of operations accepted-but-not-yet-durable are the leading signals.

Safe behavior

Never send a "committed" acknowledgement, and never broadcast the operation to peers as accepted, while the durability contract has not actually been satisfied. The pending operation stays queued locally on the client and at the owner rather than being silently discarded or falsely confirmed.

Recovery

Retry the durable write with bounded backoff; if the durable path stays unavailable, degrade to a temporary write-unavailable mode for the affected document rather than accepting writes the system cannot actually make durable.

Prevention

Keep the ack boundary strictly tied to actual durability (see Acknowledge Before vs After Durable Commit in Trade-offs & Decisions), and monitor the durable write path's health independently of the owner's own liveness.

The same operation arrives more than once, or operations arrive out of the order the client sent them.

Failure

Sources include client retries after a timeout, gateway-level retry logic, reconnect replay, and ordinary delayed network packets — none of which are bugs by themselves, but each can produce a duplicate or out-of-order arrival at the owner.

User impact

Without correct handling: an edit could apply twice (duplicated text), or operations could be applied in the wrong order and produce an incorrect result. With correct handling, this is invisible to the user.

Detection

Repeated opIds arriving at the durable path, clientSeq gaps or repeats, and unexpected baseVersion values relative to a client's known history are the signals.

Safe behavior

Deduplicate by opId before transformation/merge, and use clientSeq plus baseVersion to detect and correctly sequence reordered submissions rather than applying them in raw arrival order.

Recovery

A detected duplicate is dropped idempotently with the original operation's result returned; a detected reorder is resolved by applying operations according to their sequence/base-version context rather than arrival order.

Prevention

Stable opIds, a bounded dedupe window/store, and clientSeq plus baseVersion fields on every operation (see Operation Ordering & Idempotency in Architecture Deep Dive) prevent this from ever becoming user-visible.

A snapshot's declared version and the operation log's actual boundary disagree.

Failure

For example, a snapshot claims version 1000 but its serialized state only actually reflects operations through version 998, or the log's tail begins at the wrong point relative to the snapshot's stated version — breaking the bootstrap contract of "load snapshot, replay after its version."

User impact

A reconnecting or freshly-bootstrapping client could load a document missing or duplicating a small range of operations, producing subtly incorrect content that may not be immediately obvious.

Detection

Version markers that don't line up, checksum/integrity verification failures on the snapshot, and replay validation that produces a different state than the snapshot claims are the detection mechanisms.

Safe behavior

Never serve a snapshot that fails its own integrity check as if it were trustworthy — treat divergence as a hard stop for that snapshot, not a best-effort warning.

Recovery

Fall back to the previous known-good snapshot, replay the verified log forward from there, and rebuild the materialized state before serving new bootstraps from it.

Prevention

Attach integrity metadata/checksums to every snapshot, validate the snapshot-to-log version boundary as part of the write path (not just at read time), and make compaction jobs idempotent so a retried or interrupted compaction cannot itself introduce this divergence.

A buggy or malicious client sends an operation the system was never designed to accept.

Failure

Examples: an invalid text range, an oversized insert, an impossible base version, an unsupported schema version, or simply a flood of operations far exceeding normal typing rates.

User impact

Without validation, this single client could corrupt the shared document for every collaborator, or degrade service for everyone on that document/shard.

Detection

Schema/validation rejection rates spiking from one client, rate-limit triggers, and anomalous operation shapes relative to normal client behavior surface the issue quickly.

Safe behavior

Validate every operation at the trust boundary before it can influence authoritative state — range validity, size limits, schema version, and base-version plausibility — and reject invalid operations safely rather than letting any of them reach the transform/merge step.

Recovery

Reject the offending operation(s) with a clear error category, rate-limit or temporarily disconnect a client that keeps sending malformed traffic, and confirm the authoritative document state was never touched by the invalid input.

Prevention

Treat the gateway and owner's validation step as a hard trust boundary — never allow one malformed or abusive client to corrupt authoritative state for everyone else on the document, no matter how the malformed input was produced.

A user's access is revoked while their WebSocket connection is still open and they are actively editing.

Failure

Access control state changes (removed from a workspace, share link revoked, document made private) but the user's live session was authorized under the old state and has not yet been told anything changed.

User impact

Without live enforcement, a removed user could keep editing or viewing content they should no longer access. With live enforcement, the user loses write/read access promptly and mid-session, which needs to be communicated clearly rather than failing silently.

Detection

Permission-change events correlated against active sessions for the affected document, and a rising authorization-rejection rate right after a permission change, confirm enforcement is (or isn't) taking effect.

Safe behavior

Stop accepting new writes from the revoked session, stop delivering protected updates to it, and update its session state — all promptly, not only on the next reconnect. Do not silently reapply any offline/queued operations from that client once its authorization is gone.

Recovery

The session is downgraded or disconnected according to the new access level; the UI may preserve the user's unsynced local content for their own reference where product UX permits, without that content ever reaching the authoritative document.

Prevention

Validate permission on the authoritative mutation path for every write, not only at connect time, and propagate permission changes to active sessions rather than waiting for a future reconnect (see Authorization in Live Collaboration in Architecture Deep Dive).

A client reconnects after being offline long enough that its base version is older than the retained transform/operation history.

Failure

The operations needed to transform the client's stale queued edits against everything that happened since its base version have already been compacted away.

User impact

A naive implementation might fail the reconnect outright or silently drop the user's local edits. The correct behavior instead asks the client to rebase against current state, which may require some manual reconciliation for heavily conflicting local edits.

Detection

The reconnect handshake compares the client's reported last-acknowledged version against the oldest version the server's retained history actually covers; a gap beyond that boundary triggers this path explicitly rather than failing unexpectedly.

Safe behavior

Never attempt to replay deleted lifetime history that no longer exists. Recognize the gap up front and switch to the snapshot-rebase path rather than partially replaying and producing an inconsistent result.

Recovery

Send the client the current snapshot, and reconcile the client's locally queued edits against that current state using the supported rebase strategy — the client does not get an unbounded replay of everything it missed, only a consistent path forward from where the document is now.

Prevention

Set compaction/retention windows deliberately against realistic offline durations (see Long Operation Retention vs Aggressive Compaction in Trade-offs & Decisions), and make sure the reconnect handshake always checks for this case rather than assuming history is always available.

A peer's subscription misses an accepted operation — for example it never receives version 105 but does receive version 106.

Failure

The pub/sub or fan-out layer drops or delays a message to one subscriber while continuing to deliver later messages normally — a broadcast-layer gap, not a durability failure (the operation was still committed authoritatively).

User impact

Without detection, that one peer's view of the document could silently diverge, showing content that skips the missing operation.

Detection

The client compares the version number on each received operation against the version it last applied; receiving 106 immediately after 104 reveals the gap without needing any server-side signal.

Safe behavior

Pub/sub redelivery is helpful but must never be treated as the sole correctness mechanism — the client-side version-gap check is what actually guarantees correctness here, independent of whatever the broadcast layer's own delivery guarantees are.

Recovery

On detecting a gap, the client fetches the missing operations (if retained) or a current snapshot from the authoritative path, rather than waiting for or assuming eventual redelivery.

Prevention

Always include the authoritative version on broadcast messages so gaps are cheaply detectable, and never let the design assume pub/sub delivery is exactly-once or gap-free (see Pub/Sub is Distribution, Not Authority).

One connected client cannot consume remote updates as fast as they arrive — a slow device, a throttled network, or a stalled render loop.

Failure

Outbound messages queue up for this one client faster than it can process them, threatening unbounded memory growth at the gateway if nothing intervenes.

User impact

The slow client itself falls further and further behind the live document state; other collaborators are unaffected as long as the gateway isolates this per-connection backlog correctly.

Detection

Per-connection outbound queue depth crossing a threshold, and rising latency between broadcast and that specific client's acknowledgement, identify the slow consumer.

Safe behavior

Coalesce or drop ephemeral presence updates for this client first; never drop a committed document edit to relieve its queue pressure. Correctness for every other client must stay unaffected by one slow consumer's backlog.

Recovery

If the edit backlog for this client exceeds a safe bound, request a resync (snapshot-based catch-up) instead of continuing to stream a growing incremental backlog; disconnect and let it reconnect fresh if necessary.

Prevention

Bounded per-connection outbound buffers and proactive resync thresholds keep one slow client from becoming a gateway-wide memory or stability problem.

The background process responsible for snapshotting and compacting operation history stops running or falls behind.

Failure

The worker crashes, gets starved of resources, or is blocked, and snapshots stop advancing while the raw operation log keeps growing.

User impact

None immediately — live editing should continue completely normally, since snapshotting is an asynchronous supporting path, not part of the live edit path.

Detection

Growing snapshot age, growing uncompacted log length, and a widening gap between the latest committed version and the latest snapshot's version are the leading indicators — this failure is easy to miss precisely because it doesn't affect users immediately.

Safe behavior

Let editing continue uninterrupted; the risk here is a slow-building cost (reconnect and restore getting progressively more expensive), not an immediate correctness problem, so there is no need to protect live traffic by pausing anything.

Recovery

Restart or reassign the compaction worker and retry compaction idempotently from wherever it left off — an idempotent compaction job can safely resume without needing to know exactly how far a previous, interrupted run got.

Prevention

Alert explicitly on snapshot age and log-length backlog rather than relying on user-facing symptoms to surface this failure, and design compaction jobs to be safely idempotent and resumable from the start.

An entire region hosting gateways and/or authoritative document owners becomes unavailable.

Failure

Gateways in the failed region stop serving connections, and any document whose authoritative owner lived in that region loses its owner along with it.

User impact

Users connected through the failed region experience a disconnect and must reconnect, likely to a gateway in another region; documents owned in the failed region are briefly unable to accept writes until ownership is reestablished elsewhere.

Detection

Regional health-check failures, a connection-count cliff for that region, and a spike in owner-unreachable signals for documents homed there trigger the regional failover path.

Safe behavior

Promote new owners for affected documents in a recovery region only with a properly fenced higher epoch, using the durably replicated recovery state — do not promise zero data loss (zero RPO) unless the design actually pays for synchronous cross-region durability, which is a significant cost most designs do not take on by default.

Recovery

Reconnect storms are likely as clients fail over to healthy regions — apply the same admission-control and backoff practices as Scenario 06. Restore documents in the recovery region from replicated durable history, resume version sequencing there, and let clients reconnect and resync against the new owners.

Prevention

Replicate durable history across regions ahead of time, rehearse regional failover, and be explicit and honest about the RPO/RTO trade-off the design actually makes rather than implying stronger guarantees than the architecture pays for.

A deployment introduces a bug into the transformation or merge logic itself.

Failure

A code change to the Synchronization Engine — an OT transform rule or a CRDT merge rule — behaves incorrectly for some input pattern that wasn't covered by prior testing.

User impact

This is one of the most damaging classes of bug because it can look like ordinary Concurrent Edit Conflicts at first: garbled text, silently lost edits, or divergence between replicas that only shows up for specific concurrent-edit patterns.

Detection

Convergence/integrity checks (checksums or invariant assertions comparing replica state), a rise in transform/merge failure telemetry after a specific deployment, and canary-population divergence alarms are the key signals.

Safe behavior

Canary the rollout against a small population first, and treat any convergence/integrity alarm during a canary as a stop-ship signal rather than something to investigate after full rollout.

Recovery

Roll back the deployment, use revision history to identify and, where needed, repair documents affected during the bad window, and use a versioned operation schema so old and new logic can be told apart during the incident.

Prevention

Deterministic simulation and property-based testing of the transform/merge logic, a versioned operation schema, canary rollouts specifically for synchronization-engine changes, and always-on divergence/invariant monitoring catch this class of bug before or shortly after it reaches production.