Collaborative Editor — Architecture Deep Dive

A WebSocket is a transport channel, not the source of ordering or durability. The lifecycle around it is what determines whether the editor feels continuous or fragile.

Connect, authenticate, authorize, subscribe

The Editor Client opens a connection to the WebSocket Gateway, which authenticates identity, authorizes access to the requested document, binds a session/client identity (not just a user ID — one user may have several tabs or devices open at once), resolves the document's current owner, and establishes the client's last-known version so it can resume rather than restart cold.

Heartbeat and liveness

Lightweight heartbeats keep the gateway and Presence Service aware of liveness. Missed heartbeats should age out sessions gradually instead of ejecting a user on one delayed packet.

Backpressure and slow consumers

A slow client must not cause unbounded gateway memory growth. The gateway keeps bounded outbound buffers, coalesces or drops ephemeral presence updates first, and resyncs a client's document state if it falls too far behind — disconnecting a persistently slow consumer if necessary. It must never silently drop a committed document edit merely to relieve queue pressure; that guarantee belongs to the durable path, not to buffer management.

Reconnect and connection migration

A reconnect should resume using the last committed/acknowledged version rather than forcing a full session restart, and it may land on a different gateway instance than the original connection — gateways stay stateless enough that this is a routine event, not a special case.

Every edit travels in an envelope with enough identity and context that the authoritative path can validate, deduplicate, and reconcile it safely — not just apply it blindly.

Operation envelope

Document ID, a stable operation ID (opId), client identity (clientId), a per-client sequence number (clientSeq), the version the client believed was current (baseVersion), an operation payload, and a schema version for the payload shape itself.

Processing pipeline

client op → validation → dedupe (opId) → reconcile against baseVersion → assign next authoritative version → durable commit → ack → broadcast. Validation and dedupe happen before reconciliation so a retried or malformed operation never reaches the transform/merge step twice.

Where ownerEpoch fits

ownerEpoch does not need to originate from the client as trusted data — it is typically session/routing context the gateway or router attaches. Its purpose is explained fully in Document Owner Lease & Fencing; here it is enough to know the envelope carries it so the durable write path can validate the writer's ownership is still current.

Error categories

Unauthorized, stale owner/session, invalid operation, resync required, throttled, and temporarily unavailable are distinct error categories the client needs to handle differently — a stale-owner error means "reroute and retry," while a resync-required error means "fetch a snapshot before submitting more edits."

At scale, every document needs a clear, discoverable owner so concurrent edits do not bounce across arbitrary servers.

Stable routing key and directory

The Shard Router deterministically maps a document to one owning Document Server (or shard group) using a stable routing key, and maintains an ownership directory so gateways and clients can resolve the current owner without guessing.

One active writer per document

For this design, exactly one owner serializes accepted operations for a given document at any moment. That gives the system one place to assign authoritative versions and keeps convergence reasoning local to a document rather than global across the fleet.

Cached routing and reroute on staleness

Gateways cache routing information to avoid a lookup on every operation, but that cache must be invalidated and rerouted when ownership changes — otherwise a gateway can keep forwarding writes to a server that no longer holds the lease. See Document Owner Lease & Fencing for how a stale route gets rejected safely rather than silently accepted.

"One owner per document" is useful but incomplete on its own — the design also has to explain how two owners are prevented from accepting writes simultaneously during failover or shard movement.

The failure this prevents

A concrete failure sequence: (1) Server A owns a document. (2) A network partition makes the coordinator believe A is dead. (3) Server B takes over ownership. (4) A is actually still alive and keeps accepting edits. (5) Two divergent timelines now exist for the same document. Ownership alone does not prevent this — the system needs a mechanism that makes stale ownership unable to write.

Lease, epoch, and fencing

An ownership lease or record carries a monotonically increasing epoch/fencing token. When a new owner takes over, it receives a strictly higher epoch. The durable write path rejects any write tagged with an epoch lower than the current one — so even if server A is still alive and still believes it owns the document, its writes are rejected the moment it tries to persist them under a stale epoch. Routers and sessions refresh ownership information so clients get rerouted to the current owner, and the old owner stops accepting writes once it observes it has been superseded.

Planned handoff vs failure-driven takeover

During planned movement (rebalancing, deploys), the outgoing owner can drain in-flight operations and hand off cleanly before the new epoch takes effect. During an unplanned failure, the system cannot wait for a clean handoff — it accepts a brief write pause instead of risking two authoritative histories (see Document Owner Failover and Availability vs Split-Brain Safety in Trade-offs & Decisions).

Transport may redeliver, and clients may retry. The application — not the WebSocket layer — is what makes operation application idempotent and correctly ordered.

Per-document authoritative version

Every accepted operation for a document advances a single authoritative version counter owned by the current owner. There is no need for a global ordering across unrelated documents — ordering only has to be authoritative within one document's timeline.

Stable opId plus per-client sequence

A stable opId lets the authoritative path deduplicate retries: if the same opId arrives twice, the second arrival is recognized and not reapplied. A per-client clientSeq complements this by helping detect duplicate operations, missing operations, and reordered client submissions — useful even when transport-level delivery is not perfectly ordered.

What this does not imply

A client-generated timestamp is not treated as establishing authoritative document order — only the version the owner assigns at commit time does that. Exactly-once delivery is not assumed anywhere in this design; idempotent application at the authoritative path is what makes at-least-once delivery safe to build on.

The word "acknowledged" must not ambiguously mean five different things. This section defines the durability boundary this design uses.

Five things "received" could mean

Received by the gateway, received by the document owner, accepted by the synchronization engine, assigned a version, or durably committed/replicated — these are different guarantees, and conflating them is how "my edit disappeared after I saw it saved" incidents happen.

The durability boundary used here

A client may receive a lightweight transport-level acknowledgement earlier, but an operation is only considered durably committed once the authoritative collaboration path has recorded it according to the configured durability policy — append to a durable operation log or replicated write path, not merely an in-memory apply. If this design exposes only one application-level ack, that ack means durable/committed, not "the server saw it."

Recommended conceptual flow

Operation reaches the current owner → validate auth/schema/base state → deduplicate opId → transform/merge → assign the next authoritative version → append to the durable log/replicated write path → update materialized in-memory state → send the committed ack → broadcast the accepted operation → update/materialize a snapshot asynchronously as appropriate. The exact ordering of materialization versus broadcast can vary by implementation, but the ack guarantee itself must stay explicit and fixed.

The trade-off this creates

Waiting for durable/replicated commit before acking costs latency; acking after a faster in-memory accept is quicker but risks the edit disappearing if the owner crashes before persistence. This trade-off is compared fully in Acknowledge Before vs After Durable Commit in Trade-offs & Decisions — this page's job is to make the chosen boundary unambiguous, not to relitigate the choice.

Operational Transform explains how an authoritative collaboration service reconciles operations created against an older revision with everything accepted since then.

The mechanics, conceptually

A client creates an operation at its known base revision. The server finds every operation accepted since that base, transforms the incoming operation against them, assigns it the next authoritative version, and broadcasts the transformed, authoritative result. Any pending local operations on other clients are then transformed/rebased against that newly arrived remote operation when it reaches them. This page does not attempt a formal proof of transform correctness — the goal is the architectural shape, not the algorithm's mathematics.

Why OT fits authoritative per-document ownership

OT fits naturally when one Document Server is authoritative for a document shard and can serialize accepted operations while transforming stale ones against newer history — one owner, one ordered log, many clients consuming transformed results.

Convergence, intention preservation, and durability are different properties

The synchronization algorithm applies deterministic transformation rules designed to preserve edits and converge — but concurrent semantic intent can still be ambiguous, and OT does not guarantee it is perfectly preserved. Convergence means replicas reach the same state. Intention preservation means the algorithm attempts to keep user edits meaningful. Durability means accepted edits survive failure. These are three separate properties; a design can satisfy convergence and durability while still occasionally producing a transformed result that reads a little oddly to the user.

An OT system needs enough transformation history to reconcile an operation created against an older revision — which means retention policy directly affects what reconnect can safely do.

The dependency

Transforming a client's stale operation against everything accepted since its base version requires that history to still exist. If the required transformation history has been compacted away, the server cannot pretend to cheaply transform an arbitrarily stale operation — it has to fall back to a different recovery path.

The fallback

When a client's base is older than retained transform history, the server sends a newer snapshot and asks the client to rebase/reconcile its local state against it, rather than attempting to replay an unbounded lifetime log. This connects directly to Snapshot Compaction & Retention (what determines how much history survives) and Offline Reconnect Protocol (what the client does when it's told to rebase).

CRDTs solve the same convergence problem with a different contract: encode edits so replicas can merge them deterministically, rather than transforming every stale operation against prior ones.

What actually has to be true

Replicas create uniquely identifiable edits/elements; concurrent operations merge deterministically; convergence depends on the rules of the specific CRDT design chosen, not on CRDTs as a category; operation/state dissemination still needs real networking and durability underneath it; and duplicate handling and causal/context metadata may still matter depending on the design. None of this is automatic just because the data structure is called a CRDT.

What CRDT does not eliminate

A CRDT-based design does not eliminate the server, does not eliminate all ordering concerns, and does not make offline synchronization automatically simple — it changes where merge complexity lives, not whether it exists. Access control, durability, and presence remain external system concerns exactly as they do for an OT-based design.

CRDT designs can accumulate identifiers, tombstones, and causal/structural metadata over time — an important counterweight to "CRDT is easier offline."

What accumulates

Unique element identifiers for every inserted unit, tombstones marking deleted content that must still be represented so concurrent operations merge correctly, causal context needed to interpret concurrent edits, and structural metadata describing how elements relate to each other.

Why compaction is non-trivial here

Garbage collection of this metadata has to be safe with respect to replicas that may still reference older state — a replica that has been offline for a long window may hold operations that depend on metadata a naive GC pass already removed. This is a genuinely harder compaction problem than the snapshot/log compaction an OT-based design faces, and it deserves explicit design attention rather than being waved away as "CRDT handles it."

A Google-Docs-style document is richer than a flat string — operations should be able to target stable logical structures, not just raw byte offsets.

Why offsets alone break down

A pure byte-offset model works for a simple text stream, but formatting marks, embedded objects, lists, and tables all shift positions in ways that make raw offsets fragile targets for concurrent operations. Addressing stable logical structures (a paragraph ID, a specific formatting mark, a specific embedded object) keeps operations meaningful even as surrounding content changes.

Effect on the synchronization strategy

The synchronization strategy — whether OT- or CRDT-based — depends on the document model it operates over; a richer structural model changes what "transform" or "merge" has to reason about. This page stops short of an editor-framework implementation tutorial; the point is architectural awareness that the document model shapes the operation protocol, not a specific rendering design.

Presence is an ephemeral distributed-state problem, not a durable document-state problem — and at large audience sizes it becomes its own fan-out challenge.

Session-scoped, not just user-scoped

Presence records key on session/client identity rather than only user ID, since one user can have multiple tabs or devices open on the same document simultaneously — collapsing them to a single user-level presence entry would make one tab's disconnect incorrectly clear presence for all of them.

TTL, sampling, and coalescing

A crashed client may never send a graceful leave event, so TTL-based expiry — removing a session once heartbeats stop — is more reliable than trusting orderly disconnects. Cursor and selection updates are rate-limited or coalesced before broadcast rather than sent on every pixel movement.

Fan-out at large audience size

For a document with a very large number of viewers, not every viewer necessarily needs every cursor — sampling, coalescing, or capping the number of simultaneously rendered remote cursors keeps presence fan-out from scaling linearly with audience size the way accepted-edit broadcast must.

A snapshot only means something if it corresponds to a precise, known authoritative version — the relationship between snapshot and log has to be exact, not approximate.

The exact relationship

A snapshot record looks like Snapshot(docId, version=1840, state=...) — it must state exactly which version its state reflects. The operation log then contains every accepted operation after that version boundary, nothing more and nothing less.

Bootstrap and recovery

Bootstrap or recovery becomes a fixed sequence: load a valid snapshot, replay every operation after its version, and reach the current authoritative revision. This only works if the snapshot's stated version is trustworthy — snapshotting arbitrary in-memory state without knowing precisely which operations it includes breaks the whole recovery model.

Three logically distinct responsibilities

The current Document Store (latest materialized body), the durable operation/history store (append-only accepted operations), and the snapshot/version store (point-in-time checkpoints) are logically distinct responsibilities even if an implementation physically combines some of them. Keeping the responsibilities distinct in the mental model is what makes recovery reasoning tractable.

After a durable snapshot at version V, older operations may become eligible for compaction — but only once every consumer that still needs them has been accounted for.

What compaction eligibility depends on

Restore/history requirements (how far back should a user be able to restore from), OT transform requirements (how stale a client's base version is allowed to be before it must rebase instead of replay), the offline-client retention window (how long a disconnected client can be gone and still replay normally), audit policy, and background compaction's own idempotency (a compaction job that partially runs and gets retried must not corrupt the log).

The rule that matters most

Do not delete old operations simply because a snapshot exists if active or stale-but-recoverable clients still need them. A client with an old base version needs either the operations back to that version or an explicit rebase instruction — compacting away the middle ground without a fallback breaks reconnect silently (see Offline Client Returns After History Compaction in Failure & Scale Scenarios).

Offline sync is a replay-and-reconciliation problem, not just a retry loop — and reconnect has to talk to the same authoritative machinery as any live edit.

The reconnect handshake

1. Client authenticates. 2. Server rechecks current document permission for this user (not just the identity, the current access level). 3. Client reports its lastAckedVersion. 4. Client reports queued operation IDs/sequence. 5. If history is retained and the gap is small, the server returns the missing operations. 6. Otherwise the server returns a current snapshot plus a rebase instruction. 7. Queued local edits are transformed/merged against current state. 8. Duplicate operations are ignored idempotently. 9. Accepted replayed operations receive authoritative versions. 10. Normal live mode resumes.

Long offline gaps

If the client is far behind, sending a newer snapshot plus missing-revision metadata is cheaper and safer than streaming a huge historical tail one operation at a time — this is exactly the OT History Retention fallback applied at the client level.

Permission change while offline

A user may lose edit permission while disconnected. The system must not blindly replay queued writes after reconnect just because the client has them queued — step 2 above (recheck current permission) exists specifically to catch this. The UI may preserve or export local unsynced work for the user, but the server enforces current authorization regardless of what the client believes it's entitled to replay.

Authorization is not only checked when a document is opened — permission can change mid-session, and the architecture has to react.

Cases that must be handled live

A user loses edit permission mid-session; a document becomes private; a user is removed from the workspace; a share link is revoked. In every case, sessions that were valid a moment ago may no longer be.

Requirements

The authoritative mutation path validates permission and current capability on every write, not only at connect time. Permission changes propagate to active sessions rather than waiting for the next reconnect. Any gateway-level authorization cache has a bounded lifetime and an invalidation path — it is a performance optimization, not a substitute for the authoritative check. A revoked client stops receiving protected updates and stops being able to write, promptly.

What this does not require

Every cursor update does not need to hit a central permission database — presence traffic can rely on the same session validity the edit path already established, refreshed on the same bounded cadence rather than rechecked per keystroke.

Comments and suggestions are not necessarily the same operation stream as raw document text — they have different data semantics worth naming explicitly.

Comments as their own entity

A comment can have a stable ID, an anchored range into the document (which itself needs to stay meaningful as the document changes underneath it), thread state, an author, and a resolution state — closer to a small structured record than to an inline text edit.

Suggestion mode as proposed, not accepted, mutations

Suggestion mode can record proposed mutations separately from the accepted document state, rather than applying them as regular operations — a suggestion only becomes part of the authoritative document when explicitly accepted, which is a different lifecycle than a normal edit's immediate acceptance.

A hot document can be hot in different ways, and each way needs a different fix — conflating them leads to solving the wrong bottleneck.

Three distinct bottlenecks

Write serialization bottleneck — many concurrent editors pushing operations through one owner. Read/fan-out bottleneck — a document with thousands of viewers, where edit QPS can be low but broadcast volume is huge (a document with 10,000 passive viewers is a fan-out problem, not a write problem). Presence bottleneck — cursor and selection chatter scaling with audience size independent of edit rate.

Mitigations, matched to the bottleneck

Hierarchical fan-out and gateway-local subscription groups address read/fan-out pressure; coalesced presence addresses presence pressure; dedicated document-owner capacity and bounded queues address write serialization; viewer/read replicas can help where reads can safely lag slightly behind the authoritative write path.

Large documents need a proactive design decision, not just a failure-mode footnote — subdocument sharding is a real option, but it is not free.

Options

Treat the whole document as one synchronization unit (simplest, but every operation and every viewer shares the same ownership and fan-out path); partition by logical block/section (spreads load, but complicates anything that spans blocks); lazy-load/render large sections client-side; and keep embedded-object state separate from the surrounding flow of text.

Why partitioning is genuinely harder than it sounds

Cross-block operations, selection ranges that span partitions, formatting that applies across a boundary, and document-wide mutations (a global find-and-replace, a style change applied everywhere) all get harder to reason about the moment ownership is split below the document level. This is exactly why Whole-Document vs Block-Level Synchronization in Trade-offs & Decisions treats block-level partitioning as something to earn through actual size/hotspot pressure rather than a default.

When an owner becomes unhealthy, the system trades a brief availability pause for the guarantee that only one authoritative timeline ever exists.

Failover sequence

1. The owner is detected as unhealthy. 2. New writes stop being routed to it / are rerouted away. 3. A new owner is established with a strictly higher epoch (see Document Owner Lease & Fencing). 4. The new owner recovers the latest durable snapshot plus the log tail after it. 5. The version sequence resumes safely from that recovered state. 6. Connected clients reconnect or reroute to the new owner. 7. Any writes that still arrive tagged with the old, stale epoch are rejected.

Why a brief pause is the safe choice

Availability may pause briefly for the affected document during steps 2–5 rather than risk accepting writes under uncertain ownership — a short write interruption is preferable to two authoritative histories that later have to be reconciled by hand. Clients can continue queueing local edits during that pause exactly as they would during any other disconnect.

A reasonable baseline keeps authoritative ordering local to one region per document while distributing connections and recovery data globally.

Baseline design

Route a document/session to a home/authoritative region; let nearby gateways terminate client connections regardless of where the client physically is; keep authoritative edit ordering localized to that home region rather than distributed; replicate durable history to other regions for recovery; and avoid synchronous global consensus per keystroke unless a specific requirement justifies paying that latency cost.

The latency this baseline accepts

Collaborators physically far from a document's home region incur real network latency on their edit round trip, since the authoritative owner is not local to them. This is a deliberate trade against the alternative — active multi-region writers — which is compared in Single-Region Owner vs Multi-Region Writers in Trade-offs & Decisions. More advanced alternatives (regional sub-ownership, deeper replication strategies) can be mentioned as directions without being prescribed as the default.

Collaboration correctness can fail silently — a document can keep returning 200s while quietly diverging or losing an edit. Observability has to be built to catch that, not just to watch uptime.

Further reading

Cited by name only, without links: C. A. Ellis and S. J. Gibbs, "Concurrency control in groupware systems", ACM SIGMOD Record 18(2), 1989 (the operational-transformation approach); D. A. Nichols, P. Curtis, M. Dixon and J. Lamping, "High-latency, low-bandwidth windowing in the Jupiter collaboration system", UIST 1995 (a client–server OT design with a central authority); M. Shapiro, N. Preguiça, C. Baquero and M. Zawirski, "Conflict-free Replicated Data Types", SSS 2011 (the CRDT formalisation).