Architecture

Distributed Architecture

The moment a system spans more than one machine over a network, new laws apply: components fail independently, there is no perfectly shared clock, and coordination becomes a first-class design problem.

Mangesh MandlikLast updated: 2 November 2026

What Is Distributed Architecture?

Distributed architecture is a design approach where system components run on multiple machines connected by a network. Unlike a single process, distributed systems fail in parts: some nodes are healthy, some are crashed, some are merely slow, and from the caller's point of view those last two cases can look identical.

Messages can be delayed by variable amounts, duplicated, dropped, and sometimes reordered. Replicas can lag behind one another. Independent processes can update the same logical record concurrently. Because there is no perfectly shared global clock, timestamps alone do not tell you which write "really happened last." That is why distribution is hard: every guarantee about ordering, freshness, durability, and conflict handling has to be designed explicitly.

Partial failure

One node, rack, process, or link can fail while the rest of the system keeps running.

Network uncertainty

Messages may be delayed, dropped, duplicated, or reordered across the network.

Replication lag

A read replica or follower may return older data than the primary for some window of time.

Coordination cost

Keeping many nodes in agreement requires extra round trips, quorums, or reconciliation logic.

Visual Story

Quorum write → replication lag → partition → consistency choice

Distributed Architecture

Single Node

0 / 0

🚨Up nextWhy Distribute?

⤢ Expand for synchronized narration timeline

During an actual network partition, Scene 3 forces a choice for the affected operation: reject or delay work to preserve one current answer, or keep serving and reconcile later. The animation illustrates that partition-time trade-off; it is not a claim that the system lives in that trade-off constantly.

Problem It Solves

Why teams accept distributed systems complexity

A single machine eventually hits ceilings on CPU, memory, storage, IOPS, and network throughput. Distribution lets architects spread load, storage, and failure domains across many machines instead of betting the whole system on one box.

But adding more nodes is not just a scaling trick. The moment state exists in more than one place, you need explicit rules for replication, failover, stale reads, duplicate delivery, and concurrent updates. Those data-ownership and freshness choices continue into Data Architecture, and stretching the same system across geographies adds the routing and region-failure problems covered in Multi-Region Architecture.

Architecture Patterns

Core distributed system models

Leader-follower replication

One leader accepts writes; followers replicate and often serve reads. It centralises write ordering, but failover, promotion rules, and replica lag become critical.

Quorum-based replication

Reads and writes require acknowledgements from a configured subset of replicas. Many systems choose overlapping read and write quorums; that is often, but not always, a simple majority.

Availability-first partition handling

Some operations keep accepting work during a partition and reconcile later. Useful where temporary divergence is tolerable and service continuity matters more than immediate agreement.

Consistency-first partition handling

Some operations reject or delay work during a partition because an invariant matters more than availability. Example: never let the same account appear to have two conflicting balances at once.

CAP is about partition-time behavior

CAP does not mean a distributed system must spend all day trading consistency against availability. The theorem matters when an actual network partition prevents some nodes from talking to others. In that moment, an operation may have to choose whether to reject work or continue with possibly divergent state. Outside that partition window, many systems can and do provide strong consistency and high availability together.

CAP's C is also a specific technical property, not a synonym for generic "correctness" or "good data." Roughly, it means clients observe one current order of writes, often discussed as linearizable behavior. A product can expose different guarantees for different APIs or operations, so it is more accurate to ask which operation prefers what guarantee under partition than to permanently label an entire database or product as simply "CP" or "AP."

What quorums actually mean

A quorum is not just "wait for N/2+1 and you are done." In practice, systems choose read and write acknowledgement rules so the quorums they care about overlap in a way that supports their consistency and failover goals. Some use majority quorums, some use custom read/write sizes, and some distinguish between durable receipt, in-memory receipt, and fully applied state. The important architectural question is what has to acknowledge a read or write before you call it successful.

Consensus and coordination

Consensus protocols such as Raft or Paxos-family designs let a group of nodes agree on a sequence of values or operations despite failures. They usually rely on a leader plus a quorum of followers and require multiple message exchanges: leader election, replication, acknowledgement, and sometimes log catch-up. Consensus is powerful when a cluster needs one ordered decision stream, but it is not a magical one-round-trip shortcut.

How Architects Think

How to keep distributed systems honest

Distribution is not a feature you bolt on for scale; it is a different set of engineering physics. The winning move is often to avoid spreading mutable state until throughput, availability, ownership, or geographic requirements truly justify it.

Start from invariants and operation semantics, not product labels. Which operations must never show conflicting state? Which can tolerate stale reads for a few seconds? Which actions must be idempotent because a caller may retry after an ambiguous timeout? Those answers tell you where to pay for coordination and where to accept reconciliation instead.

What would change your decision? If a previously "nice to have" write becomes an invariant-sensitive operation, you may move that one path from availability-first to consistency-first handling during partitions without changing every other operation in the system.

Illustrative Scenario

The lost update after a failover (illustrative scenario)

A service uses a primary database with one asynchronous replica. A user updates their shipping address; the write commits on the primary and returns success. Milliseconds later, before replication catches up, the primary's host fails and the system promotes the replica.

The replica never received that last write. The user's confirmed change has vanished: a lost update. If the old primary later rejoins and still believes it is current, its state conflicts with the promoted node's state. This is the direct consequence of choosing async replication for lower write latency.

A more durable design waits for the write to be persisted on more than one node before acknowledging success. Under a defined failure model—for example, the primary may fail after commit, at least one synchronously replicated node survives, and failover only promotes a replica that durably stored the log—that can prevent a committed write from being lost during failover. If those assumptions do not hold, the guarantee does not hold either.

Trade-offs

The unavoidable tensions of distribution

Consistency vs availability during partitions

When a real network partition isolates replicas, an operation may have to choose between rejecting or delaying requests to preserve one up-to-date answer, or continuing with potentially stale or divergent state until reconciliation. Outside partition windows, many systems can provide both strong consistency and high availability.

Latency vs durability (quorums)

Quorum systems wait for acknowledgements from a configured subset of replicas before confirming reads or writes. Larger or overlapping read/write quorums usually improve failover safety, but they add latency and reduce availability when too many replicas are slow or unreachable.

Coordination vs autonomy

Consensus-based coordination gives the cluster one ordered decision stream for leader election or critical writes, but every coordinated step costs extra messages and can stall without quorum. More autonomous designs reduce coordination cost but push duplicate, ordering, and conflict handling into the application.

Synchronous vs asynchronous replication

Synchronous or semi-synchronous replication can bound data loss for a defined failure model if commits wait for durable persistence on multiple replicas. Asynchronous replication cuts write latency, but failover may expose replication lag and lose recent acknowledged writes.

What Can Go Wrong

Common distributed system failures

📎Lost update on failover

Async replication + failover. The promoted replica was behind, so acknowledged writes disappear.

🧠Split brain

Two nodes both believe they are the leader. Clients write to both; diverged state requires reconciliation.

⏰Replication lag causing stale reads

Followers lag during a write spike. Users read from the wrong replica and do not see their own recent change.

🌩️Retry storm

A transient network problem triggers retries across thousands of clients and overloads a recovering dependency.

🔁Duplicate or reordered delivery

The same command or event is applied twice, or later messages arrive before earlier ones. Non-idempotent handlers corrupt state.

⏳Clock skew causing ordering bugs

Nodes timestamp related events with different clocks. Timestamp ordering is mistaken for causal ordering and business rules break.

When To Use / When Not To Use

✓ Distribute when

→ A single machine cannot meet the required throughput, storage, or failure-domain needs
→ You need explicit replication or failover beyond one node
→ Some workloads can tolerate stale reads or reconciliation, while others justify coordination cost
→ Geographic distribution is required for latency, resilience, or residency reasons

⚠️ Stay centralised when

→ A single machine still handles the workload comfortably with headroom
→ The team cannot yet operate failover, replication, and reconciliation safely
→ You are spreading state mainly because distribution "feels scalable" rather than because a measured limit demands it
→ A simpler architecture would satisfy the same invariants with much lower coordination cost

If the hard part is ownership, freshness, and read-model design, continue to Data Architecture. If the hard part is surviving region loss or routing users globally, continue to Multi-Region Architecture.

Architect's Challenge

During a network partition, your distributed database must choose: reject writes to stay consistent, or accept writes and reconcile conflicts later. What determines which side of CAP you choose?

Remember· 30-second mental model

Distribution introduces partial failure, variable latency, concurrency, and no perfectly shared global clock. Treat those as design inputs, not edge cases.

1CAP only bites during an actual partition, and the right choice may differ by operation.
2Quorums define which replicas must acknowledge reads or writes; they are not all the same formula.
3Async replication reduces latency but can lose recent acknowledged writes on failover if replicas lag.
4Consensus creates ordered agreement through leaders and quorums, usually with multiple message exchanges.
5Design distributed operations to survive retries, duplicates, and stale reads where the domain allows them.

Key lesson

A distributed system is trustworthy only when its failure model, invariants, and replication semantics are explicit.