Distributed Message Queue System Design: From Jobs to Durable Pub/Sub

A distributed message queue is a service that sits between the code that announces something happened and the code that reacts to it, holds each message durably on several machines, and lets the two sides run at different speeds and fail independently. The word "queue" hides three quite different products, and most design mistakes begin when they are treated as one.

Start with the vocabulary, because every page of this module defines it locally. A producer is any program that publishes a message. A broker is a server that accepts, stores and serves messages; "distributed" means several brokers cooperate and copy each message so a single machine failure does not erase it. A consumer reads messages and acts on them. A message is a small record with a payload and metadata; an event is a message that states a fact that already happened, such as OrderPlaced, while a command or job asks someone to do something, such as ship-HC-2048.

The running example is HarborCart, an invented commerce company. Every name, rate and capacity on this page is a planning input for teaching, not data from a real system. When customer order HC-2048 is placed, HarborCart wants two different things to happen. One warehouse worker should pick up the shipment job exactly once in practice. Separately, the email service, the sales-analytics pipeline and the fraud monitor should each hear about the order, each at its own pace, and each should be able to catch up after an outage.

Those two wishes are the first fork in the road. In a work queue, a message is a unit of work: several competing workers pull from one line, one of them claims each job, and once it is acknowledged the job is gone. In publish/subscribe (pub/sub), a topic receives events and each subscription gets its own independent view, so adding a new subscriber does not steal anything from the existing ones. In a retained log, the broker keeps the ordered records for a configured period and each reader keeps its own position, called an offset, so a reader can go back and re-read.

An event also differs from a synchronous call in who waits. With a remote procedure call the caller holds its connection open until the callee answers, so the caller's latency and availability include the callee's. With an event the producer hands over a fact, gets confirmation that the platform has accepted it, and moves on. That trade is not free: the caller gives up an immediate answer and takes on duplicates, delay and the question of what "accepted" really promised. The rest of this page earns those words one failure at a time.

Before choosing boxes, write down what the platform must do and, just as important, what it will refuse to promise. HarborCart's requirements split cleanly into two workloads that happen to share a word.

What must work

Shipment jobs are competing work. When an order is placed, one job such as ship-HC-2048 goes on the shipment-jobs queue and one worker should perform it. If that worker dies halfway, another should pick the job up. The business can tolerate a rare duplicate attempt only if the warehouse step is safe to repeat; it cannot tolerate a job that silently disappears.

Order events are facts with several independent audiences. The orders topic carries OrderPlaced and OrderPaid. Three consumer groups read it: email-notifier, sales-analytics and fraud-monitor. A slow analytics database must never delay the confirmation email, and a new group added next quarter must be able to read recent history without the producers changing.

Properties the platform should have

Notice the asymmetry. Every left-hand cell is something we can measure and alarm on; the right-hand cells are the claims that cause outages when a team assumes them anyway. Exactly-once effects across independent systems, global total order at scale, replay beyond retention, and surviving two broker losses with writes still accepted are all things this design cannot offer, and the later chapters show why.

Capacity numbers do not pick a product, but they tell you which problems are real. All figures below are illustrative HarborCart planning inputs, chosen so the arithmetic is easy to follow.

Assume 20,000 events per second on average, a 1 KiB average encoded payload, three consumer groups, replication factor 3 and three days of retention. A KiB is 1,024 bytes; a MiB is 1,024 KiB. Average ingest is 20,000 KiB per second, which is 19.53 MiB/s. Over a day that is about 1.77 TB decimal, or 1.61 TiB.

Retention is where the storage question lives. Three days of average traffic is 4.83 TiB of logical payload. With replication factor 3, the cluster holds each byte three times: 14.48 TiB of replicated payload. That figure leaves out indexes, segment overhead, compression, skew and operational headroom, so it is a floor for discussion rather than a disk order.

A promotion changes the picture. Suppose traffic jumps fivefold to 100,000 events/s for ten minutes. Ingest is 97.66 MiB/s, and the gross volume of those ten minutes is 57.22 GiB (60 million events). The part above what the baseline would have written anyway is 45.78 GiB. State which one you mean: gross volume is what the burst writes, incremental volume is what it adds to the day.

Consumers and partitions

Suppose, purely hypothetically, that one consumer instance handles 2,000 events/s. This number is not measured anywhere. Average load then needs 20,000 / 2,000 = 10 active consumers per group, 30 across three groups. At the peak each group would need 50. A partition is one ordered slice of a topic, and within a consumer group each partition is read by at most one consumer at a time. A topic with 24 partitions therefore supports at most 24 active consumers per group, or 48,000 events/s under that hypothesis, which is below 100,000. Peak either needs at least 50 partitions, faster consumers, or a plan to accept lag.

That last sentence is the real output of sizing: a statement of which constraint binds. Note also what the numbers do not say. They do not claim a per-partition throughput limit, because that depends on message size, hardware, replication and consumer work. They are inputs for reasoning, to be replaced by measurements.

The first HarborCart design is the obvious one. The checkout service records the order and then calls, in sequence, payment, inventory, fraud screening, email, analytics and fulfillment. Every call is a plain request and response.

This is correct for the steps that decide whether the order is valid. Payment authorization and inventory reservation really do determine the answer the shopper sees, so the caller should wait for them. The trouble is that email and analytics are in the same chain even though their results change nothing about whether HC-2048 succeeded.

Imagine a Tuesday when the email provider starts answering in eight seconds instead of eighty milliseconds. Checkout threads wait on it. The thread pool fills, new shoppers queue behind the slow ones, and soon payment calls, which are perfectly healthy, time out because the service has no threads left to make them. A reporting outage in analytics can fail an order that was already paid for. The coupling is not just latency; it converts the weakest downstream into the availability ceiling of the whole path.

The distinction worth drawing here is between request correctness and notification latency. Request correctness is what must be true before we tell the shopper yes: stock reserved, payment authorized. Notification latency is how soon other systems learn about it, and that can be seconds or minutes without anyone noticing. The design problem is to move everything in the second category off the shopper's critical path without losing any of it.

The cheapest fix is a queue inside the checkout service. A request thread drops a notification task onto an in-memory queue and returns immediately; a background thread drains the queue and calls email and analytics at its own speed.

This works on a quiet afternoon and it genuinely removes the latency coupling. It is also the right first step to try, because it shows the problem clearly before adding infrastructure.

Follow HC-2048 through it. At t0 the order commits to the order database. At t1 the request thread enqueues the notification and replies, so the shopper sees confirmation. At t2 the process is restarted by a deployment, a crash or an out-of-memory kill. Everything in that process's memory is gone, including the queued notification. The order exists, the shopper believes the email is coming, and no component knows anything is missing.

There is a second problem that appears only when the service scales out. Run three checkout instances and there are three separate queues, each with its own backlog, so one instance can be drowning while another is idle, and there is no single place to look at how many tasks are waiting. Memory is also bounded; a long downstream outage lets the queue grow until the process fails, taking pending tasks with it.

The fix for a queue that dies with its process is to move the queue into its own service that owns durable storage. Checkout becomes a producer, the broker holds the jobs, and workers on any machine pull from it.

Walk through ship-HC-2048. The producer sends the job; the broker writes it to disk and the job sits in the shipment-jobs queue as *ready*. Worker A asks for work and the broker hands over the job, but it does not delete it. It marks it as *in flight* for A and starts a timer. Worker B, asking a moment later, is not given the same job, which is what makes the workers "competing" rather than duplicating.

The timer is the key to the whole design. If A finishes, it sends an acknowledgment (ack), meaning "I processed this; you may delete it", and the job is removed. If A crashes, no ack arrives, the timer expires, and the broker makes the job ready again so B can take it. This timer is called a visibility timeout in some systems and a lease or delivery timeout in others: for its duration the job is invisible to other workers. HarborCart illustratively uses 60 seconds.

Now the awkward cases. If A is merely slow, not dead, and its lease expires at second 60 while it is at second 59 of a ship operation, the broker gives the job to B and both workers now try to ship the order. The broker cannot tell a dead worker from a slow one, so the guarantee it can offer is at-least-once: a job will be delivered one or more times. A long job can renew its lease before it expires, which reduces false expiries but does not eliminate them. A lost ack has the same effect from the other side: A finished, the ack never arrived, and the job is delivered again.

There is also a mirror-image mistake that does not need a crash to cause trouble. Suppose the worker acknowledges the job as soon as it receives it, and only then starts the warehouse transaction. If that transaction fails or the worker dies a moment later, the broker has already deleted the job, so it vanishes from normal delivery and nothing will ever retry it. The acknowledgment must come after the business effect commits. When the worker knows it cannot finish, for example because the warehouse API is rejecting calls, it can send a negative acknowledgment (nack), which asks the broker to make the job ready again straight away, usually with a retry delay, instead of waiting out the 60-second lease. A nack is a request for another attempt, not a statement that the job is bad: how many attempts are allowed is the subject of Chapter 14.

Sizing the worker pool is a queueing problem rather than a guess. HarborCart expects 5,000 shipment tasks per minute, which is 83.33 tasks per second. If each task holds a worker for 0.3 seconds, Little's Law says the average number in service is arrival rate times service time: 83.33 x 0.3, about 25 concurrent workers in steady state, before headroom for bursts and retries.

When a producer is told "accepted", what exactly has been promised? The honest answer is a ladder of states, and a good platform names which rung the acknowledgment refers to.

Consider the stages for evt-HC2048-01. The producer issues the request. The broker receives bytes on a socket, which proves nothing about durability, because a process can read a socket and crash before touching disk. The leader appends the record to its log. Followers copy it. Only when the configured set of replicas has it does the broker send the acknowledgment. Later, a consumer is delivered the record, a business effect commits, and the consumer acknowledges or commits an offset. These are different events with different failure modes and it is a mistake to merge any two of them.

The producer's hardest moment is the unknown outcome. It sent the record, the connection timed out, and no ack came. The record might not have arrived, or it might be committed with only the acknowledgment lost. The producer must retry, since giving up would lose events, but retrying may write a second copy. Platforms address this with a stable producer identity and sequence numbers so that retried sends inside one producer session are recognized, yet that scope is narrow: a restarted producer or an application that resends under a new ID is not covered. The end-to-end answer is a stable event ID the consumer can use to discard repeats, which Chapter 12 builds.

The contract HarborCart writes down is simple to state. The orders topic uses replication factor 3 and a minimum in-sync replicas setting of 2, and producers wait until every replica currently in sync with the leader holds the record (the in-sync replica set, defined properly in Chapter 16) before treating a publish as successful. If fewer than two replicas are in sync, the broker refuses the write instead of acknowledging something it could lose to a single failure. Chapter 16 explains precisely what that does and does not survive.

Now the second workload. Email, analytics and fraud all want OrderPlaced. Try to serve them with the shipment queue and the design breaks in a specific way.

With one shared queue and three consumers, each message goes to exactly one of them. Email might take HC-2048 and analytics never sees it. The workaround is one queue per audience: the producer sends three copies, or a small router duplicates the message into email-queue, analytics-queue and fraud-queue. That does work, and many systems run happily this way. Its cost appears as the number of audiences grows, because the producer or router must know every subscriber, adding a new one means changing the sender, and a failure in the middle of the three sends leaves some queues updated and others not.

Pub/sub moves that duplication into the platform. The producer publishes once to the orders topic. Each subscription is an independent cursor over the topic, with its own progress and its own backlog. The email subscription can be minutes behind while fraud is current, and neither affects the other. A new team subscribes without the producer knowing.

Be careful with the common phrasing that every subscriber "receives a copy". More accurately, each subscription independently reads the single stored stream. And within a subscription that has several workers, the workers compete again: a subscription is itself a work queue for its team. That is why the three words queue, topic and subscription are used together and not interchangeably.

If subscriptions are just independent cursors, the broker does not need to delete a message when one reader finishes. It can keep the stream and let each reader remember where it is. That is the retained log.

A log here means an append-only sequence: new records go on the end and existing records are never modified. Each record gets a sequential offset within its partition: HC-2048's OrderPlaced is at offset 4,201 of orders partition 7, and the next event for the same order, OrderPaid, is at 4,202. A consumer's progress is just a number saying "I have processed up to here"; that stored number is the committed offset. Reading is a lookup by offset, not a destructive pop, so a reader can seek backwards and re-read, which is replay.

On disk the partition is stored as a series of segment files, each covering a range of offsets, with an index to find an offset quickly. Appends go to the active segment; when it reaches a size or age limit a new one starts. Retention then works by deleting whole old segments. HarborCart keeps orders for three days with the delete policy, so a consumer that falls four days behind finds its offset no longer exists. Retention is a bound, not a promise to keep everything; the log does not store every message forever.

Replay is the capability that changes how teams work. A new fraud model can be trained by reading three days of orders from the start. A bug in analytics that wrote wrong values can be fixed and the group rewound to reprocess. A compacted topic is a different mode where the log keeps at least the latest record per key and removes older ones, which suits a changelog of current state but is not a full history, and a late-starting reader can miss intermediate values.

One log on one machine has a throughput ceiling and a single point of failure. Splitting the topic into partitions gives parallelism: each partition is its own independent log, possibly led by a different broker.

The producer chooses which partition gets each record, usually by hashing a key. HarborCart keys orders by order:<orderId>, so every event for HC-2048 hashes to partition 7 and sits in append order there: OrderPlaced before OrderPaid. Ordering is guaranteed only inside one partition. Events for two different orders may land in different partitions and be read in any relative order, and that is fine, because nothing requires order 2048 to precede order 2049.

The trap is the key itself. Suppose someone decides to key by tenant to keep a customer's data together. If harbor-retail produces 30 percent of 20,000 events per second, that tenant sends 6,000 events/s to one partition, while the uniform average across 24 partitions is 833.3 events/s, about 7.2 times more on the busiest one. The partition's owner consumer becomes the bottleneck and everyone behind it waits, while the other consumers sit idle.

There is no free fix. Adding partitions changes the hash-to-partition mapping for new records, which breaks the guarantee that old and new events for the same key share a partition unless the transition is planned. Using a finer key such as the order ID spreads load but gives up tenant-level ordering. Salting a hot key spreads it at the cost of order for that key. The right design chooses the smallest unit that actually needs ordering and keys by that, and treats partition count as a capacity decision made with headroom, because it is hard to reduce later. HarborCart's 24 partitions against a possible 48 is a concrete example: 48 would double the ceiling on active consumers per group and halve the average per-partition load to about 417 events/s, but it also doubles the leader and follower assignments the brokers track, makes every rebalance move more pieces, and leaves each partition's batches smaller. Whether that is worth it depends on measured consumer speed and on how close the peak really comes to the 24-consumer ceiling.

Partitions give the data parallelism; consumer groups give the readers a way to share it. A consumer group is a set of consumer instances that cooperate on one subscription, so that the partitions of a topic are divided among them and each partition has exactly one owner inside the group at a time.

Be precise, because the term gets abused. A group is not a partition, and it is not one consumer. email-notifier is a group; with 24 partitions and 12 instances, each instance owns 2 partitions. A different group, sales-analytics, reads the same 24 partitions with its own owners and its own offsets. Inside a group, ownership prevents two instances from processing the same partition's records at once, which preserves per-partition order for that group.

The group is also the scaling ceiling. More consumers than partitions leaves the extras idle, since a partition cannot be shared. At 12 consumers HarborCart's average load per instance is 1,666.7 events/s, below the hypothetical 2,000. Scaling the group to 30 instances would not help on 24 partitions: six would own nothing.

Now a consumer dies. A group coordinator on the broker side notices missing heartbeats, and a rebalance reassigns the orphaned partitions. 11 consumers survive and 24 = 2 x 11 + 2, so a naive even assignment gives two survivors three partitions and nine keep two. Those two heavier owners now carry about 2,500 events/s each on average, above the hypothetical 2,000, so lag (a growing backlog of unprocessed records, measured properly in Chapter 15) starts to build on them even though the group's total capacity looks sufficient. Cooperative or sticky assignment strategies reduce the damage by moving only the two orphaned partitions and leaving the other consumers undisturbed, whereas an eager rebalance stops everyone briefly and reassigns from scratch.

Two details help in interviews. In Kafka's classic group protocol the group's generation counter increments on each rebalance (12 to 13 in HarborCart's example), so a zombie consumer from the old generation cannot commit offsets for partitions it no longer owns. And rebalances themselves are a failure source: a consumer that takes too long to process a batch can be declared dead, trigger a rebalance, and cause the very duplicates it was trying to avoid.

Here is the incident that makes exactly-once sound easy until you try to implement it. The analytics consumer reads evt-HC2048-01, writes a row to its database, and then crashes before committing the offset.

Consider the two orders of operations. If the consumer writes the database row first and commits the offset second, a crash between them leaves the row written and the offset behind. On restart the consumer reads the event again and writes a duplicate row. If it commits the offset first and writes second, a crash between them leaves the offset advanced and the row missing, and the event is never read again; the update is silently lost. At-least-once with duplicates is recoverable. At-most-once with silent loss usually is not.

HarborCart picks commit-after-effect and makes the duplicate harmless. The consumer keeps an inbox table keyed by consumer and event ID. In one database transaction it inserts (sales-analytics, evt-HC2048-01) and applies the business change. If the insert collides with an existing row, the event was already handled and the consumer skips the effect. Only after the transaction commits does the consumer commit the offset.

The inbox works because the dedup check and the effect live in the same database transaction. If the effect targets something outside that database, such as a payment provider or an email sender, the transaction cannot span it. For those you need the external system's own idempotency key or an operation that is naturally repeatable. Also note the inbox needs a retention policy of its own; entries older than any realistic redelivery window can be pruned, or it grows forever.

The same race exists on the producing side. The order service must save the order in its database and publish OrderPlaced to the broker. Those are two systems, and no single transaction covers both.

Suppose the service writes the order, then publishes. If it crashes between the two, the order exists and no event does: email never fires, analytics misses revenue, and nothing flags the gap. Publish first and write second, and a crash leaves an event announcing an order that does not exist. Wrapping both in a try block does not help, because the failure is a process dying, not an exception.

The transactional outbox removes the gap by making the intent durable in the same database. In one local transaction the service inserts the order row and an outbox row containing the event. They commit together or not at all. A separate relay process polls (or tails the change log of) the outbox table, publishes each row to the broker, and marks it sent. If the service crashes, the order and the event are both present; the relay publishes later.

Look at what the outbox does not give you. The relay is at-least-once: if it publishes evt-HC2048-01 and crashes before it records that, it will publish the same event ID again. Atomicity is local to the originating database, between the order and the outbox row; it is not end-to-end exactly-once, and it does not cover the broker. That is fine, because the consumer's inbox from Chapter 12 already discards repeats by event ID. The two patterns are a pair: the outbox guarantees no event is lost, the inbox guarantees repeated events do not double-apply.

Not every failure is a crash. Sometimes a message is simply one the consumer cannot process, and the way you handle that decides whether one bad record can stop a whole pipeline.

Separate two kinds of failure. A transient failure, such as a timeout or a briefly unavailable database, is likely to succeed if tried again later. A permanent failure, such as a payload that fails validation or a schema version the consumer does not understand, will fail identically every time. Retrying the second kind forever is the classic poison message: it is delivered, fails, is redelivered, fails again, and in an ordered partition it blocks every message behind it.

HarborCart's policy is bounded and explicit. A message gets at most 5 delivery attempts. Retries back off with growing delay and jitter, illustratively 2 s, then 4 s, 8 s and 16 s, with randomness added so thousands of failed messages do not all retry at the same instant. After the fifth failure the record goes to a dead-letter queue (DLQ), here orders.dlq, retained for 14 days, carrying the original payload, the error, the attempt count and the trace ID. The partition moves on and healthy messages are no longer held hostage.

The DLQ is a quarantine, not a solution. Parking a message does not prove its side effect did not happen: a consumer may have written half its work before failing. Somebody has to look, fix the cause (a consumer bug, a bad producer, a missing schema) and then redrive, which means putting the messages back for processing. For topics that capture payments HarborCart makes redrive human-reviewed. Redrive can also reorder: a message from minute 10 re-enters after messages from minute 30, so if order matters, the redrive path must be designed for it.

Retries have a second failure mode that has nothing to do with bad data. If a downstream is down and every consumer retries aggressively, the retries themselves become the load that keeps it down. Bounded attempts, backoff with jitter, and a circuit breaker on the dependency keep failure from amplifying. Strict FIFO is also fundamentally at odds with retries: once a failed message is delayed and others pass it, the sequence has changed.

The point of decoupling is that a slow reader does not slow a fast one. It also means a slow reader's problem becomes invisible until you measure it, because the producer keeps succeeding.

Say the analytics database slows down on promotion day. sales-analytics falls behind. Consumer lag is the number of records between the end of the log and the group's committed offset. Oldest event age is how old the first unprocessed record is, which is often a better signal for people: "fraud is 40 seconds behind" is clearer than "lag is 3 million". Lag measures a backlog, and its growth rate tells you whether the group is catching up or falling further away.

Catch-up has simple arithmetic. If events arrive at 20,000/s and the recovering group sustains 50,000/s, it drains at net 30,000/s. After a 30-minute outage there are 36,000,000 events waiting, so the group needs 36,000,000 / 30,000 = 1,200 seconds, 20 minutes, at constant rates. If it could only sustain 20,000/s it would never drain. And the backlog sits inside the three-day retained log, so it is not extra disk; it is about 0.69 percent of that volume. What matters is whether the group catches up before retention deletes the records it has not read.

The same figures give the disk side of an outage. Thirty minutes at the average rate writes 20,000 x 1 KiB x 1,800 s, about 34.33 GiB of new events, and the same thirty minutes at the 100,000 events/s peak would be about 171.66 GiB. Neither is extra storage if the group is simply reading from the retained log; the cost shows up as catch-up bandwidth and as the time the backlog needs to drain.

Overload needs a bound at every stage or it moves somewhere worse. The broker enforces a maximum payload (1 MiB illustratively), per-tenant publish quotas, and limits on batch size. Capacity alerts have to fire long before a disk fills, because a full disk usually takes that storage out of service rather than degrading gracefully; quotas and retention are what keep producers from getting there. Producers should react to slow acknowledgments by slowing down, which is backpressure, rather than buffering without limit in memory. Consumers apply it by capping in-flight records and pausing fetches while downstream is slow.

Batching is the main tool for efficiency under load. Sending 100,000 single-event requests per second versus batches of 100 reduces requests to 1,000 per second, cutting request overhead by about a hundredfold. Each event may wait up to the batch's linger time before it is sent, so latency can rise; batching does not automatically improve p99 (the latency that 99 percent of requests stay under).

Keep three latencies apart, because they are easy to blur. Suppose a publish is acknowledged in 15 ms, covering the network round trip and replication to the in-sync followers. That number says nothing about how long the record waits in the log before a consumer fetches it, and nothing about the 400 ms an email consumer might then spend rendering and sending. The shopper's confirmation depends only on the first figure; the email arriving depends on all three added up. A dashboard that reports only publish latency can look excellent while consumers are minutes behind.

Everything so far assumed the broker keeps its data. A single broker with a single disk does not, so each partition is replicated: one broker is its leader, handling all reads and writes for that partition, and other brokers hold follower copies that fetch from it.

The set of followers that are caught up with the leader is the in-sync replica set, or ISR. HarborCart's orders topic uses replication factor 3 and min.insync.replicas=2, and producers wait for acknowledgment from the in-sync replicas. A write is acknowledged only when at least two replicas, the leader and one follower, hold it. That means an acknowledged record exists on two machines at the moment the producer hears success.

Walk through the failures. Broker L, the leader for a partition, dies. A controller (the component that tracks cluster metadata) elects one of the in-sync followers as the new leader, and the leader epoch increments, from 7 to 8 in the example, so stale leaders can be recognized and rejected. Clients receive updated metadata and retry to the new leader. Every record that was acknowledged is on the surviving in-sync follower, so none of them is lost; unacknowledged writes in flight may fail and be retried, which is another source of duplicates.

Now lose a second broker. Only one replica remains in the ISR, which is below the minimum of 2. The partition rejects writes with a not-enough-replicas error. The committed data is still on the survivor, but write availability is gone. Replication factor 3 does not tolerate any two failures *with writes*; it tolerates one, and tolerates two only for data survival.

The tempting escape is unclean leader election, letting an out-of-sync replica become leader so writes resume. That raises availability and sacrifices the guarantee: the replica lacks the committed suffix, so records producers were told were safe vanish. HarborCart keeps it disabled and prefers a stopped partition to a silently rewritten history. A topic carrying clickstream noise might make the opposite choice, and the decision should be per topic and written down.

A platform shared by teams and tenants needs rules about who may publish, who may read, and what the events are allowed to look like.

Start with identity. Authorization has to come from a verified principal, such as a client certificate or token the broker authenticates, and not from a tenant_id field in the message. If meridian, another tenant, sets tenant=harbor-retail in its own payload and tries to subscribe to the orders topic, the broker should refuse because the verified principal has no ACL (access control list) entry for that topic. Trusting caller-supplied metadata is how cross-tenant leaks happen.

Per-tenant quotas protect shared capacity. An illustrative limit of 5,000 events/s for harbor-retail means that a runaway producer is throttled or rejected before it fills the disks of everybody else. Payload size is capped too, and every publish, subscribe and administrative change is written to an audit trail keyed by the verified principal, so that "who read the orders topic last Tuesday" has an answer. Privacy goes in the design early: personal data should be minimized or redacted before it is appended, because the log is immutable by design, and deleting one customer's record from retained segments conflicts with the append-only model. Encryption, short retention and storing references instead of personal data are the usual answers.

Events live longer than any one deployment, so the format needs rules. A schema registry stores each event schema version and rejects changes that violate a compatibility mode. OrderPlaced at schemaVersion=2 is registered with BACKWARD compatibility: consumers using the new schema can read data written with the old one. Adding an optional field is allowed; renaming a required field or changing its type, as in a proposed v3, is rejected before it can reach the topic. Upgrade order follows from the mode, and it must be agreed because a retained log means old-format records will be read long after the producer changed.

Finally, put identifiers into the event itself. A trace context in the headers, such as the W3C traceparent value, lets a single order be followed from checkout through the broker to each consumer, and the OpenTelemetry messaging conventions describe how to model those spans. Without it, an incident that crosses an asynchronous hop is nearly impossible to reconstruct.

Two regions raise a question that sounds like a configuration option and is really a design choice: where does each event live, and which copy is the truth?

HarborCart runs two independent fabrics, EU and US, each a complete cluster. A topic such as orders.eu carries personal data of European customers and is tagged for residency: it is never mirrored across the border. Non-resident topics, say product-catalog changes, may be mirrored asynchronously. Asynchronous mirroring means the other region lags by some amount, and during a failure the lagging tail can be lost or duplicated.

There is no free global total order. Two regions accepting writes concurrently have no single sequence; at best you get an order per region, merged by timestamps whose clocks disagree. Active/active designs usually avoid the problem by assigning each key a home region and routing writes there. Disaster recovery follows the same logic: if EU fails, a failover that moves orders.eu consumers to a US cluster would violate residency, so unconditional failover is a promise the policy forbids. What the design offers instead is bringing the EU fabric back, perhaps in a second EU zone, and accepting an outage window.

A synchronous stretch across regions would give a single log but would pay a cross-region round trip on every acknowledged write, and would make each region's availability depend on the other. For most workloads the cost is not worth it, and teams pick per-region fabrics plus careful mirroring.

Failing a consumer over to a mirrored copy

Take the non-resident topic catalog-updates, mirrored asynchronously from the EU cluster to a US cluster, and a consumer group that reads the EU copy. A mirror is not a disk-level copy of the log. A mirroring process reads records from the source and produces them again into the destination, which assigns its own offsets. Source offset 4,201 in partition 7 may land at destination offset 3,876 because the mirror started copying partway through the log, or because the two clusters have expired different amounts of old data. An offset is a position in one particular log, not a property of the record, so the group's committed offset of 4,201 means nothing to the US cluster.

What identifies a record across clusters is the event ID the producer assigned, plus the source offset and timestamp when the mirror preserves them in the record's metadata. A mirroring tool that supports consumer failover also writes periodic checkpoints, each mapping a source offset to the destination offset of the same record. The recovery procedure then runs in order: confirm the source group is stopped, read the latest checkpoint at or below the group's last committed source offset, start the group on the destination at that mapped offset, and let the consumer's inbox discard repeats by event ID.

The translation is imprecise by design. A checkpoint is written periodically, so it trails the group's real progress and a restart re-reads some records the group already handled; that is why the inbox is required rather than optional, and why its retention must exceed the rewind window. Rewinding is safe and skipping is not, so when in doubt start earlier. The procedure does not promise records that never reached the destination (the recovery point is greater than zero), the original ordering across partitions, or that destination offsets equal source offsets afterwards: any dashboard, alert or replay ticket that quotes an offset must say which cluster it belongs to. Replay is also bounded by the destination's own retention, which may be shorter than the source's, so anything older comes from the upstream store. Rehearse it on a non-critical group before it is needed, and check that the inbox reports some discarded duplicates and that per-partition counts reconcile afterwards. The failure page walks through the same situation as an incident (F34).

A message platform fails slowly and quietly, so its operations are mostly about making slow failures visible early.

The signals that matter fall into a few groups. For the data plane: under-replicated partitions (followers behind), disk used against retention, and publish latency p99. For consumers: lag and oldest-event-age per group, rebalance frequency, and retry and DLQ volume. For producers: unknown-outcome publishes, error rate and throttling. For the outbox: age of the oldest unsent row. Each should have an alert whose action is clear; an alert that nobody knows how to act on is noise.

Changes need a rollback path. A consumer release that mis-handles a field can be rolled back and the group rewound, provided the retained log still covers the window and the consumer is idempotent. Capacity changes such as adding partitions or brokers should be rehearsed, because rebalancing data moves real bytes at the moment the cluster is already busy. Treat the platform's own failure as a scenario: what do producers do when the brokers are unreachable? The outbox answers it, by letting orders keep committing and the relay catch up later, but the outbox table then needs its own alert and a size bound.

Put the pieces together in the order the failures demanded them. An order commits with its outbox row in one transaction. The relay publishes to a replicated, partitioned log keyed by order. Independent consumer groups read it with their own offsets, apply effects through an inbox, and retry within bounds before parking in a DLQ. Shipment jobs go on a separate competing-worker queue with leases. Telemetry and governance surround all of it.

The two workloads stay distinct on purpose. shipment-jobs is a queue because the job is consumed once by whichever worker claims it, and the message is dead after ack. orders is a log because three teams read the same facts on their own schedules and the history is valuable. Using the log for jobs complicates per-job retry; using a queue for events recreates the fan-out problem.

Be ready to say what this design cannot do. It cannot give a global total order across partitions at scale; ordering exists per key within one partition. It cannot provide exactly-once effects across independent systems without a shared transaction or idempotent operations at the boundary. It cannot replay anything past retention, nor anything compaction has removed. It cannot fail a residency-tagged topic over across borders unconditionally. And it cannot keep accepting writes after losing two of three brokers with minimum ISR 2. Each of these is a boundary of the platform rather than a defect, and each has an operational owner. The boundary also runs the other way: if HarborCart only had the shipment jobs, or one team consuming a handful of events, a managed queue or an outbox table with a poller would meet the requirements at a fraction of the operational cost, and the replicated log would be justified only by the independent readers, replay and sustained volume described above.

The animated walk through HC-2048 from the synchronous V0 to this final design is in the visual journey embedded on this page, which replays each of these failures and the component introduced to answer it. The other pages in this module go deeper on specific parts and each stands alone.

Two vendor behaviors in this page are worth checking against primary documentation rather than taking from a summary: how Kafka's acks=all, min.insync.replicas and unclean leader election interact (the Kafka documentation's "Replication" section under Design, at kafka.apache.org/documentation/#replication), and the scope of its idempotent producer, which is a producer ID plus per-partition sequence numbers ("Message Delivery Semantics" in the same section). Everything else here is HarborCart's own design with invented numbers, and other brokers such as RabbitMQ or Amazon SQS make different promises.