Message Queue System Design Tradeoffs: Ordering, Durability & Scale

A message queue design is a stack of comparisons, and each one was settled by an incident someone remembers. This page reopens 28 of them for a fictional company, HarborCart: for every decision it names the problem that forces the question, the realistic options, what each costs, what HarborCart picked, the circumstances in which a different team should pick differently, and how you would find out you were wrong.

The running example is fictional and every number is an illustrative planning input, not a measurement. HarborCart is a commerce company. Order HC-2048 becomes an OrderPlaced event (id evt-HC2048-01, key order:HC-2048, tenant harbor-retail). Two shapes of messaging appear throughout. A queue hands each job to exactly one of several competing workers; the job ship-HC-2048 on the shipment-jobs queue is the example. A topic is a named stream that several independent readers consume; orders is the example, and its readers (email-notifier, sales-analytics, fraud-monitor) are consumer groups, each with its own place in the stream.

The vocabulary needed to follow the rest, defined here so this page stands alone: a producer publishes messages and a consumer reads them. A broker is the server that stores and serves them. A partition is one ordered, append-only slice of a topic; messages within it have increasing offsets (positions). A commit records how far a consumer group has read. An acknowledgment (ack) tells a sender or broker that something was received or finished, and what it proves depends entirely on who sends it and when. Replication copies a partition to other brokers; one copy is the leader that takes writes and the others are followers. Idempotency means that applying the same message twice has the same effect as applying it once. A dead-letter queue (DLQ) parks messages that keep failing. Backpressure is the signal that slows a producer when the system cannot keep up. Replay is re-reading retained history. Four more terms appear in the policies below and are explained properly in their own decisions: a lease (a worker holds a job for a limited time and loses it if it does not finish or renew), an outbox (a database table of events waiting to be published, Decision 4), an inbox (a table of event ids a consumer has already handled, Decision 1) and a tombstone (a record that marks a key as deleted, Decision 7).

The reference policies the decisions refer to

orders has 24 partitions, replication factor 3 (RF=3, three copies), min.insync.replicas=2 (a write must reach at least two in-sync copies before it counts as committed), producers wait for the in-sync replicas to acknowledge, unclean leader election is disabled, and retention is 3 days by time. The key is order:<orderId>. shipment-jobs leases a job to a worker for 60 s, allows at most 5 delivery attempts and then parks the job in a DLQ, with illustrative backoff of 2, 4, 8 and 16 seconds plus jitter. Consumers commit offsets only after their database transaction commits, and keep an inbox table keyed by consumer and event id. The planning load is 20,000 events per second on average, 100,000 per second for a 10-minute promotion, 1 KiB per event, 3 consumer groups and 3 days of retention.

A sentence to keep in mind while reading: a message system never removes failure, it chooses where failure becomes visible and who has to deal with it. Some decisions below move it to the producer, some to the consumer, some to an operator at two in the morning. The better design is the one where the failure lands somewhere with enough information to handle it.

The first four decisions are the ones teams skip because they feel obvious. They are also the ones whose wrong answers are the most expensive to reverse, because everything else is built on top of them.

D1 · Synchronous calls or asynchronous events?

HarborCart's original checkout (call it V0) called payment, inventory, email and analytics one after another inside the request. One afternoon the email provider slowed to ten-second responses. Checkout threads waited on it, the pool filled, and orders that had nothing to do with email began to time out. The customer had been charged and saw an error. The request had accumulated every downstream's availability into its own.

There were two real options. Keep the chain synchronous and harden it with timeouts, circuit breakers and bulkheads. Or split it: do the few things that must be true before the customer sees success (take payment, reserve stock) synchronously, then publish an OrderPlaced event and let everything else react in its own time. Hardening a synchronous chain reduces the blast radius but does not remove the coupling: checkout still has to know about email and still waits for it unless you add an explicit skip. The event version removes email from the request path entirely, at the price of eventual consistency (the confirmation email arrives seconds after the order, not before the response), a new component to run, and a new class of bug where an event is published twice or never.

HarborCart chose events for email, analytics and fraud review, and kept payment and stock synchronous because the customer's answer depends on them. A synchronous call wins whenever the caller genuinely needs the answer to continue, when volume is low enough that coupling is harmless, or when you cannot yet afford to operate a broker. The operational consequence is that the question 'did the email go out?' now has an answer that lives in a consumer group's lag, not in the checkout logs; if nobody watches that lag, the failure is silent instead of loud. A good validation is to kill the email provider in a test environment and confirm checkout latency does not move.

D2 · A process-local queue or a durable broker?

After the split, V1 used an in-memory queue inside the checkout service, drained by a thread pool. During a deploy the pod was terminated with 3,000 events waiting in memory. Those orders were paid and never emailed or analyze. A queue in process memory shares the fate of the process: the event existed only in RAM, and a restart has no memory of it.

The options are a process-local queue (fast, free, no new infrastructure), a local queue backed by a local disk write-ahead file (survives a process restart but not the loss of the machine), and a durable replicated broker (survives machine loss, costs a network round trip and an operations burden). The in-memory queue is perfectly acceptable for work whose loss is harmless and rebuildable, such as warming a cache or flushing metrics. It is wrong for anything with a business effect, because the failure it hides is exactly the one that ships an invisible loss.

HarborCart moved order events to a replicated broker and kept in-process queues for non-critical batching only. The in-process option wins for single-instance tools, tests and best-effort telemetry. The operational consequence of the broker is that there is now a place where messages wait, with its own capacity, so backlog and disk need alarms. Validate by terminating a producer pod under load and counting events: nothing acknowledged by the broker may be missing afterwards.

D3 · Competing work queue, independent pub/sub, or retained stream?

Three requests arrived in one quarter. Shipping wanted each paid order handled by exactly one worker out of several. Email, analytics and fraud each wanted to see every order, at their own speed. Finance wanted to rebuild last Tuesday's revenue figures after a bug fix. These look like one problem and are three.

A work queue with competing consumers gives each job to one worker, removes it after an acknowledgment, and scales by adding workers. A classic pub/sub system copies each message to every subscription's own queue, so every subscriber sees it, and the message is typically gone once all subscriptions have consumed it. A retained log keeps messages for a set period regardless of consumption; each consumer group tracks its own offset, so a slow group falls behind without affecting a fast one, and a new group can start from the beginning. The log gives you replay and many readers, but it does not by itself give you per-message work distribution with individual retry: a partition is owned by one consumer in a group, so a single stuck message blocks everything behind it in that partition.

HarborCart uses a work queue for shipment-jobs (one worker per job, lease and retry per job), and a retained, partitioned topic for orders (three groups, replay for finance). If you only need to hand jobs to workers and never need replay, a managed queue is simpler and the log is overkill. If a hundred subscribers need independent pace and history, a log fits and per-subscription queues multiply copies. The consequence to plan for is that each shape has its own lag vocabulary: queue depth and oldest-message age for the queue, offset lag per group for the log. Validate by running the same traffic through both shapes in a test: confirm a killed worker's job is redelivered to another worker on the queue, and that a new group added to orders can read from the start of retention.

Product names map onto these shapes only loosely, which is why the search for 'Kafka vs RabbitMQ vs SQS' rarely ends in one winner. Kafka is built around the partitioned retained log with consumer groups. RabbitMQ is a broker with exchanges and queues in which a message is normally removed once consumed; it also offers streams, an append-only log that consumers can read repeatedly until it expires, and quorum queues, a replicated queue type built on the Raft consensus algorithm (checked against RabbitMQ's documentation). Amazon SQS is a managed queue service: standard queues favor throughput with at-least-once delivery and best-effort ordering, while FIFO queues order messages within a group (verify the current limits and deduplication window in AWS documentation, which this page could not check). Amazon SNS is the pub/sub front that copies one message to many queues. Several products now offer more than one shape, so choose by the contract you need from the list above, then read that product's own guarantees.

D4 · A managed service or a broker platform you run?

When the shipment pipeline reached 5,000 tasks per minute (83.3 tasks per second), someone asked why a team of four was upgrading brokers on weekends. The honest comparison is between buying operations and buying control. A managed queue service gives durability, scaling and patching as a product; you give up some tuning, some ordering and replay features, regional placement control and sometimes cost predictability at high volume. A self-operated log gives retention, consumer groups, partition-level ordering and tuning; you take on upgrades, capacity planning, rebalancing, security and on-call.

There is also a middle path: a managed service that exposes the log API. It reduces toil but does not remove the need to understand partitions, keys and consumer lag, because those are properties of your design, not of who patches the nodes. Be careful with comparisons across products: Kafka-style logs, RabbitMQ and SQS are different systems with different guarantees, and none of the guarantees named here should be assumed to transfer.

HarborCart runs shipment-jobs on a managed queue-style service because the need is only 'deliver each job to one worker, retry, park the poison', and runs the orders log on a managed or self-hosted partitioned log because it needs retention, groups and key ordering. A small team with simple needs should buy; a platform serving many teams with residency, isolation and custom replication rules may need to build. The operational consequence of buying is dependence on a vendor's quotas and limits, so the question to ask early is which limit you will hit first and what the failure looks like. Validate with a game day: exhaust a quota deliberately and watch what the producer experiences.

Once messages flow through a broker, three questions decide most of the behavior users notice: who sets the pace, what a producer's acknowledgment actually proves, and how much latency you trade for throughput.

D5 · Push delivery or pull delivery?

A promotion started and the sales-analytics consumer, a slower service, was receiving events faster than it could write them. Its memory climbed and it was killed by the container runtime, taking in-flight events with it. In a push model the broker decides when to send and the consumer must say no somehow; in a pull model the consumer asks when it has capacity.

With push, the broker sends messages to the consumer as they become available, usually bounded by a prefetch window of unacknowledged messages. Latency is low because nothing waits for a poll, and a work queue with many fast workers benefits. With pull, the consumer fetches batches when ready; the broker never overwhelms it, batching is natural, and a lagging consumer simply reads later from the retained log. The cost of pull is a polling loop, and a small added latency when the log is idle unless long polling is used. The cost of push is that flow control becomes a protocol feature you must tune, and a misconfigured prefetch can strand messages on a slow consumer while other workers sit idle.

HarborCart's log consumers pull, because a retained log already lets a consumer fall behind safely and because fetching in batches suits the inbox transaction. The work queue uses a small prefetch so no worker hoards jobs it cannot start. Push wins when end-to-end latency for individual messages matters more than the consumer's autonomy, as with interactive notifications. The consequence to watch is the prefetch window: it is the number of unacked jobs a worker can be holding when it dies, so it sets how many leases will have to expire before recovery. Validate by killing a worker that holds a full prefetch window and timing how long until every one of its jobs is running elsewhere.

D6 · Ack at memory, at the leader's disk, or at a replicated commit?

A broker restarted after a power event and an order that the producer had been told was published could not be found. The producer's acknowledgment had been returned when the leader accepted the message into memory. That acknowledgment was truthful about what had happened and said nothing about whether the message would survive a crash.

There are three places an acknowledgment can be issued. At the leader's memory (fastest, and a leader crash loses it). After the leader has written to its log (survives a process restart, not necessarily the loss of the machine or a leader failover). After the message is committed to the required set of in-sync replicas (slowest by a network round trip, survives the loss of the leader). Each step earlier saves a few milliseconds and removes a class of loss. For HarborCart, a publish acknowledgment costs roughly 15 ms in the illustrative model, while a downstream consumer takes 400 ms; the producer's wait is not the dominant cost.

HarborCart's orders producers wait for the in-sync replicas, and the topic requires at least two. A lower acknowledgment is reasonable for telemetry or click events where losing a fraction is acceptable and throughput is the point. It is not acceptable for money. Keep the boundaries separate in your head: a producer acknowledgment says the broker has stored the message under the broker's policy; it says nothing about whether any consumer has seen it, and nothing about whether the business effect happened. The operational consequence is that during broker trouble the producer will see timeouts and rejections instead of silent loss, which is the desired failure but needs a producer-side plan (Decision 11). Validate by killing a leader mid-publish and comparing acknowledged ids with what consumers read.

D7 · Batching for throughput or sending each event at once?

During the 10-minute promotion HarborCart's producers moved from 20,000 to 100,000 events per second. Sending each event as its own request would mean 100,000 requests per second, each paying a header, a round trip and a broker request cycle. Grouping 100 events per request cuts that to 1,000 requests per second, about a hundredfold less per-request overhead for the same events.

The options are no batching (lowest delay for each event, highest request cost), size-based batching (send when enough bytes accumulate), and time-based batching (send after a short linger even if the batch is small). Batching also helps compression, since similar events compress together. It is not free: an event may wait up to the linger interval before leaving, so tail latency can go up, and a batch that fails is retried as a whole, which raises the stakes for idempotence (Decision 1). Whether p99 improves or worsens depends on load; at high volume a batch fills quickly and queuing at the broker shrinks, at low volume the linger is pure added delay. Do not assume either.

HarborCart enables batching with a short linger for orders, because 15 ms of publish latency is invisible next to a 400 ms consumer, and keeps near-zero linger on a low-volume fraud-alert topic where each message is urgent. Smaller batches win for latency-critical, low-rate topics; larger batches win when throughput and broker CPU are the constraint. The consequence is that producer memory and in-flight data rise with batch size, so a producer crash loses more unsent events if the application treats the client's send call as success. Validate by comparing publish p50 and p99 at average load and at 100,000 per second with and without the linger.

Ordering and parallelism pull in opposite directions, and a partition is where you settle the argument. Two decisions here shape every consumer you will ever write against the topic.

D8 · Key-based ordering or maximum partition parallelism?

Order HC-2048 produced OrderPlaced and then OrderPaid. A consumer that saw OrderPaid first would try to mark a nonexistent order as paid. Yet if all events went to one partition to preserve order, throughput would be capped by one consumer.

There are three positions. A single partition gives total order and no parallelism. Random or round-robin placement gives maximum parallelism and no order at all. Key-based placement gives order per key and parallelism across keys. The question the product must answer is which entities need their events in order: for HarborCart it is one order's lifecycle, not the order of two unrelated customers' orders. Key by order:<orderId>, and OrderPlaced lands on partition 7 at offset 4,201 with OrderPaid at offset 4,202, in sequence. A global order across all orders is unavailable at this scale and unnecessary.

HarborCart keys by order id. A tenant-only key is simpler to reason about but concentrates a busy tenant on one partition (Decision 10). Round-robin wins for work that is independent and needs no order, such as stateless image resizing. Strict per-key order also depends on more than the key: retries after failures, multiple consumers and redelivery can still reorder what a consumer observes unless producers and consumers are designed for it (Decision 1, Decision 5). The operational consequence is that the key is a long-lived contract: changing it later remaps keys to different partitions, so a consumer can briefly see an old and a new location for the same key. Validate with a test that publishes interleaved lifecycles for many orders and asserts per-order monotonic sequence at the consumer.

D9 · How many partitions?

A partition limits parallelism within a consumer group: each partition has one owner in the group at a time. With 24 partitions, at most 24 consumers in a group do work and the rest sit idle. The promotion arithmetic shows the problem: at 100,000 events per second and a hypothetical 2,000 events per second per consumer, a group would need 50 consumers, so 24 partitions cap it at 48,000 per second and 48 partitions at 96,000, still short.

Too few partitions and a group cannot scale and a single hot partition has little room to spread. Too many and every partition costs something: more files and replication streams on the brokers, more metadata for the controller, more time to elect leaders after a failure, more consumers needed to make use of them, and a weaker ordering story if you later change the count. Increasing the count on a keyed topic changes key-to-partition mapping for new messages, so per-key order across the change needs a plan. At the planning rate of 20,000 per second, 24 partitions average 833 events per second each (about 0.81 MiB/s); 48 average 417. Do not treat any number as a per-partition capacity limit that holds everywhere; it depends on payload, hardware and the consumer's work.

HarborCart chose 24 for orders: enough for the 10 consumers per group the average needs and headroom for growth, with a documented fact that a 100,000-per-second peak either needs 50 or more partitions, faster consumers or tolerated lag. A higher count wins if peaks cannot wait and consumers are slow; a lower count wins for small topics where metadata and election cost dominate. The operational consequence is that you decide partition count against the peak you can tolerate lagging, so write that tolerance down. Validate with a rehearsal of the 10-minute promotion and measure how long lag takes to drain afterwards.

Almost every pipeline delivers at least once, because the alternative is losing work. That makes duplicates a normal event, and makes the order of 'do the work' and 'record that it was done' a decision rather than a detail.

Decision 1 · Producer idempotence or application-level deduplication?

A producer sent evt-HC2048-01, the network dropped the acknowledgment, and the client retried. The broker had already written the first copy, so there were two. The producer cannot tell the difference between 'the broker never received it' and 'the broker received it and the reply was lost'.

In recent Kafka-style clients, idempotence is on by default and relies on acknowledgment from the in-sync replicas, retries and a bounded number of in-flight requests; the broker detects duplicate sequence numbers from the same producer session per partition. This protects against transport retries. It does not protect against the application restarting and republishing with a new producer identity, nor against the outbox relay publishing the same row twice, nor against the same business event being generated by two services. Application-level deduplication uses a stable event id and an inbox table keyed by consumer and event id, and protects the actual side effect regardless of how the duplicate arose, at the cost of a database write and a lookup.

HarborCart uses both: the idempotent producer to keep transport retries from adding noise, and the inbox on every consumer as the real defense. Relying on the producer alone is tempting and wrong, since its guarantee ends at the producer session. Skip the inbox only when the effect is naturally idempotent, such as setting a field to a value. The operational consequence is inbox growth, which needs a retention rule longer than the longest redelivery window, and a metric for duplicate-hit rate that rises when something upstream misbehaves. Validate by killing the relay between publish and mark, as in the outbox scenario, and checking that the effect happens once.

Decision 2 · At-most-once or at-least-once processing?

Consider the question 'what should a consumer do if it is not sure the effect happened?'. If it skips, a job may be lost. If it repeats, a job may happen twice. Nothing in a network gives exactly-once effects across independent systems; what is achievable is at-least-once delivery plus idempotent handling, which gives an effectively-once outcome for the effects you control.

At-most-once commits or acknowledges before processing: no duplicates, possible loss. At-least-once processes and then acknowledges: no loss on a consumer crash, possible duplicates. Transactions inside one Kafka-style system can make read-process-write atomic across topics and offsets, but they do not extend to an external database or a payment API. A payment capture must not happen twice and must not be silently dropped, so neither pure posture is enough alone; the answer is at-least-once plus an idempotency key on the capture.

HarborCart treats every consumer as at-least-once with an inbox, and accepts at-most-once only for ephemeral signals such as a live dashboard tick where a missed update is replaced a second later. The consequence is that every downstream write must be designed for repetition, which is a code-review rule, not a broker setting. Validate with fault injection that crashes consumers at every step and compares end state with a run that had no crash.

Decision 3 · Commit the offset before or after the side effect?

Two incidents define the options. In the first, the analytics consumer wrote its database row and crashed before committing the offset. On restart it read the same event again and wrote it again, producing a duplicate row. In the second, a worker acknowledged a shipment job before its business transaction committed; the transaction then failed, the job was gone from the queue, and the shipment never happened.

Committing before the effect is 'at-most-once': a crash between the commit and the effect loses the message from the group's point of view. Committing after the effect is 'at-least-once': a crash between the effect and the commit repeats it. Committing in the same transaction as the effect is possible when the offset is stored in the same database as the result, which makes the pair atomic for that database. A committed offset is a statement about the broker's bookkeeping; it does not prove the payment captured or the row was written.

HarborCart commits manually after its database transaction commits, with the inbox absorbing the repeat. Storing the offset inside the effect's database wins when the sink is a single relational store and you want exact positioning without a separate inbox. Commit-first is acceptable only where loss is harmless. The operational consequence is that duplicate handling is exercised on every restart, so it must be fast; and an automatic commit interval on a consumer library can silently commit before work is done if you leave it on, so disable it explicitly. Validate by crashing between each pair of steps in a test harness.

Decision 4 · Transactional outbox, distributed transaction, or best-effort dual write?

The order service had to do two things: save HC-2048 in its database and publish OrderPlaced. Written as two steps, a crash between them leaves an order with no event, or, in the other order, an event for an order that never saved. The database and the broker do not share a transaction, so no ordering of the two calls is safe.

A best-effort dual write does both calls and hopes; it is simple and loses data during exactly the failures that matter. A distributed transaction (two-phase commit) tries to make both succeed or fail together; it needs both systems to support the protocol, holds locks across a network, and makes the broker's availability part of every order's latency. A transactional outbox writes the order and an outbox row in one local database transaction, then a separate relay reads outbox rows and publishes them, marking them sent. That is atomic where atomicity is available and at-least-once where it is not: if the relay crashes after publishing and before marking the row, the same event id is published twice.

HarborCart uses the outbox plus an inbox at consumers, so a duplicate publication is absorbed. A distributed transaction wins in a closed environment where both resources support it and the team accepts the coupling; a dual write is acceptable for notifications whose loss is cosmetic. The operational consequences of the outbox are a relay process to monitor, an outbox table that grows if the relay stalls, and publish order that follows table order rather than commit time unless designed carefully. Validate by crashing the relay at each step and checking that no order lacks an event and every duplicate is harmless.

Some failures disappear if you try again and some are the message itself. Retry policy and dead-letter policy are two answers to the same question: how do you keep a bad moment or a bad message from becoming a bad hour?

Decision 5 · Retry inline and requeue, or use retry topics and delayed delivery?

The payment-status service went down for four minutes. Consumers on shipment-jobs retried failed jobs immediately, each retry failed again, and the retries added as much load as the original traffic to a service that was trying to recover. A retry storm is a failure amplifier: the fix for the problem sits behind the load the fix generates.

Inline retry retries in the consumer loop. It is simple and fast for blips, but a long outage blocks the consumer, and in a partitioned log it blocks everything behind the message. Immediate requeue puts the job back on the queue; it can spin in a tight redelivery loop and starve other work. Retry topics or delay queues move the failed message to a place where it waits for a backoff interval (2 s, 4 s, 8 s, 16 s with jitter, illustrative), freeing the main path, at the cost of more topics, the loss of strict ordering for that key and a more complex trace of a single message's life. Jitter matters: without it, many failures retry at the same instant and arrive as a wave.

HarborCart retries briefly inline for transient errors, then moves to delayed retries up to 5 attempts, then parks. It classifies failures first: a timeout is transient, a validation error is permanent and goes straight to the DLQ without retrying. Inline retry wins when the dependency is local and the error rate is tiny; a delay topic wins when downstream outages last longer than a poll. The consequence is that ordering for a key is no longer guaranteed once one of its events takes the retry route, so per-key consumers must tolerate or detect late events. Validate by taking the dependency down for a few minutes under load and watching the retry rate and lag, not just the error count.

Decision 6 · Park poison messages in a DLQ, or stop the partition?

An OrderPlaced event with a malformed field crashed the fraud-monitor consumer every time it was read. The consumer restarted, fetched the same offset, crashed again, and the partition behind it, holding thousands of healthy events, stopped moving. A poison message is one that can never succeed however many times it is attempted.

Stopping the partition preserves order and guarantees nothing is skipped silently; it also converts one bad message into an outage for every healthy key on that partition. Skipping to a dead-letter queue keeps the line moving and preserves the message for inspection, but it means later events for the same key may be processed before the parked one is repaired, which is an ordering violation and, for stateful consumers, a possible wrong result. DLQ parking does not prove the side effect did not happen: the failure may have occurred after a partial effect.

HarborCart parks after the attempt limit into orders.dlq, retained 14 days (illustrative), with the original headers, error and attempt count, and alerts on any arrival. For payment-capturing topics redrive is reviewed by a person. Stopping the partition is the right answer when order is the entire point and skipping would corrupt state, for example a ledger where a later entry depends on an earlier one; then you accept the outage and make the pause loud. The operational consequence of a DLQ is that someone owns it: a DLQ nobody reads is a silent data loss with extra steps. Validate by injecting a poison event and measuring how long until a human is told.

A log is a promise about the past and a schema is a promise about the shape of every message. Both are made by people who do not yet know their consumers.

Decision 7 · Retention by time, by bytes, or by compaction?

A consumer group for sales-analytics was stopped for a week to fix a bug. When it came back, its committed position had been deleted by the retention window, and the oldest data it needed no longer existed. Retention decides how long a slow or broken consumer can recover from the log alone.

Time-based retention keeps data for a period, such as the 3 days for orders, and the storage need follows rate times time: 20,000 events per second at 1 KiB is 19.53 MiB/s, about 1.61 TiB per day logical, so 3 days is about 4.83 TiB logical and 14.48 TiB with three copies, before indexes, compression, skew and headroom. Size-based retention caps storage, but the time it covers shrinks during a spike, exactly when you may need it. Compaction keeps the latest value per key and deletes older ones, which suits state topics (a customer's current address) and not an audit log, since intermediate values vanish and a late-starting consumer will never see them. A tombstone deletes a key after a configured delay.

HarborCart keeps orders for 3 days by time because the recovery objective is a long weekend, and it is the length of outage the team has committed to absorbing without extra tooling. Size caps win where disk is the hard limit and consumers are never down long. The consequence is that retention is a recovery promise: any consumer down longer than 3 days must rebuild from a snapshot or the source of truth. Committed offsets also expire after a group has been empty for the configured period (a week by default in Kafka), so a returning group may restart at a position decided by its reset policy, not where it left. Validate with a restore drill that starts a group after a pause longer than the offset expiry.

Decision 8 · Strict schemas or loose evolution?

A developer renamed a field in OrderPlaced from total to grandTotal and deployed the producer. Consumers on the old schema silently read null and began writing zero-value rows. Nothing crashed; every number was wrong. Producers and consumers deploy on different schedules, so a message published today may be read by a consumer built a year ago, and a replayed message from last week by code written next month.

A schema registry with compatibility rules validates every new version against the old ones before publication. BACKWARD compatibility means new consumers can read old data; the rule HarborCart applies to OrderPlaced, so adding an optional field is accepted and retyping a field is rejected. Looser rules ('anything goes', or untyped JSON with convention) move fast and push failures to consumers at runtime. Stricter rules (full compatibility, or versioned topics) cost agility: changing a meaning requires a new topic and a migration. A later change to OrderPlaced, from schema version 2 to a version 3 that redefines what a field means, is the incompatible example, and the answer is a new event type or topic with a transition period, not an in-place edit.

Loose evolution wins in a small team with one producer and one consumer in the same repository, deployed together. Strictness pays as consumers multiply and as replay makes old data permanently reachable. The operational consequences are a build step that fails a producer change before it ships, and a policy for who may approve an exception. Validate by replaying the oldest retained message through the newest consumer in continuous integration.

A shared platform fails through its busiest participant. These three decisions are about giving a single tenant, key or consumer less power over everybody else.

Decision 9 · Shared multi-tenant cluster or isolation tiers?

A second company, meridian, wanted to publish to the same cluster as harbor-retail. A bug in a harbor-retail job then published at ten times its usual rate and the cluster's disk and network headroom shrank for everyone. Meanwhile the question of whether one tenant could read another's topic had to be answered by something stronger than convention.

A shared cluster has the best utilization and the lowest cost per tenant; isolation comes from topic ACLs bound to a verified principal, per-tenant publish quotas (illustrative 5,000 events per second for harbor-retail) and a payload cap (1 MiB illustrative against a 1 KiB average). Tiered isolation gives large or regulated tenants their own topics, brokers or clusters: noisy neighbors cannot reach them and blast radius shrinks, at the cost of duplicated operations, less efficient capacity and more things to upgrade. Dedicated clusters for every tenant do not scale operationally.

HarborCart shares one cluster across tenants with ACLs and quotas, and reserves dedicated capacity for any tenant that needs a contractual guarantee or has a regulatory boundary. Dedicated infrastructure wins when a tenant's load is large enough to dominate, or when audit requires physical separation. The operational consequence is that quotas must be monitored for throttling, and tenants told when they are throttled, or the first sign is a support ticket. Validate by running a tenant at 10 times its quota in a test and confirming other tenants' latency does not move.

Decision 10 · Re-key a hot key or keep strict per-key order?

Suppose the topic were keyed by tenant instead of order, and a large tenant such as meridian produced 30 percent of the 20,000 average events per second. That tenant's 6,000 events per second land on one partition, about 7.2 times the uniform average of 833. One consumer is overloaded, its lag grows, while the other 23 consumers sit far below capacity and the group looks 'healthy' on average. Even with the order key, a single enormous order or a bulk import can create the same shape.

Options: keep the key and accept the skew, because strict order for that key matters; add a salt so one logical key spreads across several partitions, which raises throughput and breaks order across the salted pieces; or split the work, for instance by order id within tenant, which preserves per-order order and spreads the tenant. Salting works only when the consumer does not need total order for that logical key or can reassemble order from sequence numbers.

HarborCart's real answer was to choose the right key to begin with. Where a key is inherently hot, such as a single account that is ledger-ordered, strict order is kept and capacity for it is planned, because correctness comes first. Re-keying wins for aggregation-style work that can merge partial results later. The operational consequence is that partition-level metrics matter: average lag across the group conceals a single hot partition. Validate by dashboarding per-partition throughput and lag and alarming on the ratio of the hottest partition to the median.

Decision 11 · Block producers, reject within a bound, or spool and absorb?

In the promotion, producers published faster than a partly impaired cluster could accept. Producer buffers filled and checkout threads started to block on the publish call, bringing back the V0 coupling through a different door. When a system cannot keep up, something has to give: latency, completeness or availability.

Blocking the producer when its buffer is full slows the whole source, preserving every event while risking the very request path D1 protected. Bounded rejection returns an error to the caller after a short timeout, so the application can degrade deliberately: serve the order, record the event in the outbox and publish later. Spooling to local disk or an outbox absorbs the surge and trades latency for completeness, and moves the question to how big a backlog you can afford. All three need a ceiling: an unbounded buffer is a delayed crash.

HarborCart's checkout never blocks on the broker; the outbox absorbs the burst and the relay is rate-limited to what the cluster accepts, with lag and outbox depth as the alarms. Blocking is right for batch pipelines where nobody is waiting and completeness matters; rejection with load shedding is right for non-critical streams under extreme load. The consequence of the outbox approach is that downstream consumers see events later during a surge, so 'event time' lag (Decision 17) becomes visible. Validate by throttling the broker in a rehearsal and watching checkout latency, outbox depth and drain time.

This chapter is where the design stops being about your code and starts being about machine loss, network partitions and borders. The decisions are about what a write waits for, who is allowed to write, and what may leave a region.

Decision 12 · Three-replica quorum durability or a lower-latency weaker acknowledgment?

A broker host failed with a disk fault. Partitions whose leader lived there needed a new leader, and the question that mattered was whether every message acknowledged to a producer was still on another broker. It depends entirely on what the acknowledgment waited for.

The options sit on a line. Acknowledge after the leader appends (lowest latency, a leader loss can lose acknowledged messages). Acknowledge after all in-sync replicas have it (survives losing the leader, and the number of copies that must be in sync is a setting). Require a stricter quorum of copies (more durable, more write latency, less availability when replicas are down). HarborCart's orders runs RF=3 with min.insync.replicas=2: writes continue while at least two copies, including the leader, are in sync, so losing one broker still allows writes. Losing two leaves one in-sync copy, below the minimum, so writes are rejected with a not-enough-replicas error. The data already committed is still on the survivor; what is lost is availability for writes, not the committed history.

That is a deliberate trade: refusing writes is chosen over accepting writes that exist on a single machine. Do not describe RF=3 as surviving any two failures with writes. Lowering the minimum to 1 wins for topics where continued ingestion matters more than the last few seconds of durability, such as clickstream; raising to RF=5 helps where two simultaneous losses are plausible and costs storage and replication traffic. The operational consequence is that the number of in-sync replicas becomes an alert, because shrinking toward the minimum is the early warning. Validate by killing one broker, then two, under load, and recording what producers see and what is readable afterwards.

Decision 13 · Leader–follower single writer or a quorum log?

Both models solve the problem of two nodes writing different things at the same offset. In a leader-follower partition, one broker at a time is the leader; followers copy its log, and a new leader is chosen from followers that are in sync if the leader fails. A leader epoch (7 becoming 8 in the example) lets brokers tell a current leader from a stale one, so a leader that was partitioned away cannot keep appending after it has been replaced.

In a quorum log (a Raft-style design, which RabbitMQ's quorum queues use) a message is committed when a majority of replicas (⌊N/2⌋+1) hold it, and an election requires a majority too. Leader-follower with an in-sync set lets you tune how many copies must acknowledge and keeps throughput high with replicas chosen by the controller; quorum systems have a cleaner proof of what is committed and tolerate slow replicas without explicit membership tracking, but write latency follows the majority and the replication protocol is more opinionated. A leader-follower log can lose acknowledged data if its configuration allows an unclean leader election, which lets a replica that is not in sync become leader: availability returns sooner, but the new leader lacks the tail of the old log, so that committed suffix is discarded. That is why HarborCart has it disabled. A majority-quorum design avoids this particular loss by construction, because a replica missing committed entries cannot win the vote.

The reference design uses leader-follower with in-sync tracking because the target is a high-throughput partitioned log. A quorum model wins for smaller, strongly consistent queues where operational simplicity of the commit rule matters more than raw throughput. The consequence of the single-writer model is that every partition's availability depends on a leader election, so election time during a broker loss is a recovery-time input, and many partitions per broker means many elections at once. Validate by failing a broker and measuring the gap between the last write on the old leader and the first on the new, then checking for an epoch change on every affected partition.

Decision 14 · Synchronous cross-region replication or asynchronous regional replication?

HarborCart expanded into the EU and US. Two requests collided: the US team wanted all orders visible for global analytics, and EU law required EU customers' order data to stay in the EU. A single stretched cluster would have put replicas on both sides of the Atlantic and added the round-trip to every write that waited for them.

Synchronous cross-region replication makes a write wait for a remote region, so the loss of a region need not lose acknowledged data, but every publish pays the inter-region delay and a partition between regions blocks writes. Asynchronous mirroring writes locally and copies later: publish latency is local, and a regional loss may lose the most recent unmirrored messages and requires deciding which side is authoritative after failover. Independent fabrics with no mirroring at all are the only option for residency-tagged data, and failover to another region may be impossible by rule.

HarborCart runs independent EU and US fabrics. orders.eu is never mirrored across the border; non-resident topics mirror asynchronously, and consumers must treat the mirror as eventually consistent and possibly behind. Synchronous replication wins where regions are close, latency is acceptable and zero data loss on regional loss is a contractual requirement. The design cannot promise unconditional cross-region failover once residency applies, and operators must not be given a runbook that suggests otherwise. Validate by a regional isolation drill that cuts the link, reports mirror lag in time units, and tests which topics are allowed to fail over.

The last five decisions are the ones that surface during an incident review: how a downstream system is rebuilt, how large a message is allowed to be, which clock a lag number uses, and which repairs a person has to approve.

Decision 15 · Replay-first recovery or snapshots and materialized views?

A bug in the analytics consumer wrote wrong totals for six hours. The retained log still holds those orders, so the group can be rewound to a past offset and re-run. That is the appeal of a log: the fix is a replay, not a restore. But replay re-applies effects, and replaying OrderPlaced into a system that sends emails or captures payments would repeat them.

Replay-first recovery rebuilds state by reprocessing retained events; it needs events to be the source of truth, consumers whose effects are idempotent or reversible, and enough retention to reach the fault. Snapshots and materialized views save the derived state periodically so recovery loads the snapshot and replays only the tail; recovery is faster and retention can be shorter, at the cost of storing, verifying and versioning the snapshots. Replay beyond retention is impossible, whichever you prefer.

HarborCart replays for read-model consumers such as analytics, writing to a fresh target and switching over after validation, and never replays into effect-producing consumers without a guard that disables side effects. Snapshots win when history is long, rebuilds would take days or retention is short. The operational consequence is that rewinding a group is a production change that needs a rehearsal, rate limits so it does not starve live traffic, and an expected duration. Validate by performing a full rebuild into a shadow table and comparing it with the live one.

Decision 16 · Small events or a claim-check pointer to an object store?

A team wanted to attach a customer's invoice PDF to the OrderPlaced event. At 1 KiB average, the platform sizing is easy; a 2 MiB PDF is 2,000 times larger and, at 20,000 per second, would turn 19.5 MiB/s into a stream far beyond the plan. The payload cap is 1 MiB illustrative; larger messages stress brokers, memory in producers and consumers, replication and every consumer group's fetch, since three groups would each read it.

Option one: put the large body in the message, which is simple and atomic but multiplies replication and fan-out cost and risks hitting size limits. Option two: store the object in blob storage and send a claim check, a small message with a pointer, size and checksum, so the broker carries only the reference. That keeps broker traffic small but introduces a second system: the object must exist before the event is visible, must outlive the slowest consumer, needs access control, and can be deleted while an event still points to it, which breaks replay. A third option is to split by importance: events carry the facts consumers need, with a link for the heavy detail.

HarborCart keeps events small and uses claim checks for documents, with the object retained at least as long as the topic retention and the pointer carrying a checksum. Inline payloads win when messages are modest and atomic delivery of the full body matters. The consequence is that deleting or lifecycle-expiring objects needs coordination with topic retention, or replayed events will point at nothing. Validate by replaying a three-day-old event and fetching its object.

Decision 17 · Event time or ingest time for measuring lag?

The dashboard for fraud-monitor showed a lag of 40,000 messages and nobody knew whether that was alarming. At 20,000 events per second it is 2 seconds of backlog; at 100 per second it is more than six minutes. A count of messages is not a measure of delay without the rate.

Offset lag is the distance between the log's end and the group's position; it is easy to compute at the broker and says nothing about time. Ingest-time age is how long ago the broker received the oldest unprocessed message; it measures the consumer's delay. Event-time age compares with when the event actually happened at the source (eventTime on HC-2048), so it includes upstream delays such as the outbox sitting backed up, but also depends on the producer's clock, which may be wrong. Any client timestamp comparison needs a clock-error caveat.

HarborCart alerts on oldest unprocessed age from ingest time for consumer health, and tracks event-time freshness separately for business SLOs, accepting a margin for clock error. Offset lag still has a place for capacity trends. Event time wins when the business question is how stale the data is for a customer, whatever the cause. The operational consequence is that SLOs should be written in seconds of age, not messages, so they survive traffic changes. Validate by pausing a consumer for a known time and checking that the metric reads that time.

Decision 18 · Operator-reviewed redrive or automated retry for failed payments?

After a fix was deployed, 600 payment-related events sat in orders.dlq. An engineer proposed a script to push them all back to the main topic. Some of them had failed after the payment provider had already captured the money; resubmitting those would charge customers twice. A DLQ message does not tell you whether the side effect partially happened.

Automated redrive is fast and scales, and is fine when the effect is idempotent or reads only. Operator-reviewed redrive means a person (or a verified script) checks each class of failure against the system of record before returning it, at the cost of time and staffing. A middle path is automated redrive limited by classification, say schema-rejects after a fix, with an idempotency key on each payment attempt so a repeated capture is a no-op. Redrive can also reorder events relative to newer ones for the same key, so FIFO scope is lost across it.

HarborCart requires review for payment-capturing topics and permits automatic redrive for read-model topics. Automation wins when the effect is safe to repeat and volume is large. The operational consequence is a runbook with explicit questions: what was the failure, was any effect committed, will redriven messages arrive behind newer ones. Validate in a game day with a deliberately poisoned batch that includes a half-applied case.

Decision 19 · Compaction on key-state topics or append history on audit topics?

Two topics, two jobs. customer-profile should always hold each customer's current address and plan, so a new service can start by reading the latest value per key. order-audit must show every change to every order, for a dispute months later. Using the same retention policy for both fails one of them.

Compaction keeps at least the latest record for each key, discarding superseded ones; a tombstone, a record with a null value, marks deletion and is removed after a delay. It bounds storage by the number of keys and lets a late consumer rebuild current state, but a consumer that starts late can miss intermediate values, and compaction runs in the background so older values may linger for a while. Append history with time retention keeps every event for a period, which supports audit and replay for that period only; keeping it forever is a storage decision, and beyond retention the broker is not an archive. Long-term audit belongs in an archive with its own governance.

HarborCart compacts state topics and keeps audit events in time-retained topics that are also written to an archive. Compaction wins wherever the question is 'what is true now?', append history wherever it is 'what happened, in order?'. The consequence is that compacted topics need stable keys and care with tombstones, and the permanent record is not the broker. Validate by starting a new consumer from the beginning of each topic type and confirming it gets what the use case needs.

The 28 decisions are connected. Keying by order (D8) is what makes per-order sequence possible and also what exposes a hot key (Decision 10); at-least-once (Decision 2) is what makes the inbox (Decision 1, Decision 3) mandatory; the outbox (Decision 4) is what turns a broker outage into backlog (Decision 11) rather than a failed checkout. Seen together, HarborCart's design is a set of bounded failures in which each remaining risk has an owner.

Two themes recur. First, every guarantee has a boundary and the design is mostly about placing the boundary where somebody can see it: the producer's acknowledgment stops at replication, the offset commit stops at the broker's bookkeeping, the DLQ stops at 'parked', and the idempotent producer stops at its session. Second, most choices trade a loud, bounded failure for a quiet, unbounded one. Rejecting writes when two copies are lost is loud; accepting them on one machine is quiet. A full DLQ is loud; a blocked partition is loud in a different way; a skipped message is quiet.

Some things stay out of reach and the page should not suggest otherwise. There is no global total order at scale. There is no exactly-once effect across independent systems without a shared transaction. Replay stops at retention. Cross-region failover is conditional once residency rules apply. Writes do not survive any two broker losses. Where a team needs one of these, the design question is whether the requirement is real and what it is worth.

• State the boundary each acknowledgment crosses, in writing, for each topic.
• Alert on oldest-message age and per-partition skew, not only average offset lag.
• Rehearse the promotion, the broker loss and the DLQ redrive before relying on them.
• Give every DLQ, quota and override an owner.

• Treating a producer acknowledgment as proof that a consumer saw or processed the message.
• Auto-redriving failed payment events without checking the system of record.
• Copying the illustrative HarborCart numbers into a capacity plan without measuring your own.
• Assuming a managed service removes the need to understand keys, partitions and lag.