Distributed Message Queue System Design Interview Guide

When the caller does not need the downstream result to finish its own job. At HarborCart the order must be accepted only after payment is authorized, so that call stays synchronous. Confirmation email, analytics, fraud scoring and shipment creation have no bearing on whether the customer sees success. Calling them inline means checkout latency is the sum of four services and checkout availability is their product: four dependencies each 99.9% available give roughly 99.6% together, an illustrative figure. Publishing one durable OrderPlaced event and letting each service consume it moves that coupling off the request path. The price is real. The customer can see success before the email exists, work becomes eventually consistent, and there is now a broker to run and a duplicate-handling problem to solve. If the caller needs the answer to continue, for example an inventory reservation that decides whether the order can be placed, a queue turns a simple call into a hard-to-debug saga. A queue also does not remove load; it defers it, so a downstream permanently slower than the arrival rate just builds a backlog.

• What if payment is slow? — Keep it synchronous but bounded with a timeout and circuit breaker, or accept the order as pending and complete payment asynchronously, which changes the customer-visible contract.
• Where does the order write fit? — The order row and the event must be committed together, which is what the outbox in a later question is for.

They differ in who gets a message and whether it outlives being read. A queue holds work: each message is claimed by one consumer among several competing for it, and once acknowledged the broker may delete it. shipment-jobs is this, and job ship-HC-2048 goes to Worker A or Worker B, never both except through a redelivery. Pub/sub is addressing by topic: a publisher names a topic and every subscription receives its own copy, so email, analytics and fraud each see OrderPlaced. A retained event log is a storage choice that is often put behind pub/sub: the broker appends events to an ordered partition and keeps them for a retention period, with each reader holding its own position (offset). Reading does not remove anything, so a new subscriber can start from the oldest retained event, and a fixed bug can be replayed. Many products combine these. A log can serve a work-queue pattern through a consumer group, and a queue product can fan out by binding several queues to one exchange. The question to ask of any system is what deletes a message, who decides, and whether a reader can go back.

• Is a Kafka consumer group a queue? — Within one group each partition has one owner at a time, which behaves like competing consumers, but the data stays and other groups read it independently.
• Can a queue replay? — Only if the product keeps acknowledged messages, which is what RabbitMQ streams or a log are for; a classic queue deletes on ack.

Each subscription keeps its own cursor over the same data, so no subscriber's progress affects another's. email-notifier, sales-analytics and fraud-monitor are three consumer groups on orders. When evt-HC2048-01 is appended at partition 7, offset 4,201, nothing is copied per subscriber; each group's committed offset for partition 7 says how far it has read. If analytics is down for an hour, its offset stays behind and its backlog grows while the other groups carry on at the tail. In a queue-based broker the equivalent is one queue per subscriber bound to a shared exchange or topic, where the broker really does copy the message into each queue. The log approach makes the three groups cost reads, not storage: at 20,000 events/s that is 60,000 group deliveries a second, which are logical reads and not extra appends. The caveat is that each group is separately responsible for its own duplicates, because a delivery to one group can repeat without the others seeing it. Fan-out therefore never means that every subscriber gets exactly one copy; it means each subscription gets the event at least once, within retention.

• What if a new subscriber joins next week? — It can start at the earliest retained offset and read three days of history, or at the latest offset and see only new events; either choice is explicit.
• What if one subscriber should not see some events? — Separate topics or ACLs on read; a filter inside a shared topic still ships data to the consumer.

It guarantees only what the broker has said it will have done before replying. For HarborCart's orders topic with acks from all in-sync replicas and minimum in-sync replicas of 2, the ack means the leader appended the event and at least one follower had it too, so one broker loss will not lose it. It does not say any consumer has seen it, that a consumer's database has applied it, or that the customer has been emailed. The timelines are different: a publish ack of perhaps 15 ms (an illustrative figure) covers network and replication, while a consumer might take 400 ms, and the second is none of the broker's business. An ack can also be lost on the way back, so the producer may have a persisted event and no confirmation, and a retry then publishes it again. Weaker settings change the claim: a leader-only ack survives nothing that kills the leader before replication, and an ack before the disk flush survives a process crash but perhaps not a power loss, depending on configuration. The correct reading is a sentence of the form 'this was durable up to this boundary under this failure model'.

• What is the producer state after a timeout? — Unknown: the event may be committed, not committed, or committed with a lost reply. It must retry with a stable id or record the uncertainty.
• Does acks=all mean exactly-once? — No. It governs durability on the broker, not duplicates or consumer effects.

One acknowledges individual deliveries and the other records a position. In RabbitMQ a consumer acks a delivery, and the broker may then delete that message; if the channel dies with unacked deliveries, those messages are redelivered. Publisher confirms are a separate mechanism, and the RabbitMQ documentation describes confirms and consumer acks as orthogonal: one says the broker took responsibility for the message, the other says a consumer finished with it. For quorum queues a confirm means a quorum of replicas accepted it. In Kafka-style systems a consumer reads a range of a partition and commits an offset meaning 'resume after here'; the committed offset is a watermark, so committing 4,202 implicitly says everything below it is done, whether or not each record really succeeded. That is why a slow record cannot simply be skipped and acknowledged later without a separate mechanism, and why a crash after processing but before commit replays a range. Per-message acks make selective redelivery natural but need the broker to track state for every in-flight message; offsets are cheap to store and make replay trivial but give up per-message tracking.

• Which is better for retries of one bad message? — Per-message acks, since one message can be requeued or dead-lettered; with offsets it takes a retry topic or a side table.
• Does a committed offset mean the order was processed? — Only if commit happens after the business transaction; commit means whatever the consumer chose it to mean.

The broker, not the workers, is the single arbiter of who holds a job. Workers A and B both ask shipment-jobs for work. The broker hands ship-HC-2048 to one of them and marks it leased or in flight, so the other receives a different job or none. The claim is made atomically inside the broker (a state change on a record or a delivery to exactly one consumer's channel), which means workers need no locks between them. The claim is a lease, not ownership: it hides the job for a visibility timeout (60 s, illustrative) and then lets it reappear if nobody has acknowledged it. There is a limit worth stating. The broker guarantees at most one active lease per job at a time; it does not guarantee that the previous holder has stopped. A worker that stalled for 70 s may still be running when Worker B starts the same job, so two workers can both perform the effect, and the effect needs to be idempotent. In a partitioned log the same arbitration is done by the group coordinator assigning each partition to one member.

• Can you size the worker pool? — Little's Law: 5,000 tasks/min is 83.33/s, times 0.3 s service time gives 25 busy workers, before headroom for bursts and retries.
• How is a lease extended for long jobs? — Heartbeat or renewal from the worker, with a maximum lease so a hung worker is eventually reclaimed.

The job comes back, and some of the work may already have happened. Worker A takes ship-HC-2048, books the label with the carrier, and crashes before acking. After the 60 s lease the broker makes the job visible again and Worker B takes it. In the case where A had not done anything yet, this is exactly the recovery wanted. In the case where A had already booked the label, B books a second one unless the carrier call carries an idempotency key such as the job id, or B first checks for an existing shipment. Each delivery also counts: after five delivery attempts, to follow the policy, the job goes to a dead-letter queue instead of looping. Two numbers matter when choosing the timeout. Too short and healthy slow jobs get duplicated; too long and a crashed worker's tasks are stuck for that long. A broker that detects a dropped connection, like RabbitMQ with consumer channels, can redeliver immediately instead of waiting for a timer, but a hung worker with an open connection still needs the lease.

• What if the worker acked early to avoid redelivery? — Then a crash after the ack loses the task: the broker deleted it and the label never got booked. Ack after the business transaction commits.
• Does redelivery preserve order? — No. A redelivered job can run after jobs that were enqueued later, so FIFO is not guaranteed with retries.

The event is delivered again, and a second payment is attempted unless something stops it. The consumer received evt-HC2048-01, called the payment provider, saw it succeed, and crashed before committing offset 4,201. When the group is rebalanced, the new owner of partition 7 resumes from the last committed offset, which is before 4,201, so the event is delivered to it. The broker is behaving correctly; its commit only ever described the consumer's own claim. There are three ways to deal with it. Make the effect idempotent by sending the provider an idempotency key derived from the event id, so the second capture request returns the first result. Record the event in an inbox table in the same database transaction as the local effect, and skip events already present. Or, for effects that cannot be made safe, put a human or a reconciliation job in front of any replay. Committing the offset before the effect makes the opposite trade: a crash then loses the payment instead of repeating it. Which failure is cheaper is a business decision, and for charging money it is almost always the duplicate, handled with a key.

• Can reordering the steps avoid this? — Not without a transaction spanning the consumer's store and the offset; storing the offset in the same database as the effect is the main way to get that.
• How does the inbox look? — A table keyed by consumer name and event id; the insert and the business write succeed or fail together, so a repeat hits the unique key.

They describe how many times an effect happens relative to how many times a message is delivered. At-most-once means no retries: commit before processing, and a crash drops the message. At-least-once means retry until acknowledged: process, then commit, and a crash repeats the message. Exactly-once is only meaningful with a scope attached. Inside Kafka, an idempotent producer deduplicates retried sends per producer session and partition using sequence numbers, and transactions let a read-process-write between Kafka topics and offsets commit atomically. Those are real guarantees within the broker. They say nothing about a row in HarborCart's Postgres, a call to a payment API, or a producer restart without a transactional id, because the broker cannot roll those back. So the honest statement is 'effectively once': at-least-once delivery plus idempotent effects, where the dedupe key lives in the same system as the effect. Anyone claiming exactly-once delivery across independent systems has hidden an inbox or a transaction somewhere, or is ignoring the failure that would expose it.

• Does acks=all give exactly-once? — No, it gives durability of the appended record, not absence of duplicates.
• Is at-most-once ever right? — For disposable signals like a heartbeat or a metric sample, where a lost one is cheaper than a repeated one.

No, because a broker transaction commits only things the broker owns. A consumer that reads an order event, writes a row to the order database, and then commits an offset in Kafka is touching two systems with two commit points. Kafka transactions make sense when the output is another Kafka topic: producing results and committing the consumed offsets form one atomic unit, so a crash replays the input and aborts the half-written output. If the output is a database row, a crash between the row commit and the offset commit produces a repeat, and the usual remedies are an inbox table keyed by event id in the same database, or storing the consumed offset in that database in the same transaction and seeking to it on startup. Two-phase commit across a broker and a database is possible in some stacks but couples availability of both and is rarely used for high-volume event flows. Idempotent writes are cheaper: an upsert keyed by event id is correct however many times it runs.

• What if the output is an HTTP call? — Pass an idempotency key, since an HTTP response can also be lost; rely on the receiver to deduplicate.
• What does isolation level on the consumer add? — Reading only committed transactional records avoids seeing aborted output, which matters for Kafka-to-Kafka pipelines only.

The charge request needs a stable key chosen before the first attempt, and the provider has to honor it. For order HC-2048 that key can be pay:HC-2048, derived from the order and not from the delivery. Every attempt, whether from a consumer retry, a rebalance or a human redrive, sends the same key, and the provider returns the original result for repeats. On HarborCart's side the consumer also keeps an inbox row for (payment-service, evt-HC2048-01) written in the same transaction as its own ledger entry, so a repeated event short-circuits before calling anyone. The two layers cover different failures: the inbox protects HarborCart's own state, the provider key protects the call whose reply may have been lost. If the provider offers no idempotency support, the only remaining defense is to look up whether a charge for that order already exists before creating one, which is racy under concurrency unless serialized per order, and which key-based partitioning does: all events for order:HC-2048 land on one partition and are handled by one consumer at a time.

• How long does the provider keep the key? — Providers bound it; the dedupe window must exceed the longest plausible retry or redrive delay, or the key should be recorded locally.
• What about a refund? — A separate operation with its own key, not a reuse of the charge key.

It removes the gap between writing the order and announcing it. Without it, checkout commits the order to its database and then publishes OrderPlaced; a crash between the two leaves an order that nobody downstream will ever hear about, and publishing first risks an event for an order that never committed. The outbox writes the order row and an outbox row in one local database transaction, so either both exist or neither does. A relay process reads unsent outbox rows, publishes them, and marks them sent. What remains is the relay's own ambiguity: it can publish successfully and crash before marking the row, and on restart it publishes the same event again. Outbox delivery is therefore at-least-once, with the same event id both times, and consumers or the idempotent producer deal with the copy. Two limits follow. The atomicity is local to the one database that holds the outbox, and cannot include other services. And ordering is only as good as the relay's read order and the key it publishes with.

• Why not publish inside the transaction? — A broker call inside a database transaction cannot be rolled back with it, and holds the transaction open on a network call.
• Could change-data-capture replace the relay? — Yes, reading the database log is another way to find new outbox rows; it has the same at-least-once property and adds its own operational dependency.

It is an identifier supplied with an operation so that repeating the operation is recognized and returns or reproduces the first result. For HarborCart the event id evt-HC2048-01 serves consumers, and pay:HC-2048 serves the payment call. Retention follows the longest gap between a first attempt and the last possible repeat. That window includes consumer retries over their five attempts, rebalance replays, a dead-letter redrive at up to 14 days (the illustrative DLQ retention), and a full replay of the three-day topic. A dedupe table that remembers keys for one hour will happily process a day-old redrive twice. The cheapest rule is to keep the inbox at least as long as the longest period in which the same event can be redelivered, and to make the effect naturally idempotent where possible, for example by upserting a row by order id, so that the store itself becomes the dedupe. Retention costs storage, so keys are usually pruned by age on a schedule, and anything that can be redriven past that age must not rely on the table.

• Can the key be a hash of the payload? — It can, but two legitimately identical events, like two identical purchases, would collapse; an explicit id is safer.
• What about an absent key? — Then no consumer can tell a retry from a new request; require the id at the producer's boundary.

A partition is a unit of both storage and parallel consumption, so more partitions allow more brokers to hold data and more consumers to read in parallel. In a consumer group each partition has at most one active owner, so the partition count caps the group's parallelism. With 24 partitions the average of 20,000 events/s is about 833 events/s per partition, roughly 0.81 MiB/s, and the 100,000/s promotion is about 4,167 events/s per partition. The key decides which partition an event lands on, and events with the same key stay in order relative to each other. HarborCart uses order:<orderId>, because the things that must be ordered are the events of a single order: OrderPlaced at offset 4,201 then OrderPaid at 4,202 on partition 7. A tenant-only key would be the wrong choice: if harbor-retail is 30% of traffic, 6,000 events/s hit one partition against an average of 833, about 7.2 times the uniform share, so one consumer is saturated while the rest idle. A good key has high cardinality, even spread, and matches the ordering the business needs. If no ordering is needed, a random or round-robin choice spreads best.

• What if a few customers are extremely heavy? — Keying by order already spreads one customer's orders; if one key itself is hot, salt it or give it a dedicated topic, accepting that order across the salted pieces is lost.
• Can the key be changed later? — Not in place. Existing events are already placed, so a new topic and a migration are needed.

No. It guarantees order within one partition, and nothing across partitions. Each partition is an independent append-only log with its own offsets, so event A on partition 3 and event B on partition 9 have no defined order to a consumer, even if A was produced first, because the two are read by different consumers at different speeds. A single partition would give a total order but caps throughput at one leader and one active consumer. Order inside a partition also has conditions. Producer retries can reorder unless idempotence is on, which Kafka allows with up to five in-flight requests per connection while still preserving order. A consumer that processes records of one partition concurrently on several threads loses the order unless it routes by key. And redrive from a dead-letter topic puts a failed event after later ones. In a work queue with several workers and redelivery, strict FIFO also disappears. So the usable statement is 'ordered per key, assuming one writer path and one sequential handler for that key'.

• Is there a way to get a total order? — One partition, or a separate sequencer that assigns numbers, accepting its throughput limit and a single point of coordination.
• How do downstream systems cope with cross-key order? — They carry versions or event times and apply updates only if newer, rather than relying on arrival order.

Key placement changes, and so does the ordering scope that was relying on it. With hash-based assignment the partition is roughly the hash of the key modulo the partition count, so raising 24 partitions to 48 moves many keys. Events for order:HC-2048 written before the change sit on partition 7; events written after may land on a different partition. A consumer reading the two partitions in parallel may now process OrderPaid before an older OrderPlaced, and the per-key order guarantee is broken across the boundary. Old data is not rebalanced; it stays where it is. Decreasing the count is generally not supported. The safe approaches are to size partitions with headroom at creation, to expand after draining the backlog of the keys that will move, or to create a new topic with more partitions and migrate. The capacity argument for expansion is concrete: the promotion needs 100,000/s at an assumed 2,000/s per consumer, so 50 consumers, which 24 partitions cannot use, while 48 partitions would top out at 96,000/s.

• Does adding partitions help existing backlog? — Only for newly produced data; the old partitions still hold their backlog and are read at their old parallelism.
• Is consistent hashing a fix? — It limits how many keys move, but a log's existing records still stay where they were written.

A group coordinator on the broker gives each partition to exactly one member of the group, and recomputes the assignment when members join, leave or are judged dead. Twelve consumers share 24 partitions two each. Each reads its partitions sequentially, so a member sees every event for its keys in order. A thirteenth to twenty-fourth member adds parallelism; a twenty-fifth sits idle because no partition is free. The unit of independence is the group, not the instance: email-notifier, sales-analytics and fraud-monitor each have their own assignment and committed offsets over the same 24 partitions. When one of 12 instances dies, 24 partitions over 11 survivors is 2 each with 2 left over, so two survivors take 3 partitions under a full rebalance. A sticky or cooperative strategy moves only the two orphaned partitions and leaves the other 22 alone. Per-instance load changes too: at 20,000/s each of 12 instances sees about 1,667/s, and with 11 instances the three-partition owners see about 2,500/s, over the hypothetical 2,000/s capacity, a skew capacity planning needs to anticipate.

• Can two groups share offsets? — No; offsets are stored per group, per partition.
• What is the benefit of cooperative rebalancing? — It avoids stopping every consumer while ownership is redistributed, but it does not remove the duplicate-processing window.

Assume the old owner is still running and the new owner will start the same record. Consumer 5 holds partition 7 and is half-way through a 90-second report for evt-HC2048-01 when it exceeds the maximum poll interval. The coordinator moves partition 7 to consumer 9, which resumes from the last committed offset and processes the same event. Both are now live; the group generation moved from 12 to 13, and consumer 5 carries the stale generation. Kafka-style brokers reject a commit from a member with a stale generation, so consumer 5 cannot move the offset. That protects the offset, but not the side effect: consumer 5 may still write to the database. The defences stack. Keep per-poll work short, or hand long tasks to a work queue with leases and heartbeats instead of holding a partition. Pause partitions while background work runs and keep polling. Make the effect idempotent with an inbox. And for writes that must not interleave, a fencing token such as the generation or a version number, checked by the database, rejects the stale writer.

• Why not just raise the timeout? — Longer timeouts delay detecting a truly dead consumer, so a failure leaves a partition idle for that long.
• Where should a 90 s job live? — In a queue designed for long jobs, with a visibility timeout and renewal, rather than on a stream partition.

An offset is the sequence number of a record within a partition, assigned by the broker when the record is appended. Offset 4,201 on partition 7 is OrderPlaced; 4,202 is OrderPaid. A consumer's position is where it will read next, held in memory. A committed offset is the position the consumer chose to save durably, in Kafka stored in an internal topic keyed by group, topic and partition, so that after a restart or reassignment the next owner starts there. The convention is that a committed offset N means 'resume at N, everything before it is done', so committing 4,202 says 4,201 was handled. Committing is a promise the consumer makes, and the broker has no view of whether the database write happened. Commit before processing gives at-most-once, after processing gives at-least-once, and for HarborCart commit follows the database transaction. Committed offsets are not kept forever: in Kafka they are removed after a group has been inactive for the offset retention period (seven days by default in current source), so a group that returns after a long absence may start from earliest or latest per its reset policy.

• What if the committed offset is older than retention? — The offset is out of range and the consumer's reset policy decides: start at earliest retained, start at latest, or fail loudly.
• Is committing every record sensible? — It adds a write per record; batch commits after each processed batch, accepting a larger replay window on crash.

A consumer can move its position to any offset still retained, so replay is a seek followed by normal reading. If the analytics team finds a bug in its revenue aggregation that corrupted the last thirty minutes of results, it can reset the sales-analytics group to the offsets found by timestamp, for example the first offset after 09:00 on each partition, and reprocess. Because the log is unchanged, that group's replay does not affect email-notifier or fraud-monitor. Several things make replay safe or unsafe. The target must handle the same events twice, since earlier results are already applied. Side effects such as emails or payment captures must be disabled or guarded, or the replay repeats them, so payment topics are never replayed blindly. The window is bounded by retention of three days, so replaying a week-old incident needs an archive. And replay creates load: re-reading 36 million events at a recovery rate of 50,000/s while 20,000/s keep arriving drains at 30,000/s net, about 20 minutes if rates hold. Authorization applies too, since replay can expose data that the original consumer saw only under access rules of that time.

• How do you find the right offset? — Brokers can map a timestamp to an offset per partition; record timestamps depend on producer or broker settings and clocks.
• Does replay reorder anything? — Within a partition no, but the replaying group runs alongside the live tail, so downstream state needs version checks.

Retention deletes old data by age or size, and compaction keeps the latest value per key. With the orders topic on cleanup policy delete and three days' retention, segments older than three days become eligible for removal whether or not anyone has read them, and a group that has not caught up loses its unread data. Compaction is for topics that represent current state per key: the log keeps at least the latest record for each key, and older records with the same key are eventually removed. A tombstone, a record with a null value, marks a key as deleted and is itself retained for a configured period before cleanup. A compacted topic therefore is not a full history: a consumer that starts late sees the latest value per key and misses intermediate values, and removal happens in the background, not immediately. HarborCart's order events keep history, so retention applies; a customer-profile topic that only needs the current profile would be compacted. A deletion request on a compacted topic is satisfied only after the tombstone is written and cleanup has run.

• Does compaction preserve ordering? — Surviving records keep their relative order and offsets; skipped offsets just disappear.
• Could orders be compacted? — Only if consumers want the latest state per order and can live without the transition history.

Because in a log, consuming is a cursor movement and not a deletion. When all three groups have passed offset 4,201, the record is still there, and it stays until retention removes it, three days for orders. This is deliberate. It lets a slow or new subscriber catch up, it permits replay after a bug fix, and it decouples the retention period from the slowest consumer. The costs are storage and governance. At the average load, three days is about 4.83 TiB of logical payload, and with three replicas roughly 14.48 TiB, before indexes, overhead and headroom. And personal data in events stays for the whole period, so what goes into a payload and how a deletion request is honoured both need a decision up front: minimize sensitive fields, encrypt, and for erasure use compaction with tombstones or crypto-shredding instead of editing an immutable log. A work queue behaves differently, since a classic queue deletes a job once acknowledged; what remains there are dead letters, unacknowledged jobs, or retained history in a stream.

• Is 'retain forever' reasonable? — It is a decision with a cost and a compliance profile; long-term history is usually archived to object storage, not kept on brokers.
• Who decides retention when a topic is shared? — Retention is per topic, not per consumer, so the longest need decides it.

A leader ack says one broker has the record, and a quorum or in-sync ack says several do. In Kafka terms, producing with acks=1 returns once the leader has appended, so a leader crash before the followers fetch the record loses an acknowledged event, and the follower that takes over never had it. With acks=all the leader waits for the in-sync replicas, and with minimum in-sync replicas set to 2 it refuses the write if fewer than two replicas, counting itself, are in sync. HarborCart sets replication factor 3, minimum in-sync 2, acks from all, and unclean leader election off. An acknowledged evt-HC2048-01 then exists on the leader and at least one follower, and an in-sync follower can take leadership with the record intact; the leader epoch moves from 7 to 8. Raft-style systems such as RabbitMQ quorum queues express the same idea differently: a message is confirmed when a majority of replicas, floor(N/2)+1, accepted it. The cost of the stronger ack is latency, since the slowest required replica sets the time, and availability, since too few replicas block writes.

• What does unclean election do? — It lets an out-of-sync replica become leader when none in sync is available; availability returns, but the committed suffix it lacks is lost.
• Does the ack cover a power loss? — That depends on flush settings and whether replicas fail together; replication across failure domains is what covers correlated loss.

One broker, for writes, under minimum in-sync replicas of 2. With three replicas and minimum 2, the leader and one follower can still form the required set after one failure, so producers with acks from all continue. If a second broker is lost, the remaining replica alone is 1 in sync, below 2, and the broker rejects writes with a not-enough-replicas error. The committed data still exists on the survivor, so reads of what was committed can continue, but write availability is gone until a replica returns. The statement 'three replicas survive two failures' is true only for data durability in the sense that a copy remains, and only if that survivor is still in the in-sync set; it is not true for continuing to accept writes. Setting minimum to 1 would keep writes alive at the price that an acknowledged record could be on one machine, and combined with unclean election could be lost entirely. Failure domains matter as much as counts: three replicas on one rack or one availability zone fail together, so placement across zones is part of the claim. A Raft quorum queue with three members behaves similarly, with a majority of two required to commit.

• What if you need to survive two losses with writes? — Five replicas with a majority of three, paying more replication traffic and latency.
• What do producers see when this happens? — Errors or timeouts after retries; they need a bounded buffer and a decision between failing the request and holding it.

First classify the failure, then bound the retries, then move the message out of the way. A poison message fails every time, for instance an OrderPlaced whose JSON no longer matches what the consumer expects. In a partitioned log a consumer that retries in place stops advancing on that partition, so everything behind it, including healthy orders, is stuck. The consumer therefore distinguishes transient errors, like a timeout calling a database, from permanent ones such as a deserialisation failure or a validation rule. Permanent failures go straight to a dead-letter topic with the original payload, headers and error reason, and the offset advances. Transient ones retry with exponential backoff and jitter, 2 s, 4 s, 8 s, 16 s in HarborCart's illustrative policy, up to five attempts, and then also go to orders.dlq. Retrying without a delay produces a hot loop that may overload the dependency that was only briefly unavailable. For ordered per-key processing there is a catch: parking one order's event while later events for the same order continue breaks that order's sequence, so a per-key pause or a retry topic with the same key is chosen deliberately. A work queue handles the same thing with a delivery-attempt count and a dead-letter queue.

• Where does the retry wait happen? — In a separate delay or retry topic, or the broker's delayed redelivery, so the main partition keeps moving; sleeping in the consumer holds the partition.
• What stops a retry storm across many consumers? — Jitter, a global retry budget, and a circuit breaker on the failing dependency.

It helps when somebody owns it, and it hurts when it becomes a place where messages go to be forgotten. A dead-letter queue isolates failing messages so the rest of the stream flows, and keeps them with their error for diagnosis and redrive. It is dangerous in a few specific ways. Nobody watches it, so a silent rise to thousands of parked orders is a data loss that dashboards report as healthy throughput. Parking does not prove the side effect did not happen: a consumer may have charged the card before failing on the next step, so a parked payment event may already be partially applied. Redrive can repeat a payment capture if effects are not idempotent, which is why redrive for payment-capturing topics at HarborCart is human-reviewed. Redrive also reorders: a redriven event arrives after newer events for the same order, so strict ordering guarantees do not cover it, and handlers should use versions. Finally, a DLQ has its own retention, 14 days here, and expiry there is a permanent loss. Alerts on DLQ depth and age, an owner, and a documented redrive procedure make it useful.

• Should every failure go to the DLQ? — No; transient failures should retry, and a systemic outage would flood the DLQ with healthy messages that only need time.
• How do you redrive safely? — Fix the cause, replay a small sample, rely on idempotent effects or a reviewer, and rate-limit the redrive so it does not become its own burst.

Use offsets as the primary measure and treat timestamps as a supplement with a stated error. Offset lag is the log end offset minus the group's committed offset for each partition, and it needs no clocks, but a count of 10,000 means different things at 800 events/s and at 8 events/s. Time-based lag is more meaningful to the business: oldest unprocessed event age, or the gap between now and the timestamp of the record being processed. That needs a trustworthy timestamp. A producer's eventTime comes from its own clock, so a producer host running two minutes fast makes lag look two minutes smaller, and a slow one makes it larger; a log-append time assigned by the broker is consistent across producers, but excludes any delay before the producer published. The broker clocks can differ too. So measure oldest age using the broker's append time, keep the producer's event time for business logic, alert on the trend, and track both clocks' offset from a reference such as NTP. Lag that reads negative, or jumps by minutes when a host restarts, is evidence of clock error and not of consumer behavior. A stalled consumer with zero throughput and a growing offset gap is the most reliable signal of all.

• Can lag be zero while events are lost? — Yes, if the producer never published them or retention deleted them first; lag only covers events that reached the log.
• What other signals complement lag? — Throughput of the group, rebalance frequency, and the age of the oldest retained unread record.

Work from the load to each constraint separately, since the binding one differs. For storage, 20,000 events/s of 1 KiB is 19.53 MiB/s, which is 1.609 TiB a day and 4.83 TiB over three days of logical payload. Replication factor 3 makes that 14.48 TiB of replicated payload, and the real disk figure is higher because of indexes, segment overhead and headroom, so that number is a floor, not a purchase order. Bandwidth at the broker is the inbound 19.53 MiB/s, two followers fetching 39.06 MiB/s, and three groups reading 58.59 MiB/s, roughly 97.66 MiB/s of broker-side egress on average and 488.28 MiB/s at the 5× peak, before protocol overhead and compression. Partitions come from consumer parallelism: with a hypothetical 2,000 events/s per consumer, the average needs 10 consumers per group and the peak needs 50, so 24 partitions cap the group at 24 active consumers and 48,000/s, and 48 partitions reach 96,000/s, still below 100,000/s. The choices are 50 or more partitions, faster consumers, or accepting that lag builds during the promotion. An outage of 30 minutes at the average rate is 34.33 GiB, 36 million events, already inside the retained log; draining at 50,000/s against 20,000/s arriving takes 20 minutes. The per-consumer figure has to be measured, not assumed.

• What changes if the average payload is 10 KiB? — Every byte-based figure scales tenfold while event-based figures stay the same, so storage and bandwidth dominate over partition count.
• Is there a safe per-partition throughput? — No universal number; it depends on message size, batching, hardware and consumer work, so it must be measured in the actual environment.

Use separate controls for the two problems, because they fail differently. A noisy tenant consumes shared capacity: its publish rate, bytes, connections or consumer fetches crowd out others. The remedy is per-tenant quotas enforced at the broker or an ingress layer, identity from the authenticated principal and never from a tenant_id field in the payload, and fair scheduling or throttling when a tenant exceeds its allotment (5,000 events/s for harbor-retail, illustratively, answered with a throttle or rejection and not unbounded buffering). A hot partition is a distribution problem inside the topic: one key or a skewed key function sends too much to one partition, so that tenant-wide quota may look fine while one consumer is saturated. Detect it from per-partition rate and lag, not topic-wide averages. Fix by choosing a better key (order:<orderId> instead of tenant), salting a very heavy key, or isolating a heavy tenant on its own topic or cluster. The salt has a price: ordering across the salted pieces disappears. ACLs complete the picture so Meridian cannot publish to or read harbor-retail's topics, since a quota without authorization only limits how fast someone can read data they should not see.

• What do you do if one order key is itself hot? — That order's events must be serialized, so reduce the work done per event or split the aggregate; adding partitions does not help one key.
• Should the broker or the gateway enforce quotas? — Broker-side quotas cover every client path; a gateway adds policy and per-tenant billing context.

Make the state update depend on information carried by the event, not on the order of arrival. Within one key and partition, order is kept, but across partitions, across retries and during redrive, a Canceled event may arrive before the Placed it follows. Common tools are a version or sequence number per entity so a handler ignores anything not newer than what it holds; a state machine that rejects impossible transitions and parks the event briefly or stores it pending; and upserts that apply last-writer-wins by event version, not by wall clock. Event time and processing time are different. Aggregations over time windows use the original eventTime and a lateness allowance, say a few minutes, after which late events either update the closed window with a correction or go to a side output, which is a business choice. Producer clocks are not reliable enough for strict ordering decisions between machines, so the entity version comes from the system that owns the entity. For an order, the order service increments a version with each state change and puts it in the event, and analytics can then reconstruct the sequence even if delivery is shuffled.

• Can the broker fix this? — Only partially, by keying so one entity's events share a partition; cross-source ordering is not a broker property.
• What if versions are absent? — Then the consumer must accept that out-of-order delivery can corrupt state, and a periodic reconciliation against the source of truth is the backstop.

At each stage it is a bounded resource that fills up, and a rule about what happens when it is full. At the producer, a send buffer holds events waiting to be batched; when the broker is slow or unavailable the buffer fills, and the client must either block the caller for a bounded time or fail fast. The request path then sees an error and can shed load or return 'accepted, pending' only if it has somewhere durable to put the event, which is exactly what an outbox is. At the broker, disk, network and request queues are the limits, and quotas throttle a heavy client by delaying its responses; a full disk is the harshest form, because the broker stops accepting writes, so retention, quotas and disk alerts have to act well before it. At the consumer, backpressure is the poll loop: a consumer that stops polling slows intake and lets lag grow, which is acceptable only because the log retains data. The hazard is an unbounded buffer anywhere, where the pressure disappears and memory grows until a crash. Retries add load to the stage that is already struggling, so retry budgets and jitter belong with backpressure. When 100,000 events/s arrive and consumers can do 48,000/s per group, the log is the shock absorber, and lag is the visible cost.

• When should producers reject instead of buffer? — When the buffer's age would exceed the event's usefulness or when memory is bounded; reject early and let the caller decide.
• Does batching help backpressure? — It cuts request overhead, 100,000 requests/s becomes 1,000 with batches of 100, but each event may wait for the linger interval, so latency can rise.

Through compatibility rules checked before a new schema is accepted, and through changes that respect them. A schema registry stores each version of OrderPlaced and refuses a new version that violates the chosen mode. HarborCart starts from backward compatibility: consumers using the new schema can still read data written with the previous one, which allows deleting a field or adding a field with a default, depending on the format's rules. That mode implies consumers are upgraded before producers, because old consumers are not protected from new data. Since upgrades happen in some order, it matters whether consumers or producers move first, and when old consumers must keep reading new events the registry should enforce a full (both directions) mode instead. Adding an optional field with a default is usually safe in both directions; renaming a field, changing its type, or making a new field required is not. A v2 to v3 change that removes a field old consumers still require passes a backward-only check but breaks them, and under a full mode the registry rejects it, so the change is made as a new field, a dual-write period, then removal after all consumers have moved. Events carry a schemaVersion, as OrderPlaced does here at 2, so a consumer can choose a decoder. Retained data complicates things: with three days in the log, or any longer archive, old versions are read for a long time, so a consumer must support every schema version still present. Unknown fields should be ignored and not treated as errors.

• What is the difference between backward and forward compatibility? — Backward lets new readers read old data; forward lets old readers read new data. Which you need depends on the upgrade order.
• What if a breaking change is unavoidable? — Create a new topic or event type, run both, migrate consumers, then retire the old one.

Decide per topic whether the data may leave its region, and build the recovery plan from that classification. HarborCart runs two independent fabrics, EU and US. A residency-tagged topic such as orders.eu is never mirrored across the border, so if the EU region goes down, EU orders cannot fail over to the US; the safe behaviors are to degrade, queue at the edge within the region, or wait. Non-resident topics can be mirrored asynchronously. Asynchronous mirroring means the secondary lags, so on failover the promoted copy may be missing the most recent suffix, and consumers must either tolerate gaps or reconcile from the source of truth. Offsets are not the same across clusters, since each log assigns its own, so a consumer moving to the mirror has to translate positions through a mirroring tool's offset mapping, or restart from a timestamp and rely on idempotency to handle the overlap. Failback is another event: after the original region returns, the data written during the outage has to be reconciled in the right direction. What cannot be promised is unconditional cross-region failover under residency rules, and a synchronous replication across a distance costs write latency on every publish.

• Could an EU event be replicated to the US encrypted? — Whether that satisfies residency is a legal question as well as a technical one; the design should not assume it does.
• What is tested before it matters? — A scheduled failover exercise on non-resident topics, and a documented degrade mode for the resident ones.

The signals that expose it are the ones measured at the end of the pipeline, compared with the ones at the start. Counts of published events and counts of processed events by group should match over a window; a gap means loss, stalls or filtering. Oldest unprocessed event age per group catches a stuck consumer even when the offset gap is small, and consumer lag by partition shows a single stuck partition that an average hides. Throughput of each group dropping to zero while lag rises indicates a stall, and rebalance frequency indicates a group that is thrashing and making little progress. On the producer side, publish outcomes by category, with a counter for unknown outcomes after timeouts, show where the system is relying on retries; outbox rows older than a threshold show a relay that has stopped. Dead-letter depth and age show messages parked. Under-replicated partitions and the in-sync set size show durability eroding. Tracing ties these together: carry the W3C traceparent in message headers so a consumer's process span can be linked to the producer's span (see the OpenTelemetry messaging semantic conventions). An end-to-end canary event, published every minute and checked at each group, is the most direct test of the whole path. Where possible, metrics should not travel only through the cluster they report on, or an outage silences its own alarm.

• What is the first alert to set? — Oldest unprocessed age per consumer group above its business tolerance, because it covers many different failures with one symptom.
• How do you avoid alert noise from planned catch-up? — Alert on the age and the trend of lag against capacity to drain, not on a fixed count.

A managed queue is enough when the problem is background jobs, nothing needs to be replayed, and each message has one consumer. Sending confirmation emails, resizing images or running a shipment job at a few hundred per minute needs durable storage, a visibility timeout, retries and a dead-letter queue, and a managed service provides those without anybody running brokers, partitions or rebalances. It stops being enough when several independent systems need the same event stream and each at its own pace; when new consumers should be able to start from history; when ordering per key at high throughput is needed; or when a bug fix has to be re-applied by reading old events again. Then a retained log is the better fit, with the extra operational cost of capacity planning, partition count, rebalances and retention. There are middle paths: a managed queue with a topic in front for fan-out, giving one queue per subscriber, or RabbitMQ streams for long backlogs. Neither option removes duplicates; a standard managed queue is at-least-once and still needs idempotent handlers, and ordered variants restrict throughput and have a bounded deduplication window documented per service. The honest comparison is on contract and operating burden, and the cheaper system that meets the contract is the right one.

• What about an outbox with a managed queue? — Still valuable: it makes the database write and the publish consistent, regardless of which broker follows.
• Is a log needed for a single consumer? — Not unless replay or ordering at scale matters; a queue is simpler.

Start with the contract, build the simplest thing that keeps it, and let each failure justify the next component. The contract covers who publishes, who consumes, what must survive a crash, and what may be duplicated or reordered. The first version is HarborCart's synchronous checkout, and its failures, a slow email service blocking an order, motivate a durable queue. Then comes a truthful acknowledgment: the producer is told 'accepted' only after replication under the stated policy, and a lost ack means a retry with the same id. Work like shipment-jobs uses competing workers with leases and a retry limit. Independent readers lead to a topic and a retained log with a cursor per subscriber. Throughput leads to partitions keyed by order:<orderId>, and consumer groups. Duplicates lead to inboxes and idempotent effects, and the dual write leads to an outbox. Poison messages lead to bounded retries and a DLQ. Broker loss leads to replication factor 3, minimum in-sync 2 and no unclean election. The last ten minutes cover quotas and ACLs, a schema registry, lag and oldest age, regional boundaries and the guarantees the design does not make. What is left out deliberately is global ordering, exactly-once across systems, replay past retention and write availability after two broker losses.

• What if only 20 minutes are available? — Contract, durable queue and ack semantics, partitioned log with groups, then idempotency; mention the rest as known extensions.
• What do you cut if the system is small? — Partitions, groups and the schema registry are the first to go, as one queue and an inbox table already meets many needs.