Distributed Message Queue Failures: Lost Acks, Duplicates & Recovery

Forty ways a message platform goes wrong, from a lost acknowledgment to a missing region, each told as an incident with a clock, a cause, a recovery and the guarantee that survives. The pattern across all of them is that a message passes through several states, and every failure lives at a boundary between two of them.

Every scenario here is hypothetical. They use the fictional company HarborCart (commerce), order HC-2048 with event evt-HC2048-01 on topic orders partition 7 at offset 4,201, a shipment-jobs work queue, consumer groups email-notifier, sales-analytics and fraud-monitor, and a second tenant meridian. The numbers (20,000 events/s average, 100,000 events/s for a ten-minute promotion, 1 KiB events, 24 partitions, replication factor 3, three days of retention) are teaching inputs rather than measurements of any real system.

This page can be read on its own, so the vocabulary first. A producer publishes events; a consumer reads them. A queue hands each message to one worker; a topic delivers each message to every subscription, and a retained log keeps the messages for a while so they can be read again. A partition is one ordered slice of a topic, an offset is a position inside it, and a consumer group shares a topic's partitions among its members so that each partition has one active reader. A leader replica accepts writes for a partition and followers copy it; the in-sync replicas (ISR) are the followers that are keeping up. A lease or visibility timeout hides a claimed job from other workers for a while, and an acknowledgment or offset commit tells the broker the consumer is done. Idempotency means repeating an operation has the same effect as doing it once; a DLQ parks messages that cannot be processed.

Where an event can be, and what each state proves

A published event moves through distinct states: the producer issued it, the broker received it, the leader appended it, the configured replicas hold it, the producer received an acknowledgment, the consumer was handed it, the business effect committed, and the consumer's acknowledgment or offset commit was recorded. These are never the same fact. When the broker acknowledges a publish, it says nothing about whether a consumer has done anything; when a consumer commits an offset, it says nothing about whether a payment was captured unless the code orders them correctly. Most incidents below are the result of treating two adjacent states as one.

What no design on this page can promise

Some guarantees are unavailable at any price, and the incidents keep returning to them: a single global order across a partitioned topic, an effect applied exactly once across systems that share no transaction, replay beyond the retention window, automatic cross-region failover for data that residency rules keep in place, and continued writes after losing more replicas than the policy tolerates. With replication factor 3 and a minimum of two in-sync replicas, one broker can fail and writes continue; two cannot, and writes are refused rather than risk silent loss.

How each incident is written

Each incident starts with the story and the arithmetic where numbers help, then closes with a table listing trigger, timeline, impact, root cause, detection, containment, recovery, prevention, the guarantee that still holds and the tradeoff it costs, plus the invariant (I1 to I12) it touches. The invariants are the ones used across this module: identity comes from the verified principal (I1), a durable acknowledgment follows the advertised persistence (I2), publish, replicate, deliver and effect are separate facts (I3), ordering holds only within a key's partition (I4), there is no free exactly-once (I5), retries are bounded and poison is quarantined (I6), everything is bounded (I7), committed data survives only the failures the policy covers (I8), replay is bounded by retention and authorization (I9), outbox atomicity is local to its database (I10), personal data is minimized (I11) and the key signals are visible (I12).

The map

The first four incidents follow the order in which HarborCart's design changes: a synchronous checkout that fails because of an optional dependency, a memory-only queue that forgets, a publish whose outcome nobody knows, and an acknowledgment that promises more than the broker has done.

F01 — Synchronous email timeout fails checkout

In HarborCart's original design the checkout handler calls payment, inventory, email and analytics one after another and returns only when all four answer. During a promotion the email provider starts answering in about 8 s instead of the usual few hundred milliseconds. Payment has already captured the money for HC-2048 by then, so the customer sees an error page for an order that was in fact paid.

The arithmetic explains why a slow dependency becomes an outage. Say checkout receives 100 orders/s (a hypothetical promotion rate) and each request now holds a thread for 8 s: Little's Law gives 100 × 8 = 800 requests in flight, against a pool of 200 threads. The pool is full within about two seconds, new requests queue behind it, and the payment and inventory calls that were healthy start timing out purely from waiting. Customers retry, which adds load and risks a second charge unless payment is idempotent. Nothing about email was ever important enough to block a purchase; the chain simply made it so.

Moving email and analytics behind a queue or topic removes them from the request path, but it does not remove the failure; it relocates it. The email consumer can now fall behind quietly, which is why lag and oldest-message age become the signals to watch.

F02 — Process crashes with queued work in RAM

The first fix HarborCart tried for F01 was an in-process queue: checkout drops a shipment task into a list in memory and a background thread drains it. That is fast and simple, and it is gone the instant the process is. A node is replaced during a routine deploy while 3,000 shipment tasks are waiting.

At a hypothetical working rate of 83.33 tasks/s (300,000 an hour), 3,000 tasks is about 36 s of backlog. Paid orders exist in the database; the shipment jobs that should follow them do not. Nobody receives an error, because the failure is an absence: no exception, no retry, no log line, and the customers are simply not shipped. Discovery comes from support tickets days later unless a reconciliation job compares paid orders to shipments.

A durable queue changes who owns the task. Once the job is stored outside the process, a crash only interrupts a worker, and the leased task becomes available again after its timeout. The remaining question is whether the producer knew the task was stored, which is what F03 and F04 are about.

F03 — Producer write succeeds but the acknowledgement is lost

The order service publishes OrderPlaced (event id evt-HC2048-01) to orders. The leader appends it at partition 7, offset 4,201, replicates it, and sends the acknowledgement; the TCP connection resets and the response never reaches the producer. From the producer's side there is only a timeout, and a timeout is a statement about its own knowledge, not about the broker's state.

There are exactly two worlds behind that timeout. In one the record never arrived and no event exists; in the other it is committed and the answer was lost. The producer cannot tell them apart by waiting longer. If it gives up, the order may have no event (the worse case for downstream consumers); if it retries blindly, the log may contain two copies. Within a single producer session an idempotent producer (a client feature that attaches sequence numbers per partition so the broker can drop a repeated batch) turns the retry into a no-op, but only while that session lives.

The robust answer is to treat the outcome as unknown and make both outcomes safe: keep the event in the outbox until a confirmed acknowledgement, resend with the same event id, and let consumers deduplicate on (consumer, event_id).

F04 — Broker acknowledges a memory-only write, then crashes

During a throughput experiment someone sets the shipment-jobs queue to acknowledge as soon as the broker has received the message, with no replication wait and no flush. Publish latency improves visibly. A node crash then reveals what the acknowledgement actually meant.

At 20,000 events/s, a 50 ms window of unflushed data holds 20,000 × 0.05 = 1,000 events (50 ms is a hypothetical flush interval). The broker answered 'got it' for all of them, the producers moved on and marked their outbox rows dispatched, and the crash erased the buffer. Because the producers saw success, nothing retries and nothing alerts. Only a later count of expected versus consumed events shows the gap.

The mistake is not that memory is fast; it is that the acknowledgement claimed more than had happened. A broker acknowledgment can mean any of four different things: received, appended to the log, replicated, or flushed to disk, and a sound design makes each ack name the step it covers. HarborCart's canonical setting waits for the in-sync replicas, so an acknowledged event has already survived a single-broker crash.

A replicated log is only as safe as the rule that says when a write is committed and which replica may lead after a failure. These four incidents sit on the boundary between acknowledged, committed and available.

F05 — Leader fails after replication but before the ack

Broker L leads partition 7 at leader epoch 7. A producer sends OrderPlaced for HC-2048; L appends it at offset 4,201, both followers copy it, and the record is now committed, meaning the in-sync replicas all hold it. L then loses power before it can answer. The controller elects F1, epoch 8, and the record survives on the new leader.

The producer experiences the same silence as in F03 and cannot know any of this. When it retries to the new leader, one of two things happens. If the retry comes from the same idempotent producer session and the sequence state is recoverable from the replicated log, the new leader recognizes the batch and does not append it again. If the retry comes from a freshly started producer, or the relay marked nothing and a new process picks up the row, the log gains a second copy at a later offset. Both outcomes are consistent with at-least-once publishing.

This scenario matters because it shows the commit point and the acknowledgement are different moments. Replication being complete protects the data; the ack only informs the sender. Durability and knowledge can part ways for a few hundred milliseconds, and the design must tolerate that gap.

F06 — Leader fails before the record reaches the in-sync replicas

Now invert F05. L appends the record at offset 4,201 to its own log, but the followers are still at 4,200 when L crashes. Because the producer asked to wait for the in-sync replicas, no acknowledgement was sent, so the producer never believed it was stored.

F1 becomes leader with its log ending at 4,200. The next event published to partition 7 now receives offset 4,201 on F1's log, which is a different record from the one L wrote. When L restarts it discovers that its tail diverges from the leader of epoch 8 and truncates its extra record before following. The lost record was never acknowledged, so the contract holds, but only if the producer retries. A fire-and-forget producer would have lost an order event without anyone noticing.

Truncation of an uncommitted tail is the mechanism that keeps replicas identical, and it is the reason consumers must read only up to the committed point (the high watermark). A consumer that could see offset 4,201 on L before commit would have read a record that later disappears.

F07 — In-sync replica set shrinks below the required minimum

orders is configured with replication factor 3 and a minimum of 2 in-sync replicas (the leader counts toward that minimum). A network problem makes F1 and F2 miss their fetch deadlines, and the leader removes them from the in-sync set. The in-sync set is now just L, one replica, below the minimum of two.

With acknowledgement on in-sync replicas, the leader refuses writes (the Kafka-style error is NotEnoughReplicas) rather than acknowledge something it cannot protect. Data already committed is still there and still readable; what has been lost is the ability to accept new writes on those partitions. If the producers retry for the default delivery timeout of 120 s, the order service keeps committing orders and the outbox grows: 20,000 events/s × 120 s = 2,400,000 rows in two minutes, a number worth having in mind before the relay is asked to catch up.

This is the boundary of the guarantee, not a bug. Replication factor 3 with a minimum of 2 survives one broker loss with writes continuing. Two losses leave committed data on the survivor but stop writes. The alternative, accepting writes on a single replica, would turn the next failure into data loss.

F08 — Leader elected from an out-of-date replica

Unclean leader election is disabled on orders. In the middle of an incident, with L and F1 both down, an operator sees partition 7 offline and enables it temporarily so that service resumes. F2 had been lagging by 1,200 records, so it becomes leader with a log that ends 1,200 offsets earlier than the committed end.

Writes resume at once, which looks like success. Every one of those 1,200 records had been acknowledged to producers, and they now exist only on the offline brokers. When those brokers return they must truncate to follow the new leader, so the records are erased. Consumers whose committed offsets sit beyond the new end find themselves holding positions that no longer exist; depending on policy they reset to earliest or latest, and either one can skip or reprocess data.

This is the cleanest example of availability bought with durability. The setting is not wrong in every context (a metrics topic may accept it), but for orders the cost is invisible loss of acknowledged events. The recoverable path is to bring a complete replica back, even if it takes minutes longer.

Retrying a publish is how producers survive a broken network, and it is also how duplicates and reordering get in. The outbox solves a different gap, between the order database and the broker, and has its own duplicate.

F09 — Publisher retry duplicates an event

HarborCart's outbox relay reads evt-HC2048-01, publishes it, and the batch is appended at offset 4,201. The relay's call times out locally, the relay process exits on a health-check failure, and the replacement starts a new producer. It reads the same outbox row, still not marked, and publishes again; the log now holds the event at 4,201 and again at some later offset.

The broker did exactly what it was asked. An idempotent producer protects a single session (a producer id plus its epoch, with per-partition sequence numbers), so a new process is a stranger to the broker. Application-level resends with new identifiers are also invisible to it. The protection belongs to the layer that knows the business identity of the event: the consumer's inbox keyed by (consumer, event_id).

A common reaction is to chase an exactly-once publisher. Transactions in a log can make read-process-write cycles atomic inside the log, yet they do not extend to a payment API or a mail provider. The honest design accepts duplicates and removes their effect.

F10 — Retries reorder events within a partition

The order service sends OrderPlaced and OrderPaid for HC-2048 with key order:HC-2048, so both go to partition 7. The producer allows several batches in flight and has idempotence turned off. The batch holding OrderPlaced fails with a transient error, the one holding OrderPaid succeeds at offset 4,201, and the retried OrderPlaced lands at 4,202.

The ordering promise was only ever about append order, and the append order has been changed by the retry. A consumer sees a payment for an order it has not yet seen. Depending on code it may drop the update, create a stub order, or crash into the retry path. Strict ordering is a property of the whole chain: key choice, in-flight limit, idempotence, and consumer logic.

With idempotence on, a Kafka-style producer can keep up to five requests in flight and still preserve order per partition, because the sequence numbers let the broker reject out-of-order retries. Setting a single in-flight request also works but caps throughput. Ordering across partitions is simply not promised.

F27 — Order committed, event missing: the dual-write gap

The order service writes the order row for HC-2048 to its database, then publishes OrderPlaced to the broker, as two independent steps. The database commit succeeds and the process crashes before the publish. The customer sees a confirmed order; email, fraud and analytics never hear of it.

A database transaction and a broker publish cannot be made atomic by ordering the calls. If you publish first and the commit fails, you broadcast an order that does not exist; if you commit first and the publish fails, you lose the event. Retrying inside the same process helps only until the process dies, which is exactly when it matters.

The transactional outbox removes the second write: the order row and an outbox row commit in one local transaction, and a relay publishes the outbox row later, marking it dispatched on a confirmed acknowledgement. The state of affairs after a crash is recoverable because the intent is stored. This atomicity is local to the originating database; it does not make the downstream consumers' effects atomic with anything.

F28 — Outbox relay publishes, then crashes before marking rows dispatched

The relay reads 500 undispatched outbox rows, publishes them as a batch, and is about to update their status when it is killed. The broker holds all 500 events. The database still lists them as pending.

On restart the relay selects the same 500 rows and publishes them again. The log now has 500 duplicates, adjacent to the originals or later depending on timing. Per-key order is still intact for the first copy of each event; the second copies arrive after newer events, so a consumer that treats arrival order as truth sees an old status after a new one. The relay is at-least-once by construction: the only way to make it at-most-once is to risk F27's loss.

Consumers protect themselves with the inbox and, for state, version numbers (apply an event only if its version is newer than the stored one). Smaller relay batches reduce the number of duplicates after a crash at the cost of more database round trips; neither choice removes them.

Consumers are where a delivery becomes a business effect. These incidents are about the order of the effect, the acknowledgment and the ownership claim, and what happens when ownership changes under a running worker.

F11 — Consumer applies the side effect, then crashes before committing the offset

The sales-analytics group reads OrderPlaced for HC-2048 from partition 7, offset 4,201. It inserts a revenue row into the analytics database, and the process is killed by the OOM killer before it commits offset 4,201. The committed offset, which is the group's bookmark of work finished, still says 4,200. (Kafka actually stores the next offset to read, so its bookmark would read 4,201 here; this page writes the last finished offset to keep the story easy to follow.)

When the partition is reassigned, the new owner resumes at the bookmark and reads 4,201 again. A second revenue row appears. Nothing was malfunctioning: the group committed after the side effect, which is the safe order, because the opposite order loses data (F12). It means redelivery is an expected event and a consumer that is not ready for it double-counts. Revenue dashboards drift upward by one order at a time, which is a hard error to notice by eye.

The repair belongs inside the database transaction. The consumer writes the business row and an inbox row (sales-analytics, evt-HC2048-01) together; a replay hits the unique key and is ignored. The offset commit then merely records progress, and the inbox is what carries correctness. If the effect is not in a database the consumer controls (an email, say), the effect itself needs an idempotency key supported by the provider.

F12 — Worker acknowledges the job before the business transaction commits

Worker A claims ship-HC-2048 from the shipment-jobs queue. To keep the queue clear, its code acknowledges the message as soon as it is received and then starts creating the shipment row in the order database. The database connection pool is exhausted, the transaction fails, and the process exits.

The queue believes the work is done: the ack told it to delete the message. There is no lease to expire and no retry to trigger. The task has vanished while the order remains paid and unshipped. This is the mirror image of F11, and unlike it nothing will ever repair itself. A DLQ cannot catch it either, since the message was never negatively acknowledged. RabbitMQ's own documentation describes the same trap for automatic acknowledgement mode, where a delivery counts as done as soon as it is sent, so a consumer whose connection or channel closes first loses the message.

The rule for an acknowledgement is that it asserts the work is finished or safely handed to something durable. Commit the transaction first, then ack; accept that a crash in the gap yields a redelivery, and make the work idempotent by job id (the shipment table's unique key on ship-HC-2048).

F13 — Work lease expires while a long job is still running

A work queue hides a claimed message from other workers for a visibility timeout (also called a lease): HarborCart uses 60 s as an illustrative value. Worker A claims ship-HC-2048, which includes generating a customs document that normally takes seconds and today takes 95 s because a partner service is slow.

At t=60 s the lease lapses and the queue makes the job visible again. Worker B claims it and begins its own 95 s run. For roughly 35 s both workers work on the same job, and both attempt to create a shipment record and print a label. Each did precisely what the system allows; the queue assumed 60 s was enough.

Three defences work together. Workers extend the lease with a heartbeat while they are alive and making progress, so a slow job is not mistaken for a dead one. The shipment table's unique key on the job id makes the second creation fail harmlessly. And a fencing token (a number that increases at each claim) lets the table reject a write from the stale claimant. Setting the lease to an enormous value is the tempting shortcut and it delays recovery from a genuinely crashed worker by the same amount. Real products expose the same dial under different names: Amazon SQS hides a received message for a visibility timeout (30 seconds by default, at most 12 hours from the receive) and lets a worker extend it with ChangeMessageVisibility; a RabbitMQ quorum queue returns a delivery to the queue if the consumer holds it unacknowledged past a delivery acknowledgement timeout (30 minutes by default in current documentation) or closes its channel. In both, a consumer that outlives its claim can still be running, which is why the fence and the unique key matter.

F14 — Rolling deploy rebalances the group while batches are in flight

The email-notifier group has 12 consumers and 24 partitions, two partitions each. A rolling deploy restarts the consumers one by one. Each restart triggers a rebalance, which is the group coordinator redistributing partition ownership, and the group generation advances from 12 to 13 and onward. With an eager protocol, every member gives up all its partitions at each rebalance; with a cooperative protocol only the partitions that must move are revoked.

The duplicate arises in the gap. A consumer holds a polled batch (offsets 4,201 to 4,300 on partition 7) and has already sent 40 emails. The rebalance revokes the partition before the offset is committed and hands it to another member, which restarts at the last commit and sends those 40 emails again. For a short period two members may be handling the same offsets, one finishing, one beginning.

Cooperative rebalancing reduces the work interrupted: when one of twelve consumers dies, 24 partitions over 11 survivors means two take three partitions and nine keep two, and a sticky assignor moves only the two orphans. Static membership (each consumer keeps a fixed instance id, so a quick restart within the session timeout does not trigger a rebalance at all) is the other lever for rolling deploys. Handling revocation by committing progress for finished records narrows the window; only an inbox or an idempotency key at the email provider closes it.

F15 — Orphaned consumer with a stale membership epoch writes anyway

Consumer C3 in sales-analytics hits a 50 s garbage-collection pause. The coordinator, hearing no heartbeat past the session timeout, declares C3 dead and gives its partitions to C5 at generation 13. C3 resumes still believing it owns partition 7 at generation 12.

C3's attempt to commit its offset is rejected, because the commit names an old generation; the broker fences it. That is good, and it is only half of the protection. The database insert that C3 issues before committing carries no generation at all, so the analytics table accepts it. C5 is by now processing the same offsets. The two write paths race, and the later write may overwrite a correct row with a stale one.

The offset fence guards the log, not your database. To protect external state, carry the generation (or any monotonically increasing claim number) into the write: UPDATE ... WHERE owner_generation <= :mine, or store it with the row and refuse anything lower. The same idea appears as a fencing token for locks and as the lease fence in F13.

Failures that repeat deterministically need different handling from ones that pass. Here the subject is classification, bounded retries, parking, redrive and the schema changes that create poison events.

F16 — Poison event blocks a partition

A consumer reads partition 3 and meets an OrderPlaced whose payload fails parsing, because a bug in a producer wrote a truncated JSON body. The consumer throws, does not commit, and the next poll reads the same offset. In a log, one consumer owns a partition and reads strictly forward, so the group cannot skip it without an explicit decision.

The average rate on one partition is 833.3 events/s (20,000 over 24). While the consumer loops, a backlog grows by that much: after 10 minutes the partition is 500,000 events behind. Every key that hashes to that partition is delayed, which is 1/24 of traffic, while the other 23 partitions look fine and the aggregate lag graph hides the problem in an average.

The guard is a classification step. A permanent failure (the bytes cannot ever parse) goes to the dead-letter topic after a bounded number of attempts, with the offset committed past it, and the consumer continues. A transient failure (a timeout) is retried with backoff. Parking the event also breaks ordering for that key until someone decides what to do, and an operator must know which events are waiting.

F17 — Infinite immediate retry creates a hot loop

The fraud-monitor consumer calls an inventory service that starts returning 503. The code catches the error and retries immediately in a tight loop with no limit. Each of the 12 consumers does this about 100 times per second, so a service that was already struggling absorbs 1,200 extra calls per second (hypothetical loop speed) at the moment it needs relief.

This is an amplification: the failing dependency's load rises because it is failing. Meanwhile the consumers make no progress, lag grows, and when the dependency recovers it faces the retry storm plus the real backlog together. Without jitter, consumers that failed together retry together, so the load arrives in synchronized waves.

Bounded retry with exponential backoff and jitter spreads the load: 2 s, 4 s, 8 s, 16 s plus random variation, at most 5 attempts, then the dead-letter path. A circuit breaker gives the dependency room to recover, and a retry budget limits the fraction of traffic that is retries. See retry with backoff and the retry storm module for the general mechanism.

F18 — The dead-letter queue grows silently

After a schema mistake, about 40 events per minute (hypothetical) are sent to orders.dlq, which retains data for 14 days. The consumer group is healthy, lag is zero, and every dashboard is green, because parking an event counts as a success for the main pipeline.

The numbers accumulate quietly: 40 × 60 × 24 = 57,600 a day, and after 14 days 806,400 events. On day 15 the retention policy starts removing the oldest parked events and those orders are permanently gone, never examined. A DLQ is itself a queue with its own retention, so unattended it is only a slower way to lose messages.

In Amazon SQS the clock is even less forgiving: a message's expiry is measured from its original enqueue time and is not reset when it moves to the dead-letter queue, so AWS advises giving the DLQ a longer retention period than the source queue. Treat the DLQ as an operational inbox. Alert on depth, arrival rate, and oldest age (say, a page when the oldest is above 24 hours); tag each entry with the failure reason, original topic, partition, offset and attempt count; and give someone an owner and a runbook.

F19 — DLQ redrive repeats a payment capture

A payment-capture job times out five times against the card processor and is parked. In reality the processor captured the funds on the first call and only the response was slow. HarborCart's incident shows 300 such parked jobs (hypothetical) after a processor brownout. An engineer bulk-redrives them into the main topic, and 300 customers are charged twice.

Parking an event does not prove the side effect did not happen. A timeout is the same unknown outcome as in F03. Redriving also reorders: the redriven job reaches the consumer after later events for the same order, so a refund for HC-2048 can be processed before the capture it refers to. FIFO guarantees, where they exist, apply to the original delivery within a key, not to the moment an operator reinjects a message.

For payment-capturing topics redrive is human-reviewed. Before reinjection, query the processor by idempotency key, send the idempotency key with each capture so that a repeat returns the earlier result, and redrive in small batches with the consumer's inbox in place. Where order matters, redrive to a separate topic or hold later events for the same key until the redrive completes.

F20 — A schema change breaks an older consumer

The OrderPlaced contract is on version 2 and fraud-monitor still reads v2. A producer team ships v3, renaming the required total field to totalAmount, through a hotfix path that skipped the registry's compatibility check. The registry is set to BACKWARD compatibility, meaning a new reader schema must be able to read data written under the old schema; that mode is there to make consumers upgrade first.

Here the producer went first. fraud-monitor deserializes the new records, finds no total, and treats it as 0, so every order now looks harmless to the fraud rules. Nothing crashes, which is worse than a crash. Other consumers fail loudly and park events by the thousand. The log is retained for 3 days, so poisoned events are recoverable but cannot be unwritten.

Compatibility has to be checked before the schema is accepted, enforced in CI and at runtime registration, with a non-bypassable path. Changes that break must go through a new topic or version, with a dual-write migration period. The check cannot prove semantic compatibility (a field that keeps its name but changes meaning).

F38 — A rolling upgrade changes protocol or schema compatibility

A new OrderPlaced v3 producer rolls out region by region, while consumers upgrade over a week. Halfway through, the schema registry accepts v3 because someone set the subject's compatibility to NONE for convenience. Old consumers that still decode with v2 begin failing on records from upgraded producers, and failed events retry, then flow to orders.dlq.

There are two layers. The wire protocol between clients and brokers has its own version negotiation, and a request using a version the other side does not support is refused with an explicit unsupported-version error. That is visible and loud. Schema compatibility is quieter: the data parses at the transport layer and breaks in the application. With BACKWARD compatibility, new schemas can read old data, which suits consumers upgraded first. Upgrade order matters: consumers before producers for backward changes, producers before consumers for forward ones.

A rolling upgrade should be a series of reversible steps with explicit gates: upgrade brokers, then clients, then flip the new schema or feature on only when the lowest-version member of every fleet understands it.

Redrive, ordering and what a DLQ does not prove

Parking an event in a dead-letter queue records that processing failed, not that nothing happened. The consumer may have charged the card and then failed on the confirmation email, so a parked payment event is potentially a completed payment. That is why orders.dlq redrive for payment-capturing topics is human-reviewed, and why every consumer on the redrive path needs the same inbox check as on the live path: the event id decides whether the effect already exists.

Ordering is a second trap. A partition delivers its events in order to its single consumer, and that is the whole ordering promise. Once an event is parked, its successors for the same key continue on the main path, and when the parked event is later redriven it arrives after them, so OrderPaid may be processed before the redriven OrderPlaced. Redrive therefore cannot restore the original sequence; consumers need version numbers or state checks to tolerate it, or a stricter design pauses the whole key until the poison event is resolved. Queue-style FIFO guarantees are narrower still: strict first-in first-out breaks as soon as there are several consumers, retries or redeliveries, and is meaningful only inside one ordering scope such as a message group.

Capacity problems in a log are mostly about where load lands. Hot partitions, idle consumers, slow groups, full disks and reconnect storms each need a different lever.

F21 — High-volume customer creates a hot partition

A partition is the unit of parallelism and of ordering: records with the same key go to the same one, and one consumer in a group owns it. Someone chooses tenant as the key, reasoning that per-tenant order is desirable. HarborCart's own harbor-retail tenant is 30% of the traffic.

At 20,000 events/s, 30% is 6,000 events/s, all of it hashed to a single partition, against a uniform share of 833.3 events/s (20,000 over 24). That partition receives 7.2 times its fair share, one broker's disk and one consumer's CPU absorb it, while the other 23 partitions run at about 609 events/s each (14,000 over 23), well within what a partition can carry. Adding consumers does nothing, since a partition has one owner at a time. Adding partitions does not help either, because the tenant is still one key.

The canonical key is order:<orderId>, which spreads a tenant's orders while keeping each order's events in sequence. The wider lesson is to pick the narrowest key that still carries the ordering the business actually needs. When a single entity really is that hot, split it with a salt (order:<id>:<n>) and accept that ordering applies within each salted lane, or move the heavy tenant to a dedicated topic with its own partitions.

F22 — Slow subscriber's backlog grows while other groups are healthy

orders has three independent consumer groups (named sets of consumers that share the partitions of a topic so each record is processed once per group). email-notifier and fraud-monitor are healthy. sales-analytics slows down because its database is under maintenance, and it manages 12,000 events/s against an arrival rate of 20,000.

Its backlog grows by 8,000 events/s: after 30 minutes, 14,400,000 events. In time terms the oldest unprocessed record is 14,400,000 ÷ 20,000 = 720 s, 12 minutes, old. Because each group keeps its own offsets, the slow group does not slow the other two, which is the whole point of the log over a shared queue. Retention, however, ignores consumers: the broker deletes by age or size whether or not the slow group has read the data, so a slow group is not a claim on disk but a race against the retention clock, and the end of that race is F23.

Lag measured in offsets and lag measured in age tell different stories. Offsets say the group is 14.4 million behind; age says 12 minutes. The second is what a business owner understands. Alert on both, and keep the answer to a lagging group proportionate: scale its consumers if partitions allow, fix the downstream, or accept lag for a non-urgent view such as analytics.

F24 — Broker disk fills during a promotion burst

The promotion pushes ingest to 100,000 events/s for 10 minutes, five times the 20,000 baseline. At 1 KiB each, that is 97.66 MiB/s and 57.22 GiB gross over 600 s; the amount above the baseline is 45.78 GiB. With RF=3, every extra byte is stored three times. A broker that had less headroom than its share of that growth reaches a full disk, because time-based retention cannot free anything younger than 3 days.

Once a log directory fills, the broker stops being able to append and may take the affected partitions offline. Leaders move to the replicas, which hold the same data and may be nearly as full. The failure then cascades: failover adds load to survivors, replicas fall out of sync, and writes begin to be rejected (F07). Producers retry and the outbox grows.

The remedy starts before the disk: producers need a bounded buffer and a load shedding policy (drop or delay low-priority events), consumers need backpressure signals, and operators need an alert on projected time to full, using the rate of growth rather than a fixed percentage. Size for peak plus failure, not for the average, and decide in advance what is shed first.

F25 — Reconnect storm saturates the metadata and control plane

A five-minute network blip drops the connections of 600 producers and 150 consumers (hypothetical fleet). They all reconnect within the same second, each requesting cluster metadata for 24 partitions, and the group coordinator receives join requests from every consumer at once.

The data path might survive, but the control path is a smaller, shared resource. Metadata requests queue, rebalances begin and restart as members time out, and the group enters a rebalance loop in which nobody processes records. The incident is self-inflicted: clients behave the same way at the same time because they run the same code with the same timeouts.

The remedy is jitter and limits on the client side (randomized reconnect backoff with a cap), server-side rate limits on metadata requests, longer session timeouts relative to blip duration, static membership for stable consumers, and separating the control plane's capacity from the data plane's. Rolling restarts should be paced too, since a deploy is a controlled version of the same storm.

F26 — Partition expansion changes key placement and ordering scope

To prepare for the promotion, the platform team grows orders from 24 to 48 partitions. Key-to-partition placement uses a hash modulo the partition count, so the mapping changes for roughly half the keys: a key stays only if its hash modulo 48 equals its hash modulo 24.

Take order:HC-2048. Its earlier events, OrderPlaced at partition 7, offset 4,201 and OrderPaid at 4,202, are on partition 7. A later OrderShipped hashes to partition 31, which is owned by a different consumer who may process it before the first consumer has finished the older events. For the period around the change, a per-key order that was guaranteed is no longer guaranteed, and the existing history cannot be moved without rewriting the log.

Expansion therefore is a planned migration. Quiesce or drain before expanding where order is essential; use the new partitions for new topics; or version the key with the partition count and have consumers tolerate a transition window. Expanding is also one-way (partition counts normally cannot shrink), so the cost of over-provisioning early is lower than the cost of expanding late.

F39 — A few extremely heavy keys unbalance partitions

order:<id> keys spread HarborCart's load well in normal times, but a different topic keyed by sku carries one flash-sale item whose key produces 6,000 events/s, and a second hot SKU hashes to the same partition. That partition receives 12,000 events/s while the 24-partition uniform average is 833.3 events/s, about fourteen times higher. Under the hypothetical per-consumer capacity of 2,000 events/s, the owner of that partition is six times overloaded and lags without bound, however many consumers exist.

Unlike F21, where a whole tenant was concentrated, here the hot units are individual keys, and there are only a few. Adding partitions doesn't split one key; both keys will still map to some partition each (possibly the same one). Adding consumers doesn't help either, as a partition has a single owner within a group.

Real options are all about the key. Salt the hot key into several sub-keys (sku#0 to sku#7) and accept that order is then preserved only within each sub-key, or route hot keys to a dedicated topic, or handle the ordering requirement downstream. The first question is whether strict ordering of that key's events matters at all.

F40 — Scaling the group beyond the partition count changes nothing

The sales-analytics group is behind during a promotion. It has 24 consumers for 24 partitions, each with the hypothetical 2,000 events/s capacity, so the group tops out at 48,000 events/s against 100,000 arriving. The team scales to 40 consumers. Sixteen of them join, receive no partitions, and sit idle. Throughput stays 48,000 events/s, the bill grows, and lag keeps growing at 52,000 events/s.

In a Kafka-style group each partition has exactly one owner at a time, so the partition count is the upper bound on parallel consumption. Extra members act as standbys that can take over if an active one fails. To process 100,000/s at 2,000 per consumer you would need 50 partitions' worth of owners, so the real choices are more partitions, faster consumers, or accepting lag until the burst ends.

Raising the partition count has side effects (F26): key placement changes and ordering across the transition is not preserved, so it is a planned operation rather than an emergency lever. A queue with competing consumers on one shared list, such as the shipment-jobs work queue, behaves differently, since workers are not bound to partitions.

A retained log promises replay only within limits. These three incidents show consumers that assumed more history than the log was configured to keep.

F23 — Retention deletes messages before a lagging group catches up

The log keeps records for 3 days by time, regardless of whether anyone has read them. A deploy accidentally leaves sales-analytics paused. Nobody sees it over a long weekend, and 4 days pass.

By the third day the oldest unread offsets cross the retention boundary and the broker deletes whole segments from the log's start. When the group restarts, its committed offset points at data that no longer exists, an out-of-range error. The reset policy decides what happens: earliest jumps to the oldest retained record and silently skips what was deleted, while latest jumps to the end and skips everything. One day of data at 20,000 events/s is 1,728,000,000 events, permanently missing for that group, though other groups that kept up have it processed.

Committed offsets are also subject to expiry once a group is empty for a long time (the default is 7 days in recent Kafka versions), so a long outage can lose the bookmark itself. The defense is to alert on lag age against retention (for example, page at 50% of retention), keep retention longer than the longest tolerable outage including weekends, and have a rebuild path from the source of truth.

F35 — Compaction removes history a consumer assumed was there

The customer-address topic is compacted: the broker keeps at least the latest record for each key and eventually discards older ones. A new analytics job starts from the beginning of the topic expecting every address change since launch, so that it can build an address-history report. What it reads is the latest address per customer, plus whichever older values had not yet been cleaned.

Compaction is a state-snapshot feature, not an archive. It lets a late consumer rebuild current state without replaying every update, and in exchange gives up intermediate values. Even for the latest value there is a subtlety: a delete is recorded as a tombstone (a record with a null value), which is itself removed after delete.retention.ms (24 hours by default). A consumer that is offline longer than that can miss that the key was deleted.

The report built on that data looked plausible, which is the real danger: no error, just missing history. The fix is to put history where it belongs. Keep a retained, non-compacted event topic, or an event-sourced store or warehouse, for audits, and let the compacted topic feed only the current-state view.

F36 — A consumer starts from an offset outside retention

The fraud-monitor group is paused for a long weekend of maintenance, for 5 days. Retention on orders is 3 days, so the oldest 2 days of what it still needed has been deleted. On restart the group asks to resume from offset 3,100 on partition 7; the log now starts at 3,980. The broker reports the position is out of range.

What happens next is a configuration decision someone made long ago: the reset policy. With 'earliest', the group jumps to the oldest available record and silently skips roughly 880 events on that partition. With 'latest', it jumps to the end of the log and skips everything not yet read, and with 'none' it fails loudly and waits for a human. The wrong default on a fraud consumer is worse than the outage.

For a consumer whose misses are costly, 'none' is the safer default: the group stops, and an operator chooses the restart point deliberately (an offset looked up by timestamp, or the oldest retained record plus a backfill job for the gap). The expiry of committed offsets described in F23 applies here as well; a group empty for longer than the offsets retention period would not even have offset 3,100 to ask for.

A shared platform needs identity, bounded inputs and controlled configuration. Each of these incidents is a trust boundary that was drawn in the wrong place.

F30 — Cross-tenant subscription attempt

A second tenant, meridian, runs a legacy integration against HarborCart's platform. Its subscription request carries a tenantId field in the request body, and the legacy gateway trusts it. A bug (or an attacker) sets it to harbor-retail. The gateway authorizes the subscription and starts delivering harbor-retail orders to Meridian's client.

The failing assumption was that the tenant is something the caller says. Identity has to come from the credential: the authenticated principal maps to a tenant, and topic ACLs grant that principal specific operations on specific topics. A payload field can describe the data; it cannot authorize access to it. In this scenario, about 1,000 events (hypothetical) were delivered before an alert, including customer addresses.

Containment means revoking the credential and listing every delivered offset. Correct enforcement checks the ACL on the verified principal at subscribe time and on every fetch, logs denials, and treats a denial pattern as a security signal. Quotas keep a single tenant from using the shared cluster as an exfiltration channel.

F31 — A huge or hostile payload overwhelms the validator

HarborCart caps a single event at 1 MiB (an illustrative limit) while the average order event is 1 KiB, so the cap is 1,024 times the typical size. A compromised integration for tenant meridian starts publishing 1 MiB documents full of deeply nested JSON, at 400 requests/s. That is about 400 MiB/s of inbound traffic, roughly twenty times the entire 19.53 MiB/s average ingest of the platform, and each payload makes the schema validator do real parsing work before the broker can reject or accept it.

The harm arrives from two directions. Network and CPU on the ingest tier are consumed first, so legitimate producers see timeouts. Anything that does get through is replicated three times and held for three days, so a few minutes of abuse leaves a long tail of useless bytes on disk. A size check that runs after full deserialization protects nothing, because the expensive step has already happened.

The defense is ordered cheapest-first: reject on the declared length before reading the body, apply a per-principal byte-rate quota, then validate with a bounded parser (depth and field-count limits) and only then append. Quota rejections should be explicit and retryable so a well-behaved client backs off rather than hammering.

F32 — An unauthorized retention override exposes customer data

Retention on orders is three days, and each event includes shipping addresses. An engineer debugging a replay request sets the topic's retention to unlimited from a shared admin console that did not require a specific permission. Weeks later a security review finds roughly a month of address data still sitting in the log, readable by every group that is allowed to consume orders, and past the period the privacy policy promised.

Two separate mistakes combined. First, configuration was a privilege that nobody had to justify: altering topic settings is a control-plane action that deserves its own ACL and audit record. Second, the log was holding personal data in plain form. An append-only log cannot easily delete one customer's record, so deletion rules are best implemented by limiting what goes in (redact or tokenize before append), encrypting sensitive fields with per-subject keys that can be destroyed, and keeping retention short by policy.

After discovery the immediate step is to restore the intended retention; segments past it are removed on the next cleanup. Removing earlier copies that were mirrored or replayed into other systems is a separate hunt, because the log is not the only place that data lives.

Two regions give you two failure domains and a policy question about what may cross between them. Both incidents are about what a regional failover can and cannot preserve.

F33 — A region fails while residency policy forbids mirroring

HarborCart runs two independent messaging fabrics, EU and US. Topic orders.eu carries European customers' data and is tagged so it is never copied across the border. At 14:02 the EU fabric loses most of its brokers because of a provider incident. US operations ask whether they can simply route EU checkouts to the US fabric until it recovers.

The honest answer depends on policy, not on engineering convenience. Routing EU events into the US would violate the residency rule, so it is off the table. The EU checkout path therefore has a defined degraded mode: accept orders into each region's local database and outbox, hold the events in the outbox until the EU fabric returns, and show customers an honest 'confirmation will follow' state. Nothing is lost provided the outbox survives, and the relay drains the backlog afterward.

Non-resident topics (orders.global, say) can be mirrored asynchronously and fail over, with a cost (see F34). Residency-tagged topics trade that failover for compliance. The business should know in advance which side of that line each topic is on, rather than discover it during an outage.

F34 — Async mirror falls behind, then is promoted with a missing suffix

For a non-resident topic, HarborCart replicates US to a standby cluster asynchronously. During a busy hour the mirror link slows, and the standby trails by 12 s. At 20,000 events/s that gap is 240,000 events, about 234 MiB at 1 KiB each (illustrative). The primary region then fails and the team promotes the standby.

The standby is a consistent prefix of the primary's log, but it is missing the last twelve seconds. Producers acknowledged those events under the primary's replication policy; the standby never saw them. After promotion, consumers either re-read events they had already handled (their offsets from the old cluster do not map onto the new log) or silently skip the gap, depending on how offsets were translated.

This is the price of asynchronous cross-region copying: recovery point is greater than zero. The decision to promote must weigh that loss against continued unavailability, and the missing events can often be rebuilt from upstream sources of truth. For orders, the order database and outbox still hold them, and the relay can republish with the same event ids so inbox dedupe absorbs the overlap. If the old primary comes back holding that missing suffix, it cannot simply resume as primary: its log has diverged from the promoted cluster's, so it must be rebuilt or reconciled before any failback.

You can only run what you can see accurately. One incident misreads a metric because of clocks, and the other loses all visibility because monitoring shares the broken cluster.

F29 — Clock skew misreports lag

A team builds a lag dashboard from event time: now minus the eventTime field the producer wrote. One producer fleet's clocks are 180 s behind. The dashboard shows email-notifier three minutes late, even though it consumed the event milliseconds after it was appended.

The on-call engineer sees a red graph and scales the consumer group from 12 to 16. Partitions are 24, so scaling is possible but not needed, and a rebalance is triggered for nothing. In the opposite direction, a fleet whose clocks run fast makes real lag look smaller or even negative, hiding a true delay.

Use two measures and understand each. Offset lag (the log end offset minus the committed offset) uses no clock. Oldest message age uses the broker's append timestamp, which relies on the broker's clock instead of the many producer clocks. Event-time lag has value for business questions such as 'how stale is this report,' and it must carry an error bar from clock synchronization.

F37 — The metrics pipeline depends on the broken cluster

HarborCart ships broker, consumer and lag metrics through the same message platform that it monitors. When a controller problem makes orders unavailable, the telemetry topic stalls too. Dashboards flatline at 09:14, which looks like 'quiet', and the alert that would have fired never arrives because it depends on the same path.

A monitoring system that shares fate with the monitored system fails exactly when it is needed. The pattern appears in many incidents: the status page needs the database that is down, the on-call tool needs the VPN that is down. Here the symptom is subtle because a missing series reads as zero lag, and the graph looks healthy.

The remedy is independence. Brokers and clients should expose metrics through a path that doesn't traverse the cluster (scraped over HTTP by a separate monitoring stack), alerts should treat missing data as a signal (a dead-man's switch that fires when no heartbeat arrives), and there should be a synthetic producer-consumer probe running end to end with its own alert route.

The incidents reduce to a short list of signals, a few drills and some standing rules. Having them before the first failure turns most of the forty into short events.

Signals worth paging on, and signals worth a ticket

Set alerts relative to consequences. Lag of 3,000,000 records may be a minute at peak or an hour at night; oldest-record age and the ratio of lag to retention say far more. A consumer-lag figure alone cannot tell you the difference between a healthy burst and a stuck group.

Game days worth running

• Kill one broker during load and confirm writes continue with two in-sync replicas, then kill a second and confirm writes are refused rather than lost (F05 to F07).
• Crash a consumer after the database commit and before the offset commit; confirm the replay is absorbed by the inbox table (F11).
• Run a rolling deploy of consumers and count duplicates per rebalance (F14, F15).
• Publish a poison event and verify it reaches the DLQ after five attempts without stalling the partition (F16, F17).
• Pause a group for longer than half of retention and check that the alert fires before data expires (F23, F36).
• Fail the EU fabric and rehearse the degraded mode and outbox drain (F33).
• Fail the cluster and confirm that the dead-man's-switch alert still arrives (F37).

• Acknowledging or committing before the business effect is durable (F12).
• Treating a DLQ as proof that the effect did not happen (F19).
• Trusting a tenant id in the payload instead of the verified principal (F30).
• Using event-time lag alone to decide on scaling (F29).
• Scaling consumers past the partition count and expecting more throughput (F40).
• Relaxing schema compatibility to ship faster (F20, F38).
• Monitoring a cluster through the same cluster (F37).

Verification notes

Platform defaults quoted here were checked against the Apache Kafka source (producer delivery timeout of 120 s, at most five in-flight requests with idempotence, consumer session timeout of 45 s, offsets retention of 7 days, tombstone retention of 24 hours, log retention of 7 days by default, unclean leader election off by default). Consumer acknowledgment, automatic requeue on channel close and the delivery acknowledgement timeout come from the RabbitMQ documentation; the SQS visibility timeout, five-minute FIFO deduplication interval and dead-letter expiry rule come from the Amazon SQS developer guide. Behavior and defaults change between versions, and managed-service limits vary, so read the current documentation of whatever you run. Every HarborCart figure is illustrative.