Message Queue Architecture: Partitions, Replication, Offsets & Delivery

A broker moves bytes, but everything that makes those bytes safe to process, route, deduplicate and trace has to travel with them in an envelope. Getting the envelope wrong is expensive because it is the one contract that every producer and every consumer, written in any language and deployed on any schedule, has to agree on.

The running example is fictional. HarborCart is an invented commerce company, and every number in this page is an illustrative planning input rather than a measurement: 20,000 events/s on average, 100,000 events/s during a 10-minute promotion, about 1 KiB per encoded event, three consumer groups, replication factor 3 and three days of retention. Two systems recur as reference points: Kafka is an open-source distributed log in which topics are partitioned, replicated and retained, and RabbitMQ is an open-source broker built around queues with per-message acknowledgment; the page names them to show where products draw lines differently, not to prescribe either. A shopper places order HC-2048, and the order service emits an OrderPlaced event with id evt-HC2048-01.

A few terms before we use them. A producer is any program that publishes messages. A consumer reads and processes them. A topic is a named stream that producers append to and many independent subscribers can read; a queue in the work-queue sense is a named buffer whose messages are each handed to exactly one of several competing workers. A partition is one ordered slice of a topic, and an offset is a record's position inside its partition. Later chapters give each of these a proper treatment; for now the envelope only needs to carry enough information for them to work.

The fields and who is allowed to set them

The distinction in the third column that matters most is the tenant. A payload field called tenant_id is data supplied by the caller, and a caller who can write anything into the payload can claim to be anyone. The broker therefore derives the tenant from the authenticated principal on the connection and stamps it on the record itself; if the payload also carries a tenant value, the broker compares the two and rejects a mismatch instead of picking one. This is how a request from tenant meridian that names harbor-retail in its body fails at the door rather than landing in somebody else's topic.

Ordering and serialization across languages

The key decides the partition and the partition decides the ordering. If HarborCart used harbor-retail as the key, every event for the tenant would land in one partition and be ordered together, which is more ordering than anyone needs and the cause of the hot partition we meet in the failure pages. Using order:HC-2048 orders the events of one order relative to each other (OrderPlaced at offset 4,201, OrderPaid at 4,202 in the same partition) and nothing else. The consumer must not assume anything about the relative order of two different orders.

The payload is serialized with a schema-based format so that a Java producer and a Go consumer agree on field types, optionality and defaults. A free-form JSON blob with an implied shape works until a producer team renames a field. The schema_version field and the schema registry (discussed in the control-plane chapter) turn that rename into a compatibility check rather than an incident. The cap on payload size is also part of the envelope contract: HarborCart's average event is about 1 KiB and its illustrative maximum is 1 MiB, large enough for a rich event and small enough that a single record cannot monopolize a batch, a fetch response or a consumer's memory.

• Keep message_id stable across every retry and resend of the same logical event.
• Preserve occurred_at when an event is redriven or replayed; add a separate delivery timestamp if you need one.
• Reject, do not reconcile, a payload tenant that disagrees with the credential tenant.
• Cap payload and header size at the broker, and fail loudly above the cap.

• Using a random value as the key when the consumer needs per-entity ordering.
• Putting personal data in headers, where it escapes schema review and field-level redaction.
• Generating a new id inside a retry loop.

Most confusion about queues comes from two systems using the same word for different things. This chapter pins down the vocabulary as conceptual structures, then notes where Kafka-style logs, RabbitMQ-style queues and managed queue services draw the lines differently.

A broker is a server process that stores messages and serves producers and consumers. A cluster has several, because one broker's disk and network are finite and its failure must not end the service. A topic is split into partitions; each partition is an append-only sequence of records in segments, which are files that roll over by size or time. Every record gets a monotonically increasing offset within its partition. Each partition has one leader broker that accepts writes and serves reads, and followers that copy the leader's log; the replicas that are caught up form the in-sync replica set (ISR).

On the consuming side, a consumer group is a named set of consumers that cooperate to read a topic: each partition is owned by at most one member of the group at a time, and different groups each get their own complete view of the topic. A membership epoch (Kafka calls related values the group generation and member epoch) increases whenever the group's membership changes, so a member that was removed can be recognized as stale. The group's committed offset per partition is a bookmark meaning "resume reading here". A subscription, in the pub/sub sense, is a consumer group or a per-subscriber queue attached to a topic.

Two warnings keep the model honest. First, a queue and a log answer a different question. A work queue asks "who should do this job, once?" and forgets a message after it is acknowledged. A log asks "what happened, in order?" and keeps records until retention removes them, leaving each reader to keep its own bookmark. A product can offer both shapes, and a design should say which shape each stream needs. Second, none of these words implies a guarantee on its own: a partition leader does not imply durability, and a committed offset does not imply the business work finished. The chapters on acknowledgment and replication say exactly what each one does mean.

The producer's job looks trivial, send a message, until a network timeout leaves it unsure whether the broker stored the record. Every design decision in the producer API exists to answer one question honestly: what does the caller know right now, and what must it do about what it does not know?

DURABLE means the broker has satisfied the advertised condition, which for HarborCart's orders topic is "appended by the leader and replicated to enough in-sync replicas to meet min.insync.replicas=2". REJECTED means a definite refusal: the payload exceeded the cap, the principal lacked permission, the schema failed validation, the quota was exceeded. OUTCOME_UNKNOWN is the honest answer to a timeout. The request may have been lost on the way, or stored and the response lost on the way back. The producer cannot tell by looking, and treating it as REJECTED invites either a dropped event (the caller gives up on something that was in fact stored, or on a retry that never happens) or a duplicate under a new id, and treating it as DURABLE invites silent loss when the request never arrived.

Batching, compression and the latency trade

Sending one record per network request wastes most of the work on headers, syscalls and round trips. Producers therefore group records bound for the same partition into batches, optionally compress the batch, and send it in one request. Kafka's producer exposes this with batch.size and a linger interval (linger.ms, which in recent versions defaults to 5 ms and in older ones to 0). The arithmetic is simple: at the 100,000 events/s promotion rate, one-event requests mean 100,000 requests per second, while batches of 100 mean 1,000 requests per second, so request overhead drops by about a hundredfold. The cost is that an event may wait up to the linger interval for company. Batching raises throughput and can raise individual latency; it does not automatically lower p99 (the latency that 99 percent of requests beat), and a claim that it does needs a measurement from your own system.

Batches also change failure granularity. A batch that times out has an unknown outcome for every record it contains, so the retry sends all of them again. That is acceptable only because of the next mechanism.

Retries, the delivery deadline and the duplicate they create

A producer that retries after a timeout can store the same record twice: the first attempt succeeded and its acknowledgment was lost. Retrying forever is not an option either, so a producer has a delivery deadline. Kafka's default delivery.timeout.ms is 120,000 ms, and the retry count is effectively bounded by it. After the deadline the caller receives a failure and must treat the outcome as unknown.

The idempotent producer narrows the duplicate problem. The broker assigns the producer an id and an epoch for the session, and the producer numbers its records per partition with sequence numbers. If a retry of sequence 41 arrives after the leader already stored sequence 41, the broker recognizes it and acknowledges without appending again. Kafka enables this by default (enable.idempotence=true), which requires acks=all, a positive retry count and at most five in-flight requests per connection; within those limits it also preserves the order of a partition's records across retries.

The limits matter as much as the feature. The sequence state belongs to one producer session and one partition. If the process restarts it gets a new producer id and its sequence numbers begin again (Kafka's separate transactional-id mechanism can carry a producer's identity across restarts, but it is a different feature with its own costs), so a resend of evt-HC2048-01 from a restarted service looks like a fresh record. Idempotence does not compare payloads or business ids, and it does not see a resend that the application deliberately produced with a different internal id. It also does not reach into the consumer's database. Broker-scoped idempotence suppresses one duplicate source; the application still needs stable message ids and a consumer-side check, which the delivery-guarantees chapter covers.

Key-based routing completes the picture. The producer hashes the key to pick the partition, and the same key goes to the same partition as long as the partition count does not change. Changing the count remaps keys, which the partition-key chapter explains. The producer must also respect a maximum payload, 1 MiB illustratively for HarborCart, by checking before it sends rather than learning from the broker's refusal in production.

When the order service says publish, a small chain of events has to occur before anyone can honestly say the record is safe: find the leader, validate, append, replicate, and only then answer. The place where the broker answers is the acknowledgment boundary, and everything before it is a promise the producer has not yet been given.

Take evt-HC2048-01. The producer's metadata says the key order:HC-2048 maps to partition 7 and that broker L leads partition 7. L receives the batch and checks the envelope: the credential may write to orders, the payload is under the size cap, the schema version is registered and compatible, the tenant quota has headroom. A record that fails any check is rejected here, with a reason, and never touches the log.

A valid record is appended to the active segment of partition 7 and receives offset 4,201. At this instant it is in the leader's log, possibly only in the operating system's page cache; whether it has reached disk depends on the flush policy, and many log-based brokers rely on replication rather than flushing every write. The followers, F1 and F2, fetch from the leader's log and append to their own. The leader tracks how far each follower has copied. The high watermark is the highest offset replicated to every in-sync replica, and consumers may read only up to it, so no consumer ever sees a record that a leader crash could still erase.

With acks=all the leader answers only after every replica currently in the in-sync set has the record, and min.insync.replicas=2 adds a floor: the set must contain at least two members, here the leader and one follower, or the write is refused. This is the acknowledgment boundary for orders. If only the leader remains in sync, the leader refuses the write with a not-enough-replicas error rather than acknowledge something one disk failure could destroy.

RabbitMQ draws the same distinction in its own vocabulary: publisher confirms and consumer acknowledgements are described in its documentation as orthogonal. A confirm tells the publisher the broker has taken responsibility for the message (for a persistent message in a durable classic queue, after persisting it; for a quorum queue, after a quorum has accepted it). A consumer acknowledgment tells the broker the delivery was processed so it can delete the message. Different mechanisms, the same lesson: the publisher's success and the consumer's success are separate events with separate failure modes.

A work queue exists to give each job to one worker, and to give it to another worker if the first one disappears. Both halves depend on a lease: a time-limited claim that the broker can take back.

Walk through the HarborCart job ship-HC-2048 on the shipment-jobs queue. Worker A calls receive with a 60-second lease (an illustrative figure). The broker marks the message invisible to other workers and returns it with a receipt token and a deadline. Worker A starts the shipping label work. If it finishes inside the lease it calls ack(token), and the broker deletes the message. If worker A crashes, nobody calls ack; at the deadline the message becomes visible again and worker B receives it with a new token and a delivery attempt count of 2.

Three details distinguish a correct implementation from a leaky one. The first is the stale receipt: if worker A merely paused (a long garbage collection, a stalled network) and wakes up after worker B was handed the job, A's old token must be refused. The receipt identifies one lease, not the message, so an acknowledgment from an expired lease cannot delete a message that someone else is now processing. The second is lease extension: a job with variable duration should renew its lease periodically instead of choosing one enormous timeout, because a giant timeout means a crashed worker's job sits invisible for that whole time. The third is the attempt counter: each redelivery increments it, and when the count reaches the limit (HarborCart uses 5) the broker sends the message to a dead-letter queue instead of handing it out again.

A stale receipt only protects the queue. It cannot stop worker A's side effects: if A paused after charging the shipping carrier but before acknowledging, then B repeats the call. The work queue provides at-least-once handing out of jobs, so the job handler must be idempotent, for example by passing the job id as an idempotency key to the carrier. The lease controls who may acknowledge, not who may act.

Why this is not an offset commit

Lease-based queues acknowledge individual messages; log-based systems store one bookmark per partition. They are not interchangeable. In a queue, message 7 can be acknowledged while message 6 is still leased, because each message has its own state. In a log consumer, committing offset 8 states that everything before it is done, so a consumer that processes records concurrently must be careful to commit only the lowest contiguous completed offset. Treat ack(receipt) and commitOffset(partition, next) as two different contracts, as the subscriber chapter will show.

A subscriber's contract is shorter than a producer's and its failure modes are nastier. Read some records, do something with them, then tell the broker how far you got. The order of the last two steps decides whether a crash loses work or repeats it.

Consumers pull: they ask for records when they are ready. This is deliberate, because it makes the consumer's own capacity the flow-control signal. A push system must guess how much a consumer can take and, if wrong, either overwhelm it or leave it idle; pull systems that need push-like latency use long polling. A consumer also bounds how much it holds at once, the prefetch or maximum in-flight amount, so that a slow consumer cannot silently accumulate gigabytes of unprocessed records in memory.

The commit rule is simple to state: commit the offset only after the business effect has committed. HarborCart's sales-analytics consumer reads evt-HC2048-01 at offset 4,201, writes a row to its analytics database, commits that database transaction, and then commits offset 4,202 to the broker (the next offset to read). If it crashes between the database commit and the offset commit, restart resumes from 4,201 and sees the event again: a duplicate delivery, but never a loss. If it did the reverse and committed the offset first, a crash before the database write would lose the event permanently, because the group's bookmark says it was handled.

The third row deserves a note, because it appears in real designs. A consumer that stores its offsets in the same database as its output, and updates both in one transaction, gets exactly-once application to that database: either both the row and the offset move or neither does. The broker's own stored bookmark is then secondary, and on startup the consumer seeks to the position its database recorded. It works only for one database, and it gives up the broker's group tooling. The more common pattern is the inbox, covered in the delivery chapter.

Seek and replay reposition a group. Because a log keeps its records until retention removes them, a bug fix can be followed by moving sales-analytics back to the offset or timestamp of the bad deploy and reprocessing. Replay is bounded by retention (three days for orders), by authorization (the group needs permission to read old data) and by whether the consumer's effect is safe to repeat. Replaying a payment capture is not safe merely because the log still has it.

Rebalance callbacks are the broker telling a consumer that partition ownership is about to change. onRevoked is the last chance to finish in-flight work and commit offsets for partitions that are leaving, and onAssigned is where the consumer loads state for the ones arriving. Handlers that take too long here stall the whole group. Finally, there is no implicit transaction between a consumer and an external database. The broker cannot make UPDATE analytics ... and commit offset 4,202 atomic across two systems, which is the reason the inbox exists.

Choosing which events a subscriber receives

HarborCart's orders topic carries OrderPlaced, OrderPaid, OrderShipped and OrderCancelled. sales-analytics wants all of them; email-notifier wants only shipped and canceled, say 15% of the traffic (hypothetical). There are four ways to give each subscriber what it needs, and they have different costs.

One topic per event class (orders.shipped) lets a subscriber read exactly what it wants, but it splits ordering: OrderPlaced and OrderShipped for the same order now sit in different logs with no order between them. Consumer-side filtering keeps one topic and one order: the group reads everything and discards by a type header. It costs reads, since at 20,000 events/s and 1 KiB each, email-notifier pulls about 19.5 MiB/s and throws away about 16.6 MiB/s of it. Filtered-out records must still be committed past, or lag never falls. Broker-side routing is the AMQP-style approach: the publisher sends to an exchange with a routing key such as order.shipped.eu, and each queue is bound with a key or pattern. On a topic exchange the key is a list of dot-separated words and * matches exactly one word while # matches zero or more, so order.shipped.* and order.# are different subscriptions. The broker copies matching messages into each bound queue, which moves the filtering cost to publish time. Some managed pub/sub services instead attach a filter on message attributes to a subscription and drop non-matching messages before delivery; log-style brokers such as Kafka mostly leave content filtering to the consumer or to a derived topic written by a stream job.

The routing question is separate from the fan-out question. Work-queue routing decides which one worker handles a job, so shipment-jobs bound by region sends each job to a single worker in the right region. Pub/Sub routing decides which independent subscribers each receive a copy. On an exchange both come from the same mechanism: consumers that share one queue compete, and separate queues bound to the same exchange each get their own copy. Three traps recur. The routing key is a contract, so renaming it breaks every binding. A wildcard such as order.# silently picks up event types added later. And a message published with no matching binding may be dropped without any error unless the publisher asks to be told. HarborCart keeps one keyed orders topic so per-order ordering survives, lets groups that need most events filter by type, and gives email-notifier a derived topic only if its read cost becomes the problem.

Ordering in a partitioned system is not a property of the topic. It is a property of one partition, and the key you choose is the only lever that decides which events share one.

The producer hashes the key and takes the result modulo the partition count. order:HC-2048 maps to partition 7 of 24 and stays there as long as the key and the count stay the same, so OrderPlaced (offset 4,201) and OrderPaid (offset 4,202) land in one partition in the order they were published. A consumer that owns partition 7 reads them in that order. Another order, order:HC-2049, hashes elsewhere and has no ordering relationship with HC-2048 at all.

Choosing a key

The key should be the smallest entity that needs ordering. HarborCart needs OrderPlaced before OrderPaid before OrderShipped for the same order, so the key is the order id. A tenant-only key would order all of the tenant's events together, and a busy tenant would then concentrate its load on one partition. Ordering by customer would be right for a loyalty-points consumer. If no consumer needs ordering, a null key lets the producer spread events across partitions for balance (Kafka's default partitioner sticks to one partition per batch, then moves on), giving up any ordering at all. The arithmetic of skew is plain: with 20,000 events/s over 24 partitions the uniform average is 833.3 events/s per partition; if one tenant is 30 percent of traffic and is the key, one partition takes 6,000 events/s, about 7.2 times the average (illustrative figures).

Per-key serial processing and its exceptions

Partition ordering only becomes processing order if the consumer processes one partition's records serially, or at least serially per key. A consumer that fans a partition's records out to a thread pool can finish OrderPaid before OrderPlaced. Retries introduce the second exception: if record 4,201 fails and is moved to a retry queue while 4,202 proceeds, the order has been broken on purpose. Either the consumer blocks the key until the retry resolves, which delays everything behind it, or it accepts reordering and makes handlers tolerant of it, by checking state versions. Neither is free.

Changing the partition count

Adding partitions changes the modulus, so many keys map to new partitions. During the change, records for order:HC-2048 published before the expansion sit in partition 7 and records published afterwards may go to partition 31. Consumers reading both partitions no longer see a single ordered history for that key. Teams handle this by over-provisioning partitions at design time (an expansion in HarborCart's case from 24 to 48 would be planned, not improvised), by draining in-flight work before an expansion, or by creating a new topic and migrating. Kafka lets you increase partitions but not decrease them.

A consumer group splits a topic's partitions among its members so that each partition has exactly one reader at a time. The interesting part is not the split but what the group does in the seconds after a member dies.

email-notifier has 12 consumers and orders has 24 partitions, so each consumer owns two. At the illustrative average rate every partition carries about 833 events/s, so each consumer handles about 1,667 events/s, comfortably below the hypothetical capacity of 2,000 events/s that this page assumes for a single consumer. Nobody has measured that capacity; it is a planning input.

One consumer crashes. The group coordinator, a broker that tracks the group's members, notices when the member misses its periodic heartbeats beyond the session timeout, or when the process closes its connection, and starts a rebalance. The group generation moves from 12 to 13; this is the membership epoch. Every surviving member rejoins and receives a new assignment under the new generation. With 11 consumers and 24 partitions, 24 = 2 × 11 + 2, so nine survivors hold two partitions each and two survivors hold three. The three-partition owners now handle about 2,500 events/s, above the 2,000 events/s assumption, so lag grows on those partitions even though the *average* looks fine. A capacity plan must look at the worst-loaded member, not the mean.

Eager versus cooperative rebalancing

In the classic eager protocol every consumer revokes all of its partitions and the group re-forms from scratch, so processing pauses everywhere for a moment. Cooperative (incremental) rebalancing, and the newer group protocol, move only the partitions that must move. The two orphaned partitions are reassigned and the other 22 keep running. Sticky assignment keeps unchanged ownership where it can, which also preserves the caches and state a consumer has warmed up.

In-flight work and fencing

The dead consumer may have been in the middle of a batch. Its uncommitted records will be re-read by the new owner, so the new owner may repeat work, and the inbox makes that harmless. The harder case is a consumer that is not dead but slow: it missed heartbeats during a long pause, the group moved on, and then it wakes up and tries to commit offsets for a partition it no longer owns. The commit carries the old generation, and the coordinator rejects it. This is fencing: the epoch is a token proving the sender still owns the partition. Without it, the stale consumer could overwrite a newer bookmark and cause skipped records.

Fencing protects the broker's bookmark, but not the stale consumer's database writes. If the old consumer writes to the analytics database after losing ownership, only a database-side check, such as comparing the write against a monotonic fencing value or enforcing the inbox uniqueness, prevents damage. Rebalances also have a cost that scales with how often they happen: a consumer whose processing loop exceeds the poll interval gets ejected, which triggers a rebalance, which pauses others, which creates more lag, which causes more ejections. Tune poll intervals to the slowest legitimate batch.

People say a system is exactly-once when they mean one of at least four different things. Separating delivery from effect is the single most useful clarification in this entire subject.

At-most-once means a message is delivered zero or one times: commit first, process after, and a crash loses the message. At-least-once means it is delivered one or more times: process first, commit after, and a crash repeats it. These two describe *delivery*. Exactly-once effect is a statement about a business outcome, such as one analytics row for evt-HC2048-01. It is built from at-least-once delivery plus deduplication, never from a delivery that is somehow guaranteed single.

The inbox pattern is the workhorse. HarborCart's consumer database has a table inbox(consumer, event_id) with a unique constraint. For each record, the consumer opens a transaction, tries to insert the pair, and applies the business change only if the insert succeeded; both commit together. Redelivery of evt-HC2048-01 after a crash finds the row, applies nothing and still advances the offset.

The inbox is local to the consumer's database. If the consumer's effect is a call to an external payment API, the inbox cannot make the call safe, because the inbox row and the API call do not share a transaction. The usual answer is to pass a stable idempotency key (the event id) to the external API, and for effects without one, to accept and document that risk. That is why HarborCart's payment-capturing topics have human-reviewed redrive: the retry itself is a business action.

An order service that writes its database and then publishes to a broker has made two writes to two systems with nothing tying them together. Crash between them and either the order exists without an event or the event exists without an order. The outbox moves the problem into the one place where atomicity is available: the service's own database.

In HarborCart's order service, placing HC-2048 runs a single local transaction that inserts the order row and an outbox row describing OrderPlaced. Both commit or neither does. A separate relay process reads undelivered outbox rows, publishes them to the broker, and marks each row dispatched after the broker acknowledges it. The service no longer talks to the broker on the request path, so a broker outage delays events but cannot fail checkouts.

The relay can crash after the broker accepted the record and before it marked the row. On restart it finds the row still undelivered and publishes again, with the same event_id. The topic now holds the event twice. The outbox guarantees that no committed order loses its event; it does not guarantee that the event appears once. Duplicates are expected and handled where they matter, by the consumers' inbox and, for transport-level repeats, by the idempotent producer, whose sequence state is lost if the relay restarts.

Consumer side: one transaction, then the offset

In SQL the guard is an INSERT INTO inbox (consumer, event_id) VALUES (...) ON CONFLICT DO NOTHING, and the business change runs only when a row was actually inserted. The unique constraint, not an application-level SELECT, is what survives two concurrent deliveries of the same event.

Operating the outbox

The outbox table grows, so a cleanup job deletes dispatched rows after a retention period chosen to cover the investigation window. The relay needs a claim mechanism so that two relay instances do not publish the same row concurrently: row locking with skip-locked reads (each relay takes rows no other relay has locked) is a common choice, and an alternative is reading the database's change log. Ordering matters too: if events for one key must stay in order, the relay publishes that key's rows in insertion sequence and does not parallelize within a key. Outbox lag (the age of the oldest undispatched row) is a metric of its own, since a stalled relay produces no errors anywhere else.

Retries convert transient failures into success and permanent failures into load. A retry design is therefore three decisions: which failures to retry, how long to wait and how many times, and where a message goes when the answer is no.

The first split is transient versus permanent. A timeout calling the fraud service, a deadlock, a throttled downstream: these may succeed later. A payload that fails schema validation, a missing mandatory field, a business rule violation: these will fail identically every time, and retrying them burns capacity while delaying everything queued behind. The consumer classifies the error and acts accordingly. Unknown errors default to a bounded number of retries rather than infinite ones.

For transient errors, HarborCart's illustrative policy is at most 5 attempts with exponential backoff 2 s, 4 s, 8 s, 16 s, each with random jitter. Jitter matters: without it, a thousand messages that failed at the same moment retry at the same moment and recreate the spike that caused the failure. The retry budget is also a protection for the downstream, which may already be struggling, so consumers combine backoff with a circuit breaker or a rate limit instead of hammering.

Where the waiting happens

Sleeping inside the consumer blocks the partition or the lease for the delay. Alternatives: a delayed-delivery feature of the queue itself, a series of retry topics (orders.retry.2s, .4s and so on) consumed by workers that wait for the due time, or a scheduler that republishes after the delay. Retry topics unblock the main stream and preserve visibility, at the cost of breaking per-key order, because a retried record is now behind later records. Whether that is acceptable depends on whether the handler can cope with an older event arriving after a newer one.

Poison messages and the DLQ

A message that exhausts its attempts is moved to the dead-letter queue (orders.dlq, retained 14 days in HarborCart's illustrative policy) along with the failure reason, attempt count and original envelope, including the unchanged occurred_at and trace context. The main stream continues. The DLQ turns an invisible stall into a visible, alertable backlog. Parking is not the same as safe, though: a message in the DLQ may already have caused a partial side effect before it failed, so inspection must consider what the handler did before throwing.

Redrive sends dead-lettered messages back to the source after the cause is fixed. For HarborCart, redrive on payment-capturing topics is reviewed by a person, since replay could capture twice unless the downstream honors an idempotency key. Redrive also reorders: a message parked three hours ago now lands behind everything since. This is why strict FIFO guarantees do not extend across a DLQ, and why a consumer sensitive to order should carry a version or sequence it can compare.

A log broker's storage design is simple on purpose. Records are appended to the end of a file, files are deleted whole, and reading is a sequential scan from an offset. That simplicity is what makes retained replay cheap, and it is also what makes retention a hard limit.

Each partition is a directory of segments. The newest, the active segment, receives appends; when it reaches a size or age limit it is closed and a new one starts. Each segment has a sparse offset index that maps some offsets to file positions, so finding offset 4,201 needs a lookup and a short scan, not a read from the beginning. A time index maps timestamps to offsets, which is how a consumer can say "replay from 09:00". Retention deletes the oldest closed segments when they exceed the configured time or size, so it works at segment granularity and a record may outlive the nominal limit until its whole segment qualifies.

HarborCart's orders topic uses retention of three days with the delete cleanup policy. By the planning numbers, ingest at the average is 19.53 MiB/s, or about 1.77 TB (decimal) per day, which makes three days about 4.83 TiB of logical payload and, with replication factor 3, about 14.48 TiB of replicated payload. These exclude indexes, segment overhead, compression, skew and operational headroom, so they are a lower bound for discussion, not a disk order. Apache Kafka's own default for retention.ms is seven days and retention.bytes is unbounded; HarborCart overrides the time, and a size limit would be a second guardrail. A topic with only a time limit can fill disks during a promotion.

Compaction is not history

A compacted topic keeps at least the latest record for each key and discards superseded ones in the background. It suits state-like data: the current address of a customer, the current status of an order. A tombstone, a record with a key and a null value, marks the key for deletion and is itself retained for delete.retention.ms (24 hours by default in Kafka) so lagging consumers can observe it. Compaction does not retain every event: a consumer starting late sees the latest value per key and may miss intermediate changes, so a compacted topic cannot rebuild a full audit trail. Event sourcing built on a compacted topic is a design error unless that is understood.

Retention bounds replay and privacy duties

Replay can never reach beyond retention, so a consumer that has been down for four days on a three-day topic has lost the oldest day of events, and the recovery is a rebuild from the source system. The same property helps with privacy: data in a log is deleted by expiry, but a deletion request inside the retention window has no per-record delete in an append-only log. Practical answers are minimizing personal data before it is appended, encrypting sensitive fields with per-subject keys that can be destroyed, or using compaction with tombstones for keyed data. A design that claims deletion on request should say which of these it relies on.

Replication is how a log broker survives machine failure, and its guarantees are narrower than the phrase suggests. The honest statement has two halves: which failures an acknowledged record survives, and which failures stop writes.

Each partition has one leader and followers. A follower fetches from the leader, appending what it receives; the in-sync replicas (ISR) are the leader plus the followers that are sufficiently caught up. The leader advances the high watermark when every ISR member has a record. A follower that falls behind is removed from the ISR, and when it catches up it is added back. The leader epoch is a number that increases with every leadership change. For orders partition 7, epoch 7 becomes epoch 8 after the election below, and any request carrying epoch 7 from a deposed leader is refused.

HarborCart configures orders with RF=3, min.insync.replicas=2 and producers that wait for the in-sync replicas (acks=all). Consider broker L leading partition 7 with F1 and F2 in sync. A record is acknowledged once L and at least one follower have it. If L dies, the controller elects a new leader from the ISR, say F1; F1 has every acknowledged record, so nothing acknowledged is lost, and F1 gets epoch 8. Producers refresh metadata and write to F1. With only F1 and F2 left, two in-sync replicas remain, so writes continue.

The second row is the guarantee, and the third row is the price. With two brokers gone the survivor holds the committed data but cannot accept new writes, because acknowledging a write on a single replica would break the promise made when minISR was set to 2. This is deliberate: HarborCart chose to reject writes over accepting writes that one more failure could erase. The outbox absorbs the gap, since relays keep retrying while the checkout continues.

Unclean leader election

Suppose the leader dies and every other ISR member is also gone, leaving only a replica that was out of sync, perhaps behind by 300 records. An unclean election lets that replica become leader anyway. Availability returns immediately, but the 300 records acknowledged earlier are lost, and consumers who already read them now hold data the new log does not contain. Kafka's default is unclean.leader.election.enable=false, which keeps the partition offline until an in-sync replica returns; HarborCart keeps it off for orders. A telemetry stream where fresh data matters more than completeness could turn it on knowingly.

A broker deposed by partition or pause and then returning must not keep writing as the leader. Its stale epoch gets it refused, and its log is truncated to the new leader's history before it rejoins. Raft-based queues, such as RabbitMQ quorum queues, take a related approach with majorities: a write commits when a majority (⌊N/2⌋+1, so 2 of 3) accepts it. That tolerates one failure in three nodes while writing, and shares the same two-failures limit.

A queue converts a rate mismatch into a delay, and a delay is only free while there is space to hold it. Flow control is the set of limits that decide what happens when space runs out, and the design question is who is told, how soon, and what they are asked to do.

Start with the capacity worksheet, because the limits come from it. All figures are HarborCart illustrations. Average ingest is 20,000 events/s × 1 KiB ≈ 19.53 MiB/s; the 10-minute promotion at 100,000 events/s is about 97.66 MiB/s. Over those 600 seconds the gross volume is 57.22 GiB (60,000,000 events), of which 45.78 GiB (48,000,000 events) is the increment above the baseline. Followers copy every byte twice at RF=3, adding 39.06 MiB/s at the average, and three groups reading add 58.59 MiB/s, so broker-side egress is about 97.66 MiB/s at the average and 488.28 MiB/s at the promotion peak, before protocol overhead, compression and batching.

Consumer sizing uses the page's hypothetical 2,000 events/s per consumer: ten active consumers per group at the average and fifty at the peak. Three groups at 20,000 events/s are 60,000 group-deliveries/s, which are logical reads and not extra appends. For the work queue, Little's Law (concurrent items in flight = arrival rate × time each spends in service) gives the worker count: 5,000 shipment tasks per minute is 83.33 tasks/s, and at a 0.3 s service time that needs about 25 concurrent workers in steady state, before headroom, retries and bursts.

Where the pressure goes

A burst crosses several stages. The producer's local buffer fills if the broker is slow, and then the producer blocks or fails its publish calls; blocking a checkout thread is rarely what you want, which is another reason for the outbox, where the backlog accumulates in the database instead. The broker enforces quotas in layers, cluster-wide limits above per-tenant limits above per-principal or per-topic limits, so one tenant cannot consume the cluster and one noisy service cannot consume its tenant's share: HarborCart's per-tenant publish quota is an illustrative 5,000 events/s for harbor-retail, applied by throttling the producer, which slows down and eventually receives refusals instead of everyone sharing the damage. On disk, retention and size caps bound growth, and a disk-high-water alarm should fire well before the volume fills, because a full disk takes brokers offline.

On the consumer side, pull plus bounded prefetch is natural backpressure: the consumer takes only what it can hold. When consumers are slower than arrivals, lag grows, but the log is the buffer, and lag is limited by retention. Downstream limits matter too: a recovering group that reads at 50,000 events/s against an arrival rate of 20,000 events/s drains at a net 30,000 events/s, so a 30-minute outage's backlog of 36,000,000 events (34.33 GiB, already inside the retained log) clears in 1,200 seconds, 20 minutes, assuming constant rates. If the recovering group can only manage 20,000 events/s it never drains, and a downstream database's write limit often sets the real rate.

Shedding on purpose

When something must give, the choice should be written down in advance. Low-value events (clickstream) are sampled or dropped; payment events are never dropped and instead cause the caller to be told to retry later. A rejected publish with a clear code and retry hint is better than a hung connection. For the full discussion of the patterns, see backpressure and load shedding.

A shared message platform runs a data path, which carries messages, and a control path, which decides who may use it and where things live. Mixing them lets a metadata problem take down the data path, and mixing up tenants lets one customer read another's events.

The data plane is brokers handling produce and fetch requests and replication. The control plane holds cluster metadata (topics, partition leaders, ISR), the ACL store, quotas, schema registry and topic configuration, the group coordinator that tracks consumer membership and committed offsets, and the controller that elects leaders. Brokers cache what they need, so a transient control-plane outage should not stop established traffic; but it can prevent leader elections and topic creation. A design says which operations stop during a control-plane failure, and the answer is a monitoring item, not a surprise.

Identity, ACLs and namespaces

The broker authenticates each connection, over TLS, to a principal. Authorization is a topic ACL: harbor-retail may write orders and its subscribing services may read it; meridian may not read orders at all. A tenant namespace prefix on topic names (harbor-retail.orders in a shared cluster) keeps names from colliding and gives ACLs a pattern to match. Credentials are short-lived where possible, rotated without downtime, and granted per service, not per team, so a leaked key has a narrow blast radius. Every denied request and every administrative change is written to an audit log; a cross-tenant attempt such as meridian publishing into a topic owned by harbor-retail is both refused and recorded.

Quotas, schemas and placement

Quotas bound publish and fetch bandwidth, request rates and topic or partition counts per tenant; an overrun is throttled, then refused. Quotas are also the foundation of billing and capacity planning. The schema registry stores versions of each event schema and applies a compatibility rule when a producer registers a new one. HarborCart uses BACKWARD compatibility on OrderPlaced, meaning consumers using the new schema can still read data written with the previous one; a v2-to-v3 change that adds a required field with no default is rejected at registration, because a v3 reader could not fill that field when reading older records, so fraud-monitor is protected before the change ships. (Removing a field is normally allowed under this rule, since a new reader simply ignores it.) Placement decides which brokers hold which partitions, spreading replicas across racks or zones so one failure domain does not hold all three.

Sensitive data has its own rules. Personal data is minimized before append, fields that must exist are encrypted, and the retention chapter's deletion caveats apply to an immutable log. Encryption in transit is TLS; encryption at rest is volume-level or application-level, depending on whether brokers should ever see the plaintext.

A queue that is running is not a queue that is working. The signals that matter measure whether records are flowing to their consumers on time, and each needs its own alert, because they fail for different reasons.

Lag in offsets and lag in time answer different questions. Offset lag is cheap and exact, but without the arrival rate it tells you little: a 36,000,000-event backlog at 50,000 events/s of catch-up is 20 minutes of work, and at 20,000 events/s is a permanent problem. Oldest event age is what a business cares about ("orders are reaching email 9 minutes late") and works when computed against a trustworthy time; because it uses a producer-supplied timestamp, sanity-check it against broker append time and expect clock error.

Tracing across a queue needs deliberate propagation. The producer writes the trace context (the traceparent header) into the envelope. A consumer's processing span links back to the producing span (OpenTelemetry's messaging conventions describe both a parent relationship and span links for this). With this in place, one order's trace shows commit, outbox dispatch, broker append and each group's processing as one story, including the gap in time spent waiting in the log.

Operational safety is more than dashboards. Alert on rates of change, not just thresholds: ISR shrinking on three brokers at once is a network event, not three disk problems. Practice the failure paths (kill a broker, expire a lease, force a rebalance) in a staging cluster, and write down the redrive and replay procedures before the day you need them. For the broader operational picture see Kafka consumer lag.

Everything in the previous sixteen chapters meets in the life of one event. Reading it as a timeline with explicit causal boundaries shows exactly which failures each safeguard covers, and which no safeguard covers.

HC-2048 is placed. In one local transaction the order service inserts the order and the outbox row for evt-HC2048-01. From this point the event is guaranteed to exist as long as the order does, and the checkout response can go back to the shopper without touching the broker. Boundary: order and outbox committed.

The relay claims the row and publishes to orders, key order:HC-2048. The broker authenticates, authorizes, validates the schema and quota, and routes to partition 7. The leader appends at offset 4,201, followers replicate, and with two in-sync replicas holding it the leader acknowledges. The relay marks the outbox row dispatched. Boundary: durably committed per policy. If the relay crashed just before marking the row, the event is published a second time at a later offset, same id, and the inbox will absorb it.

Three groups read partition 7 independently. email-notifier fetches offset 4,201, sends the confirmation through a provider with the event id as the idempotency key, commits its inbox row and then its offset. fraud-monitor is slow and sits at offset 3,900, far behind; its lag grows, but it neither slows nor blocks email. sales-analytics writes the analytics row and the inbox row in one transaction, then crashes before committing the offset. On restart it re-reads 4,201, finds the inbox row, applies nothing, and commits 4,202. Boundaries: delivered, business effect committed, offset committed, each per group.

OrderPaid arrives at offset 4,202 in the same partition and is processed after OrderPlaced, because they share a key. A day later a bug is found in the analytics logic. The team deploys a fix and seeks sales-analytics back to the time of the bad deploy, replaying from the retained log; the inbox is cleared for the affected range or versioned so the corrected logic runs. Replay works because three-day retention still holds the records, and because the analytics effect is safe to recompute. A payment capture could not be replayed the same way.

What this design cannot promise

• Global total ordering across partitions at scale.
• Exactly-once effects across independent systems without a shared transaction.
• Replay beyond retention, or of compacted-away history.
• Unconditional cross-region failover where residency rules forbid copying data.
• Accepting writes after any two broker losses at RF=3 with minISR=2.

These limits are not defects to be engineered away; they define the problem. A good design states them, picks the cheapest mechanism that handles what remains, and puts a metric on every place a guarantee ends.