Pub/Sub

Publish/subscribe, often called pub/sub messaging, is a messaging pattern in which a producer publishes a message to a topic or channel without addressing any particular consumer. Subscribers declare interest in the topic, and the messaging system delivers eligible messages to each of them. The producer is decoupled from who the subscribers are, when they are deployed and, often, whether they are online at that moment, so a new reaction to a business event can be added without editing the service that produced it.

Suppose checkout completes an order and the order service publishes OrderPlaced. Inventory reserves stock, notifications send a confirmation, analytics records the conversion and fraud monitoring updates its model. Those are four responsibilities with different owners, different latency needs and different failure modes. If the order service had to call all four synchronously just to confirm that an order was recorded, its availability would be the product of theirs, and every new consumer would be a change to the order service.

Pub/sub does not promise that every subscriber receives every event. That depends on whether the subscription is durable, whether the broker retains messages, whether the consumer acknowledges them, and whether the publish itself was durable. A volatile in-memory channel and a persistent, replayable log can both be called pub/sub and have completely different recovery behavior. The rest of this lesson is about telling them apart.

The mental model has two independent dimensions, and most pub/sub bugs come from mixing them up: fan-out happens between subscriptions; load sharing happens inside a subscription. Each independent subscription (a Kafka consumer group, a RabbitMQ queue bound to an exchange, an SQS queue subscribed to an SNS topic) gets its own view of the stream. Within one subscription, several consumers divide the work. Getting an event to every interested service is the first dimension. Getting it processed once per service by one of that service's instances is the second.

A direct call couples the caller to the callee in three ways. The caller must know the callee's identity (address, API), it must wait for the callee to be available now, and it is released only when the callee has finished. Publishing to a topic removes all three. The producer knows a topic name and a message contract. Subscribers can be offline, slow or added months later, and the producer's request completes when the broker has accepted the event, not when anyone has acted on it.

Take the three-month-later case. The recommendations team wants order data. With direct calls they open a ticket against the order service, wait for a release, and the order service now has a fifth outbound dependency. With a topic they create a new subscription and start reading, and the order service changes by zero lines. If the broker retains history, they can also start from older events to backfill their model; how, and what it costs, is covered under adding a subscriber below.

What does not go away is the contract. The producer still cannot rename a field without breaking whoever reads it; the coupling has moved from call signatures to event schemas, and it is harder to see because the consumers are not in the producer's codebase (see schema evolution). Nor does it remove shared failure modes: the broker cluster, its storage quota, a hot partition or a burst of publishing can still degrade every subscriber at once. And it removes the immediate answer: the publisher learns that the event was accepted, never that the inventory was actually reserved. Anything that needs that answer is a synchronous interaction or a coordinated workflow, not an observer.

In a competing-consumer work queue, a message is normally processed by one worker out of a group; that is the subject of Queues & Workers. In pub/sub fan-out, each independent subscription receives its own copy (or its own position in a shared log). Inside each subscription, multiple consumers may still divide the work. The two behaviors compose, and the figure below shows both at once.

Read it left to right. The order service publishes OrderPlaced keyed by order_id, and the broker acknowledges once the record is appended (boundary 1; what that acknowledgment means depends on the broker and its replication settings). The topic has three partitions. Three consumer groups each read the whole topic. Inventory runs two instances that split the partitions: inventory-1 owns P0 and P1, inventory-2 owns P2, so every event is handled by exactly one inventory instance. Email and analytics each have a single consumer that owns all three partitions, so each of them sees every event, independently of inventory. Each group tracks its own progress and acknowledges by committing its own offsets (boundary 2); one group committing says nothing about the others. The log itself is retained by time or size and not by consumption (boundary 3), which is what makes a late or recovering subscriber possible.

The most damaging topology bug follows directly: inventory and email must not share a Kafka group ID. If they do, the group has one set of partitions to divide among all members, so inventory and email each see only a fraction of events and neither fails loudly. Try it below. Flip the checkbox and watch the share of events each service sees drop below 100%.

The same figure shows the other limit: within a group, a partition is owned by one member at a time, so more instances than partitions leaves the extra instances idle. Real Kafka assignors (range, round-robin, sticky, cooperative sticky) distribute partitions differently from the simple modulo used for illustration, and a rebalance reassigns them whenever members join or leave. The shape of the guarantee is the same.

Pub/Sub vs a message queue

The practical difference is who the message is for. A queue exists to get work done: each message is a task, and exactly one of several competing workers should handle it, so adding workers adds throughput. Publish/subscribe exists to tell several independent parties that something happened: each interested subscriber gets its own copy, so adding a subscriber adds a reaction, not capacity. A queue answers "who will do this?"; a topic answers "who needs to know?". The second shape is what makes an event-driven design extensible, because new consumers attach without the producer changing.

Real brokers do not sort neatly into two product categories, so it is more accurate to describe delivery semantics per subscription. Across subscriptions, every subscription receives the message. Within a subscription, consumers either compete (a consumer group, a shared subscription, workers on one queue) or there is a single consumer. The common products are combinations of these two rules:

• Kafka: each consumer group is a subscription that tracks its own offsets; members of a group split the partitions, so a group with several members behaves like a work queue and several groups behave like pub/sub.
• SNS with SQS: SNS fans a message out to several subscribed SQS queues, and the workers reading each queue compete for that queue's messages.
• RabbitMQ: a fanout or topic exchange copies a message into every bound queue, and consumers on one queue compete for its messages.
• Google Cloud Pub/Sub: every subscription on a topic receives each message, and the subscribers attached to one subscription share its messages.

Retention and replay depend on the broker rather than on the pattern. In a log-based system such as Kafka, records stay until retention removes them, independent of consumption, so a group can rewind or a new group can read history. In a queue-based system such as classic RabbitMQ queues or standard SQS queues, a message is removed once it is acknowledged or deleted, so there is nothing to replay and a subscriber added later sees only new messages. Some queue-style services add limited replay (for example, Google Cloud Pub/Sub can retain acknowledged messages and seek a subscription back, and RabbitMQ Streams are log-based), so check what the specific broker and configuration keep. For the competing-worker side, including acknowledgments, visibility timeouts and retries, see Queues & Workers.

A topic names a stream or category of messages. A subscription represents one recipient's interest and, in durable systems, holds its delivery position. A partition is an ordered shard of a log: it buys parallelism and typically confines ordering to that shard. A consumer group is a coordination unit whose members divide partitions or messages between them. No broker exposes all four in the same way, so the useful question is always "what is the independent delivery path here, and what holds its position?"

The table is a map, not a feature comparison: each row hides configuration that changes the answer (replication and acknowledgment settings, queue types, retention policy). For per-broker depth see the Kafka, RabbitMQ and Redis modules. The delivery semantics behind each acknowledgment column, including SQS's visibility timeout and RabbitMQ's prefetch, are worked through in Queues & Workers.

Kafka is the reference point for the rest of this lesson because its model separates the pieces cleanly: the log is shared and immutable, and a consumer group is nothing more than a name plus committed offsets. That is why fan-out is cheap there (a new group adds no copies of the data) and why a wrong group ID is so quietly destructive. A RabbitMQ topology gets the same result with a different cost model: every subscriber has its own queue, so the broker stores a copy per queue and a slow subscriber's queue grows on its own.

An event says something happened: PaymentCaptured, OrderCancelled. A command asks a component to do something: CapturePayment. Events suit fan-out because any number of independent consumers may react and none of them is the event's owner. A command has exactly one accountable handler, and the sender usually needs to know whether it succeeded. Publishing SendWelcomeEmail to a topic as if it were a fact hides the fact that one service is responsible for executing it, and nobody is watching to see that it was. Commands belong on a queue or in a job system that tracks outcome.

Name events in the past tense. Give each a stable event_id, the aggregate it concerns, a type, a schema version and the time it occurred, plus the business payload. Do not rely on timestamps for uniqueness or ordering: clocks differ across hosts and equal timestamps are common. Keep secrets and unnecessary personal data out of broadly readable topics, because events get retained, replicated and copied across teams.

{
  "event_id": "evt_8f2d",
  "event_type": "OrderPlaced",
  "schema_version": 2,
  "aggregate_id": "ord_1042",
  "aggregate_version": 7,
  "occurred_at": "2026-10-29T10:15:00Z",
  "data": {
    "order_id": "ord_1042",
    "currency": "INR",
    "total_minor_units": 259900
  }
}

event_id is what consumers use to detect duplicates, and aggregate_version is what they use to reason about order. The version describes order but does not enforce it: a consumer must still compare it with what it has stored and decide what to do with an event that arrives late or out of sequence, as shown in the next section. Money is carried in minor units as an integer, so the payload does not depend on a floating-point round trip.

One more design choice is how much the event carries. A thin event (an ID and a type) forces every consumer to call back to the source, which reintroduces runtime coupling and reads a state that may have changed since. A fat event carries the facts consumers need at the moment of the event, so they can work and replay without the producer being available. Carry the facts that were true then; let consumers look up anything that is allowed to change.

A distributed topic generally cannot offer arbitrary horizontal parallelism and a single global order at the same time. Kafka resolves the tension by guaranteeing order only within a partition. If every event for one order uses order_id as its key, the default partitioner hashes the key and routes all of that order's events to the same partition, where the broker preserves append order. Events for different orders may land on different partitions and have no defined relative order, which is almost always what you want.

That guarantee has edges you should know about:

• Producer retries. If a producer retries a failed send while later sends are already in flight, the retried batch can land after them. Kafka's idempotent producer (enable.idempotence=true, the default in recent client versions) prevents both duplicates from retries and this reordering for a bounded number of in-flight requests. Verify the setting rather than assuming it.
• Changing the partition count. The key-to-partition mapping depends on the number of partitions. After an increase, new events for an existing key can go to a different partition than its older events, and no order is defined across that boundary while consumers drain the old one.
• More than one publisher. If two services publish events about the same aggregate, the broker orders them by arrival, not by business causality. Per-key order assumes one logical writer per key, or an explicit version.
• Downstream effects. Partition order does not make the consumer's database write happen once, and it does not survive a handler that processes records concurrently or retries out of line.

Choose the key from the unit that needs ordering, not from whichever field has the highest cardinality. order_id orders one order's lifecycle. customer_id would order everything for a customer, which is stricter than most consumers need and concentrates traffic. The opposite failure is a hot key: one very large merchant or one celebrity account can produce so much of the traffic that its partition becomes the bottleneck for the whole topic. Because a partition is read by one consumer per group, adding consumers does not help.

If you split a hot key (for example by appending a sub-key) you gain throughput and lose simple per-key serialization. When strict order still matters, push the guarantee into the data rather than the transport: carry an aggregate_version and make the consumer apply it conditionally.

CREATE TABLE order_projection (
  order_id          text    PRIMARY KEY,
  status            text    NOT NULL,
  aggregate_version integer NOT NULL
);

-- Apply an event only if it is newer than what is stored.
-- 0 rows affected means the event was a duplicate or arrived late.
INSERT INTO order_projection (order_id, status, aggregate_version)
VALUES ('ord_1042', 'placed', 7)
ON CONFLICT (order_id) DO UPDATE
   SET status = EXCLUDED.status,
       aggregate_version = EXCLUDED.aggregate_version
 WHERE order_projection.aggregate_version < EXCLUDED.aggregate_version;

A consumer that assumes OrderCreated always precedes OrderPaid breaks the first time they travel on different topics, different partitions or different producers. The usual cause is not the network reordering packets; it is that the two events never shared a partition key in the first place.

A durable subscription lets a consumer resume after downtime within the broker's retention or delivery limits. In Kafka the group's position lives in committed offsets, and records are kept by retention policy (a time or size limit per topic, 7 days being a common broker default) or by compaction, not by whether anyone has read them. In an ephemeral channel such as Redis Pub/Sub, a disconnected consumer simply misses messages. Some brokers expire unacknowledged messages, subscriptions or retained data according to policy, so "durable" always has a limit you should be able to state.

The timeline shows the limit being crossed. Producers publish for ten days into a topic with 7-day retention. A group commits its position at day 1 and then is down for nine days. When it returns at day 10, records older than day 3 have been deleted. The two days between its committed position and the oldest retained record are gone for that group. What the consumer does next is a configuration, not a law: in Kafka, auto.offset.reset decides whether it jumps to the earliest retained record, to the end of the log (silently skipping the rest), or fails. Choose deliberately, and alert on the age of the oldest unprocessed event relative to retention, not only on absolute lag. Retention applies at log-segment granularity, so the exact boundary is approximate.

Adding a subscriber

Replay and its side effects

In Kafka a consumer can reset its offsets and re-read retained records, which is how you rebuild a search index, repair a projection after a bug, or populate a new store. Replay re-executes the handler, so it repeats any external side effect the handler performs: replaying OrderPlaced through a handler that sends email sends the emails again. Replay-safe consumers separate pure projection work from effects that touch the outside world, and either make the effects idempotent or switch them off during replay.

Replay also assumes the events still mean what they used to. If a six-month-old event refers to a customer record that has since been deleted or changed, replay may not reconstruct the original business context. Decide per event whether it carries enough historical fact to be self-contained or whether consumers must look something up, and how that lookup behaves after data changes. Log compaction is a different retention mode: Kafka keeps at least the latest record per key, which suits state topics (the current profile of each user) but deliberately discards intermediate history, so it is not an audit trail. The full event-log-as-source-of-truth design is event sourcing; pub/sub with retention is a weaker thing and should not be mistaken for it.

A producer change that looks harmless can break a consumer you have never met. Adding an optional field with a documented default is usually compatible. Changing a field's meaning, unit, identifier or requiredness usually is not: a consumer that parses total_minor_units as paise will silently produce wrong numbers if the producer begins sending cents. Compatibility is specific to the format and the policy: reusing a Protobuf field number is hazardous, JSON Schema needs explicit validation rules, and Avro depends on reader and writer schemas and the registry's compatibility mode. "Never remove or rename a field" is a good habit but not a theorem; it depends on who reads it and whether they read by name or position.

A safe rollout is staged because the deployments are independent. Publish a compatible schema first, while readers still accept both shapes. Verify that consumers tolerate it, using their deserialization error metrics and contract tests, not an assumption. Migrate readers to the new field. Remove the deprecated behavior only after every reader has moved, which requires knowing who the readers are. For a substantial semantic change, a new event type or an explicitly versioned contract is clearer than overloading the old one: publish both until the consumers have moved, then retire the old one.

A schema registry enforces structural compatibility; it cannot know that a field changed meaning. That is why teams that run many subscribers keep an event catalog (the event, its owner, its schema, its known consumers, its retention) and treat a consumer registering interest as a reviewed change. Hidden consumers are the failure that bites later: the producer changes a field, an unknown subscription breaks, and the incident begins hours after the deploy. Replay compounds this, since a consumer reading old events has to understand every historical version still in the log.

Every subscriber should assume a message can be delivered more than once unless the broker and the scope of the processing explicitly rule it out. The duplicate comes from a window that cannot be closed from the consumer's side: it finishes the work, then fails before the acknowledgment (an offset commit, an ack, a delete) reaches the broker. The broker cannot distinguish that from a consumer that never did the work, so it delivers again. A rebalance in the middle of a batch causes the same thing. Not every system offers more than at-most-once; Redis Pub/Sub, for example, loses messages instead of repeating them. The delivery guarantees, and why exactly-once effects are built from at-least-once delivery plus idempotency, are covered in Queues & Workers and Idempotency.

Here the consumer polls offset 41, runs the database transaction that records the event ID and applies the reservation, and the database commits. It then crashes before committing the offset, so the group's position is still 41. After restart the broker delivers 41 again; the dedupe insert conflicts, the effect is skipped, and the consumer commits 42. The delivery was duplicated and the effect was not. The key is that the dedupe record and the effect commit atomically in the same database.

CREATE TABLE processed_events (
  consumer     text        NOT NULL,
  event_id     text        NOT NULL,
  processed_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (consumer, event_id)
);

CREATE TABLE stock_reservations (
  order_id         text    NOT NULL,
  reservation_type text    NOT NULL,
  sku              text    NOT NULL,
  quantity         integer NOT NULL CHECK (quantity > 0),
  PRIMARY KEY (order_id, reservation_type)
);

BEGIN;
WITH first_time AS (
  INSERT INTO processed_events (consumer, event_id)
  VALUES ('inventory', 'evt_8f2d')
  ON CONFLICT DO NOTHING
  RETURNING 1
)
INSERT INTO stock_reservations (order_id, reservation_type, sku, quantity)
SELECT 'ord_1042', 'order', 'sku_77', 1
FROM first_time
ON CONFLICT (order_id, reservation_type) DO NOTHING;
COMMIT;

consumer = subscribe(topic="orders.events", group="inventory", auto_commit=false)

loop:
    records = consumer.poll()
    for r in records:
        handle(r)            # one DB transaction: dedupe row + effect (above)
    consumer.commit()        # committed position = last handled offset + 1

# Committing BEFORE handling flips the failure window: a crash loses the
# event (at-most-once) instead of repeating it (at-least-once).

The boundary of this guarantee matters. A database transaction covers rows in that database. It does not cover an email provider, a payment API or the broker. If the handler also calls an external API, crashing after the call and before the dedupe row commits repeats the call. For effects outside your database, pass a stable idempotency key (the event_id works well) to the downstream system, or accept and document that a duplicate is possible. Kafka transactions give exactly-once semantics for a read-process-write cycle entirely inside Kafka; they do not extend to your database or an external call. Treat "exactly once" as a property of a bounded scope, never of an end-to-end pipeline.

The dedupe table grows forever unless you decide a horizon. Keep entries at least as long as the broker could redeliver or a replay might run, then expire them; if you do replays beyond that window, the business key constraint is the backstop. Publishing has the same duplicate window on the other side: a producer that writes to the database and then to the broker can lose the second step, which is the problem the Transactional Outbox exists to solve.

One slow analytics subscriber should not stall inventory, and with independent subscriptions it does not: each has its own position (or its own queue) and falls behind on its own. That is the real benefit of fan-out. It is not total isolation. A shared broker, a shared storage quota, a hot partition or a publishing burst can still couple subscribers' performance. In a queue-per-subscriber broker like RabbitMQ, a stalled subscriber's queue grows until memory or disk limits or a queue length limit starts to hurt the node every other queue lives on. Loose coupling does not mean zero shared failure modes. When one subscriber is too slow, the correct responses live in backpressure and bulkheads, not in the producer.

What to watch, per subscription rather than per topic:

• Consumer lag and, more usefully, the age of the oldest unprocessed event relative to retention.
• Publish errors and publish latency on the producer side, and event size.
• Deserialization failures, which are the earliest sign of a schema problem.
• Rebalance churn: frequent group rebalances mean consumers keep pausing and re-reading.
• Consumer throughput against the arrival rate, and dead-letter arrivals.

Zero lag can also lie. If the producer stopped publishing because its outbox relay is stuck, every consumer looks perfectly caught up while the business is silent. Pair consumer lag with a producer-side signal: the rate of events published, or a periodic heartbeat event.

Poison messages

A malformed or unprocessable event leaves a consumer two bad choices: skip it silently (data loss) or retry forever (the partition or queue is blocked behind it). Retries with backoff deal with transient failures; for one that will never succeed, the message must be parked somewhere visible. RabbitMQ has dead-letter exchanges and SQS has a redrive policy to a dead-letter queue; Kafka has no built-in dead-letter topic, so a consumer publishes the failed record to one itself. Configure one per subscription, alert on arrivals, and plan the replay path. That is the subject of Dead-Letter Queues.

The order service commits the order and an outbox row in one database transaction. A relay publishes OrderPlaced to orders.events, keyed by order_id. Three groups read it.

• Inventory (group inventory, four instances) writes a reservation guarded by the unique (order_id, reservation_type) constraint from the previous section, so a duplicate or a replay cannot reserve twice.
• Notifications (group notifications) records the email send keyed by event_id and passes the same key to the email provider, so a redelivery does not send a second message.
• Analytics (group analytics) updates a projection it can rebuild by replaying retained events with its own offsets.

Suppose the notification service is down for six hours. The order transaction is unaffected, inventory keeps reserving stock, and analytics keeps projecting. The notification group's lag grows, then drains when it returns, as long as six hours is well inside retention and the oldest unprocessed event is still readable. Nobody had to change or alert the order service. If the notification group is down for longer than retention, the situation becomes the one in the retention figure above, and the missed confirmations must be reconstructed from the orders table rather than the log.

What this design does not do is coordinate the three. If a reservation must succeed before the order counts as confirmed, reserving stock is a step in a business workflow with an owner and a failure path, not an optional observer of an event. Pub/sub on its own does not give distributed transaction consistency; when you need ordered steps, compensation and an explicit final outcome, reach for a saga. The same shape appears wherever one fact has several independent readers: a ride-hailing TripCompleted feeding billing, receipts and rewards, or a profile change feeding search indexing and recommendation refresh. In every case the producer stays ignorant of the readers.

Use pub/sub when several independently owned components need to react to the same event, when replay has value, or when the producer should not know its consumers. Use a task queue for one-owner work distribution. Use a synchronous call when the caller needs an immediate, authoritative result: "is this card valid?" and "what is this user's balance?" are questions, and an event is an answer nobody asked for. Prefer a workflow engine or saga when the business process needs ordered steps, compensations, deadlines and an explicit outcome.

Follow the questions from the left. If the caller needs an answer before it can reply, call synchronously. Otherwise, if several owners react to the same fact, publish it to a topic. Otherwise, if the work has ordered steps or must be undone on failure, use a workflow. Otherwise, one owner doing the work is a queue. These are not exclusive. A common production shape is a topic that fans out to one queue per subscriber (SNS to SQS, or an exchange to bound queues in RabbitMQ), which gives each service both its own copy and competing workers with per-message acknowledgment; and workflows routinely publish events about their own progress.

The cost of the pattern is the cost of indirection. Debugging a flow means following an event through consumers that are not in the producer's code, so you need correlation IDs carried in the event and an event catalog. Behavior is eventually consistent: a consumer 500 ms behind reflects state from 500 ms ago, and your design must tolerate that. And it requires discipline that direct calls do not: idempotent consumers, a schema policy and a retention policy that someone owns.

• Pub/sub separates producer identity, timing and deployment, not the event contract: schema evolution and an owned event catalog are still required.
• Fan-out copies between subscriptions and competes within one. A topic that fans out to one queue per subscriber (for example SNS to SQS) gives each service its own copy plus competing workers.
• Order is a per-key property bought with the partition key; choose a key such as order_id so related events share a partition, and accept no relative order across keys.
• Durable has a limit: a consumer offline longer than retention loses events, so alert on the age of the oldest unprocessed event relative to retention and choose the offset-reset behaviour deliberately.
• Delivery is at-least-once, so consumers acknowledge after the effect and deduplicate by event id; replay re-triggers side effects unless handlers are idempotent.
• Pair consumer lag with a producer-side signal, because zero lag can mean the producer stopped publishing.

Publishing reliably is the first gap: see the Transactional Outbox for getting a database change and its event out together. How a single subscription's workers acknowledge, redeliver and scale is in Queues & Workers, and what to do with the events that never succeed is in Dead-Letter Queues. If you are drawn to treating the retained log as the source of truth, event sourcing is the next step, and the Kafka fundamentals module goes deeper into partitions, offsets and consumer groups.