Notification System Architecture Deep Dive

The Notification API accepts notification requests from producers and durably enqueues them for asynchronous processing. It should acknowledge within milliseconds.

On receipt: validate the notification (recipient, content, channel, priority). Assign a unique notification ID / idempotency key. Durably persist intent and enqueue the work. Return the notification ID to the producer for status tracking. The API never calls the provider directly.

Between "the API said yes" and "the notification is safely queued for a worker to pick up" lies a real distributed-systems problem: two separate systems (the database and the message broker) must both end up in a consistent state, and there is no single transaction that spans both of them.

The ingestion sequence

validate → assign notification/idempotency ID → persist intent/state → enqueue work → acknowledge. Each step must complete, in order, before the API tells the producer "accepted".

The dual-write problem

If the API writes to its own database and separately publishes to the queue, there are two independent failure windows: (1) the database write succeeds, but the queue publish fails or times out — the notification exists in state but no worker will ever pick it up; (2) the queue publish succeeds, but the database/state write fails — a worker may process a notification that was never durably recorded, breaking status tracking and idempotency bookkeeping. Neither order is safe on its own, because a crash or network partition between the two writes leaves the system in an inconsistent state with no atomic way to detect or repair it after the fact.

The transactional outbox pattern

One well-established solution: the transactional outbox pattern — write the notification row and an outbox row (representing "publish this event") in the same database transaction. A separate outbox publisher process reads unpublished outbox rows, sends them to the broker, and marks them published — retrying on failure until the broker acknowledges receipt.

Duplicates can enter this system at three distinct boundaries, and each boundary needs its own defense — there is no single dedup check that closes all of them.

Producer → API

A producer's network call can time out even though the API actually processed it, prompting the producer to retry — creating a duplicate logical request for the same intended notification. Defense: an idempotency key supplied by the producer (or derived deterministically from the request), which the API uses to recognize "I've already accepted this exact request" and return the original result instead of creating a second notification.

Queue → Worker

At-least-once message brokers can redeliver a message the consumer already processed — after a crash before acknowledgment, a visibility-timeout expiry, or a rebalance. Defense: worker processing and state transitions must be idempotent — applying the same message twice should produce the same end state as applying it once, not a duplicate side effect.

Worker → Provider — the boundary that cannot be fully closed

This is the boundary the rest of the system's guarantees cannot fully protect. Walk through the race explicitly: (1) a worker receives a notification; (2) it checks idempotency/dedup state and finds no prior record; (3) it calls the external provider; (4) the provider accepts the request and (conceptually) begins sending; (5) the worker crashes, or loses the network response, before it can record that the call succeeded; (6) the queue redelivers the notification because the original message was never acknowledged; (7) the next worker checks dedup state — it finds no record of success, because the crash happened before recording — and cannot know with certainty whether the first external side effect actually occurred; (8) retrying at this point may produce a duplicate delivery to the end user.

Reduce (not eliminate) this risk with: provider idempotency keys where the provider supports them (so a retried request to the same provider is deduplicated on the provider's side); persisting the provider's request/message ID as soon as it's returned, even before full confirmation; and reconciling delivery state using provider callbacks/status APIs after the fact, where available, rather than trusting only the synchronous call outcome.

Five distinct kinds of "duplicate"

Precision matters here: duplicate event ingestion (the same producer request submitted twice), duplicate queue processing (a broker redelivering one message), duplicate provider requests (a worker calling the provider more than once for one logical notification), duplicate end-user delivery (the recipient actually receives the message twice), and idempotent processing (a system correctly absorbing a duplicate without a duplicate side effect) are five different concepts. A design can guarantee the first, second, and fifth; it can only reduce, not guarantee away, the third and fourth.

Durable queues buffer notifications between ingestion and delivery. How they are organized — and what ordering guarantee they provide — is a real design decision, not an implementation detail.

Topology options

Per-channel queues isolate push, email, and SMS so one channel's backlog or failure never blocks another. Channel + priority queues add a second dimension, so within push (for example) a security-alert lane is separate from a promotional lane. Logical topics/partitions (as in Kafka) allow horizontal consumer scaling while preserving ordering within a partition. Retry/delay queues hold messages that failed and are waiting for their next scheduled attempt, separate from the main intake queues.

Ordering — most notifications don't need it

Most notifications are independent of one another and do not require global (or even per-topic) ordering — a shipping update and a promotional email can be processed in any relative order without breaking correctness. If a specific product requirement demands per-recipient ordering (for example, a sequence of related account-security messages that must display in the order they were generated), the standard technique is partitioning by an appropriate key such as recipient ID, so all of one recipient's messages land on the same partition and are processed in order relative to each other. The trade-off: partitioning by a hot key (a very active recipient, or a poorly chosen key) reduces effective parallelism for that partition and can create a localized backlog even while other partitions are idle.

"Always process high-priority first, and only touch normal when high is empty" sounds correct, but it has a real failure mode: if high-priority traffic never actually empties (a busy promotional period generating a steady trickle of security alerts, for example), normal and low-priority queues can be starved indefinitely — not merely delayed, but never served.

Production alternatives to strict priority

Weighted polling processes a fixed ratio across tiers per scheduling round (for example, 8 high : 2 normal : 1 low) rather than draining one tier to zero before touching the next. Reserved capacity dedicates a guaranteed minimum share of workers or throughput to each lower tier regardless of high-priority volume. Weighted fair scheduling generalizes this with configurable weights per class, more precisely trading off latency guarantees against fairness. Aging increases a message's effective priority the longer it waits, guaranteeing that even low-priority messages eventually cross the threshold to be served, bounding worst-case wait time. Separate worker budgets (dedicating a subset of workers exclusively to each tier) achieve a similar effect to reserved capacity, at the cost of losing some elasticity across tiers.

Before a notification is actually eligible for delivery, a real system must resolve recipient policy — and this resolution belongs in the routing path, not as an afterthought bolted onto the worker.

What must be resolved before delivery

Is the notification's category enabled for this recipient (security/transactional notifications are typically never fully opt-out-able; promotional notifications usually are)? Is the specific channel allowed (has the user disabled push but kept email)? Has the recipient opted out entirely? Are quiet hours in effect for this recipient's locale/timezone, and does this notification's urgency override them? Is the destination itself valid (a verified email, a non-expired push token, a real phone number)? Is a fallback channel allowed if the preferred channel is unavailable? And does this notification's category (security/transactional vs. promotional) carry a different policy for whether preferences can suppress it at all?

The routing shape

notification intent → recipient policy/preferences → eligible channel(s) → channel queue/worker. An "auto" channel selection ("deliver this however is best for the user") must itself be policy-driven — it resolves through the same preference/eligibility checks, it does not bypass them.

Push notifications route to specific device tokens, not directly to users — and those tokens have their own lifecycle that the routing layer must track.

What has to be tracked

One user commonly has multiple devices (phone, tablet, desktop browser) each with its own token. Device/provider tokens rotate over time (app reinstalls, OS-level token refresh) and can silently become invalid. Before sending, the system selects the currently active token(s) for a user rather than assuming a single stable identifier. When a provider's send response reports a token as invalid or unregistered, that specific token should be deactivated so future sends stop targeting it. Logout or explicit device removal should also affect eligibility — a token tied to a session the user logged out of should generally no longer receive that user's notifications.

Notifications are rendered from templates, not hand-written per send — and rendering itself has correctness questions that only show up under real production conditions like delayed delivery.

What a template reference needs

A template ID and version (not just an ID — see below for why), the variables to interpolate (recipient name, order number, amount, and so on), the recipient's locale (so the correct translated copy and formatting are used), channel-specific rendering (an email template has subject/HTML/plaintext parts; a push notification has a short title/body with strict length limits; an SMS has a plaintext body with its own length constraints), and validation that all required variables were actually supplied before the notification is queued, not discovered missing at send time.

The delayed-render question

If a notification sits in a queue for retry or backlog reasons and the underlying template changes in the meantime, should the eventual delivery use the template version that existed when the notification was created, or the current version? There is no universally correct answer — but the system needs to make this an explicit decision. The standard approach: reference a specific template version at creation time (or snapshot the rendered content at creation time), so a mid-flight template edit does not silently change in-flight notifications' content.

Per-channel workers dequeue notifications, resolve policy/preferences, apply rate limits, and deliver to providers.

Worker responsibilities: (1) dequeue from the channel/priority queue; (2) resolve preferences and eligibility (see Preferences & Channel Eligibility); (3) check idempotency/dedup state — but recognize this reduces, rather than eliminates, duplicate risk (see Idempotency & Duplicate Suppression); (4) acquire a provider rate-limit token; (5) deliver to the provider and persist the provider's request/message ID as soon as it is returned; (6) update delivery state to reflect provider acceptance, distinct from confirmed delivery; (7) on failure, classify the error and either schedule a retry with backoff or route to the DLQ.

"Acquire a rate limiter token before calling the provider" is easy to say and considerably harder to implement correctly once many worker processes, potentially across multiple hosts, are all sharing one provider account's quota.

What actually needs to be rate-limited

Provider/account-level quota (a single SMS or email account has one contractual send rate, regardless of how many workers exist). Channel-level quota, which can differ from the account quota if a provider segments limits by message type. Tenant or destination-level limits, where relevant — for example, capping how fast any single recipient can be messaged to avoid abuse or accidental floods.

Shared vs. leased/local quota

A centralized shared limiter (for example, a token-bucket counter in a fast shared store) gives an accurate global view of remaining capacity but adds a network round-trip and a potential contention point on every send. A leased/local quota approach periodically grants each worker a slice of the total budget to spend locally, reducing round-trips at the cost of some imprecision — a worker with an unused lease is idle capacity, while another worker may be artificially blocked despite quota being technically available elsewhere. This trade-off is expanded in Trade-offs & Decisions.

Handling 429s and dynamic throttling

Providers can throttle dynamically — a 429 response, or a provider-supplied Retry-After header, is a signal that the assumed rate limit is currently wrong and should be adjusted downward, not merely retried blindly. A well-behaved integration treats provider throttling responses as authoritative feedback that adjusts the local/shared rate-limit estimate, at least temporarily, rather than treating them as an anomaly to retry through at the same rate.

"Wait, then retry" is not implemented as a worker thread sleeping — that would tie up a worker slot for the entire backoff window and does not survive a worker restart. Retry timing is a scheduling problem, not a sleep statement.

How retries are actually scheduled

A delayed/scheduled queue (many brokers support message-level delay, or a separate delay-queue-plus-mover pattern) holds a failed message until its next attempt is due, then makes it visible to workers again. Retry topics/buckets (common with Kafka-style logs, which don't support arbitrary per-message delay) route a failed message to a dedicated retry topic per backoff tier (retry-1, retry-2, ...), each with its own consumer that reads messages after their delay has elapsed. Where the broker supports native scheduled delivery, that mechanism can be used directly instead of a separate delay-queue pattern.

What must be persisted per attempt

Attempt count (to enforce a max-retry limit and to route to DLQ once exhausted), the next-attempt timestamp (so the scheduler knows when to make the message visible again), the last error (for diagnosis and for permanent-vs-transient classification), and an expiry/deadline for the notification itself — because a retry schedule that keeps extending backoff can eventually push a delivery attempt well past the point where the notification is still useful.

Each channel integrates with one or more external providers behind a normalized adapter, so worker code does not need to know provider-specific request/response shapes.

What the adapter layer normalizes

A normalized request/response shape across providers for the same channel. Provider-specific error classification into a shared taxonomy (retryable/transient vs. permanent vs. rate-limited) so worker retry logic doesn't need to know each provider's exact error codes. Ongoing provider health tracking (recent error rate, latency) informing routing decisions. Per-provider rate limits. And persistence of the provider's own message ID for later reconciliation.

Why immediate failover on every timeout is a mistake

A timeout does not mean the request failed — the primary provider may have already accepted the send and simply been slow to respond. Immediately failing over to a secondary provider on every timeout can therefore deliver the same notification twice: once via the (possibly successful) primary, once via the secondary. Failover selection should weigh: whether the primary's outcome is a known rejection (safe to fail over) versus ambiguous (timeout, connection drop — riskier to fail over without further signal); the urgency of the notification (a security alert may justify accepting duplicate-delivery risk to guarantee it reaches the user; a promotional message usually does not); the acceptable duplicate-delivery tolerance for this notification class; current provider health; and relative cost/capability differences between providers.

Many providers report final delivery outcomes asynchronously via webhook callbacks, rather than in the original synchronous send response — and those callbacks bring their own correctness problems.

The reconciliation pipeline

provider → authenticated webhook → normalize → deduplicate callback → state transition. Webhook authenticity should be verified (signature or shared-secret validation) before trusting a callback's contents, since this endpoint is internet-reachable.

What can go wrong with callbacks

Callbacks can arrive duplicated (the provider retries its own webhook delivery), delayed (arriving well after the notification was sent), missing entirely (network issues, provider-side bugs, or a channel/provider that simply doesn't support delivery callbacks), or out of order (a "delivered" event arriving before a "sent" event due to independent retry paths on the provider's side).

How to process them safely

Deduplicate on the provider's own callback/event ID before applying a state transition. Make state transitions idempotent and monotonic — a state machine that only allows forward progress (e.g. SENT → DELIVERED, never DELIVERED → SENT) naturally absorbs an out-of-order or duplicate callback without corrupting the record. For notifications where no callback ever arrives, reconcile with the provider's own status/query API where available, on a reasonable polling cadence, rather than assuming silence means failure or success.

The Complete Design's batch-submission endpoint accepts a request for many recipients at once — but naively expanding that into millions of individual queue writes inside the API request itself would defeat the entire "return in milliseconds" contract.

The fanout pipeline

batch/campaign → recipient enumeration → chunks/fanout jobs → policy evaluation → channel queues. The API accepts the campaign definition and returns immediately with a batch ID; a separate fanout process enumerates recipients (potentially millions of rows) in bounded chunks, and each chunk becomes its own unit of work that resolves preferences/eligibility and enqueues onto the normal channel/priority queues — the same path individual notifications use.

What real fanout needs

Pacing — releasing chunks at a rate the downstream channel workers and provider quotas can actually absorb, rather than flooding the queues instantly (see Massive Campaign Overload in Failure & Scale). Cancellation — an in-progress campaign must be stoppable mid-fanout without requiring every already-enqueued message to be individually pulled back. Expiry — a campaign paused or delayed long enough that its content is stale should not silently resume days later. Isolation from transactional traffic — campaign fanout must never be allowed to compete for the same queue capacity or provider quota that security/transactional notifications depend on; this is exactly why priority queues and (optionally) channel-specific queue topology exist.

A Delivery Tracker persists notification lifecycle state and is the source of truth for customer support, operational dashboards, and reconciliation — its data model needs to be deliberately designed, not an afterthought.

Useful fields per delivery record

notification_id, recipient/channel, current state, attempt_count, the provider used, provider_message_id (once known), relevant timestamps (created, queued, last-attempted, delivered), next_retry_at, last_error, and an expiry/deadline for the notification itself.

State transitions distinguish acceptance from delivery

ACCEPTED/CREATED → QUEUED → PROCESSING → PROVIDER_ACCEPTED/SENT → DELIVERED, with additional terminal/exception states: RETRY_SCHEDULED, FAILED_PERMANENT, EXPIRED, SUPPRESSED (blocked by preferences/policy), DLQ, BOUNCED, UNDELIVERABLE. PROVIDER_ACCEPTED/SENT means the provider took the request — it is explicitly not the same claim as DELIVERED, which means the provider (or a downstream signal) confirmed the message actually reached the recipient.

Storage at volume

At high notification volume, delivery-state records accumulate quickly and have a strongly time-skewed access pattern — recent records are read/written constantly, older records are read rarely if ever. Conceptually, this suggests partitioning by time or notification age, a retention policy for how long detailed per-attempt history is kept in the primary hot store, and an archive path for older records that preserves auditability without keeping every historical attempt in the same hot-path table. The right specific storage technology depends on actual read/write access patterns (point lookups by ID vs. range scans by time vs. aggregate dashboards) — this deep dive does not prescribe one database without that access-pattern context.

Guarantees by boundary