Designing a Notification System

Design a system that reliably delivers push notifications, emails, and SMS messages at high volume — across traffic bursts, provider rate limits, and partial failures.

Services across your platform need to send notifications: order confirmations, security alerts, payment failures, shipping updates, promotional campaigns. The volume is high — potentially millions per hour. Some notifications are critical (security alerts must arrive immediately). Others are promotional (important but not urgent). Providers — FCM, SendGrid, Twilio — have rate limits and fail occasionally.

Functional and non-functional requirements that shape every architectural decision.

Functional Requirements

Accept notifications from any producer service (via API or events). Route to the correct delivery channel: push, email, or SMS. Handle priority — security alerts should get preferential treatment without indefinitely starving lower-priority traffic. Retry failed deliveries with exponential backoff and jitter, distinguishing transient from permanent failures. Dead-letter messages that exhaust retries (no silent drops). Reduce duplicate sends aggressively at every internal boundary. Track delivery status through the complete lifecycle.

Recipient & Channel Policy

Before a notification can be delivered, the system must resolve recipient policy: user notification preferences (which categories and channels the recipient has enabled), opt-out/unsubscribe state, current channel availability (is the destination valid — a verified email, a live phone number, a non-expired push token), quiet hours where applicable, and the notification's own category (security/transactional notifications are typically never fully suppressible; promotional notifications usually are). Invalid or expired destinations, such as stale push tokens, must be detected and excluded from delivery rather than attempted and retried indefinitely. (Legal/regulatory opt-out requirements exist for some channels — kept generic here, not a compliance tutorial.)

Non-Functional Requirements

High throughput — millions of notifications per hour. Low producer latency — the calling service must return immediately once its request is durably accepted. Reliable delivery — durable at-least-once processing with aggressive but not guaranteed-complete duplicate suppression (see Deduplication). Channel isolation — email provider failure must not block push delivery. Observability — every notification's state is queryable.

Capacity estimate (illustrative)

Illustrative planning numbers, not measurements of any real system. Assumption: "millions per hour" means 10 million notifications/hour averaged over a day, with a peak 5× the average. Average rate: 10,000,000 ÷ 3,600 s ≈ 2,800/s. Peak rate: 2,800 × 5 ≈ 14,000/s. Daily volume: 10M × 24 = 240 million notifications/day. Implication: against the 1,000/s push-provider ceiling used as an example in Rate Limiting & Backpressure, even the average rate is nearly 3× one provider quota if all of it were push, so per-provider quota — not worker count — is the binding constraint, and the queue only bridges a gap that closes again.

Before designing the internals, define the contract producers depend on.

The simplest design: the producer calls the Notification Service, which calls the provider directly and waits for a response.

Works fine when the provider is fast. But when the email provider slows to 8-second p99 latency, the Notification Service blocks — and every producer calling it blocks too. An order service trying to send a confirmation email is stalled by a slow SMS provider. The producer's latency is now coupled to the provider's latency.

The producer writes to a durable queue and returns in milliseconds. Workers dequeue and deliver to the provider independently.

The producer no longer waits for the provider. If the provider is slow — the queue absorbs the backlog. If the provider goes down — the queue retains messages until recovery. Ingestion rate and delivery rate do not have to be the same. The queue is the buffer between them.

A large event fires ten thousand notifications simultaneously. The queue absorbs the burst while workers drain at a safe rate.

Five thousand notifications per second arrive at the queue. Workers deliver at eight hundred per second — the safe rate for the provider. The queue grows. That's fine. It's doing its job. The producers returned milliseconds ago. They don't know or care about the backlog.

Real notification systems deliver over multiple channels: push (cheap, fast), email (medium latency), SMS (expensive, rate-limited).

A Channel Router resolves recipient policy before dispatch: notification intent → recipient policy/preferences → eligible channel(s) → channel queue/worker. This means checking whether the category is enabled, the channel is allowed, the recipient hasn't opted out, quiet hours don't apply, and the destination is valid — before a channel worker ever sees the message. An 'auto' channel selection (letting the system pick the best channel) is not a shortcut around this — it must resolve through the same policy check, choosing among whichever channels are actually eligible. Each channel worker then talks to its own provider. If the email provider goes down — push and SMS continue unaffected. Channels fail independently. They scale independently. Each channel can have its own queue depth, concurrency, and rate limit.

Without any priority mechanism, a security alert joins the back of a large promotional backlog. At the eight hundred per second delivery rate used above (illustrative), a ten-thousand-item backlog delays it about 12 seconds, and a campaign backlog of roughly 860,000 items delays it about 18 minutes.

Separate queues per priority tier: High (security alerts, payment failures), Normal (order updates, shipping notifications), Low (promotions, newsletters). A naive policy — always process high first, only touch normal when high is completely empty — sounds correct, but has a real failure mode: if high-priority traffic is sustained enough to never fully empty, normal and low queues can be starved indefinitely, not just delayed. Production systems instead use weighted polling (serve a fixed ratio across tiers per round), reserved capacity (guarantee each tier a minimum worker/throughput share), or aging (increase a waiting message's effective priority over time so it eventually crosses the threshold to be served).

Workers want to send 5,000/s (an illustrative assumption). The push provider allows 1,000/s (also illustrative). Without rate limiting: HTTP 429 errors, blocked accounts, degraded delivery.

These are two related but distinct concepts. Rate limiting controls the rate at which workers are allowed to call the provider — it protects the provider's capacity directly. Backpressure is the system's broader response when downstream capacity is lower than incoming demand — the queue growing is the visible form of backpressure here. The queue absorbs the temporary mismatch between arrival rate and sustainable delivery rate; it does not, by itself, create additional provider capacity. If arrival rate stays above the provider's sustainable delivery rate indefinitely (not just in a temporary burst), the queue grows without bound no matter how it's tuned.

This is why retention and a maximum acceptable notification age matter as explicit design inputs: a queue is only a useful buffer up to the point where its oldest messages are still worth delivering. Beyond that, sustained excess arrival needs a different answer — provider capacity increase, traffic shedding/prioritization, or isolating and pacing lower-priority sources like campaigns (see Batch/Campaign Fanout in Architecture Deep Dive and Massive Campaign Overload in Failure & Scale).

The email provider goes down. Workers retry immediately. A large volume of retry requests hits the recovering provider at once. It fails again. This is a retry storm.

The fix: exponential backoff with jitter. First failure: wait ~1s. Second: ~2s. Third: ~4s (illustrative values). And jitter: add a random offset to each interval. Without jitter, all workers retry at exactly the same moment — synchronized spikes. With jitter, retries spread across time.

Not every failure should retry the same way. Distinguish retryable/transient errors (timeouts, 5xx, temporary throttling) from permanent errors (invalid destination, malformed request, permanently rejected) — permanent errors should go straight to DLQ or a suppressed state, not consume retry budget. Where a provider supplies retry guidance, such as a Retry-After header on a 429, respect it rather than retrying on a fixed schedule that ignores the provider's own signal. And every retryable notification needs a deadline: if a notification has a natural expiry (a one-time passcode, a time-sensitive alert), a backlog that takes hours to drain should not result in it being delivered stale — the retry scheduler must check the deadline before attempting delivery, not just the retry count.

After exhausting retries, failed messages go to the DLQ. No silent drops. Every failure is observable and replayable.

The DLQ is not a trash can — it's an observable failure store. Engineers can inspect it, diagnose root cause, and replay messages once the underlying issue is fixed. The key property: every notification either delivers, expires, is suppressed by policy, or ends up in the DLQ where a human can act on it.

At-least-once processing means duplicates can arrive. A dedup store keyed by notification ID reduces double-sending, but does not make it impossible.

Before delivering: check if this notification ID has already recorded a successful send. Yes? Skip. No? Proceed, and record success immediately after the provider accepts the request. This meaningfully reduces duplicates, but it is not a guarantee: if a worker crashes or loses the network response after the provider accepts the call but before it can record that success, the next delivery attempt cannot tell whether the first send actually reached the provider. Retrying in that specific window can still produce a duplicate. Reinforce this with a producer/API idempotency key (for retried client requests), provider idempotency keys where the provider supports them, and reconciliation via provider callbacks/status APIs after the fact. The honest contract is durable at-least-once processing with aggressive duplicate reduction — not exactly-once end-user delivery. See Architecture Deep Dive → Idempotency & Duplicate Suppression for the full race.

Every notification has an observable lifecycle — and that lifecycle needs to distinguish the provider accepting a request from the message actually reaching the recipient.

Conceptual states: ACCEPTED/CREATED → QUEUED → PROCESSING → PROVIDER_ACCEPTED/SENT → DELIVERED, with additional states for the paths that don't end in a clean delivery: RETRY_SCHEDULED, FAILED_PERMANENT, EXPIRED, SUPPRESSED (blocked by preferences/policy), DLQ, BOUNCED, UNDELIVERABLE. PROVIDER_ACCEPTED/SENT is not the same claim as DELIVERED — it means the provider took the request, not that the recipient received it. Not every channel or provider supplies a reliable DELIVERED receipt; some (notably SMS in many setups) can only confirm as far as carrier handoff.

Every component was justified by a specific failure mode or correctness gap. Nothing was added speculatively.

Notification API + durable ingestion: durably accepts intent (via transactional outbox or equivalent), returns fast. Notification/state store: the source of truth for lifecycle state. Priority/channel queues: fair scheduling prevents starvation; channel isolation contains failure. Policy/preferences routing: resolves eligibility before a channel worker ever sees the message. Device-token data: tracks which push destinations are currently valid. Channel workers: stateless, per-channel delivery logic. Provider rate limiter: shared or leased quota protects each provider's actual ceiling. Retry scheduler/queues: schedules retries without tying up workers, respects deadlines and retryable/permanent classification. DLQ: every unrecoverable failure is observable and safely replayable. Idempotency/dedup store: reduces — not guarantees away — duplicate delivery. Provider adapters: normalize requests/errors and support cautious failover. Provider callback/webhook ingestion: reconciles delivery status asynchronously. Delivery tracker: distinguishes acceptance from confirmed delivery across the whole lifecycle.