Notification System Failure & Scale Scenarios

The email provider goes down. Workers receive errors. Recovery uses exponential backoff with jitter — but a real outage raises questions backoff alone doesn't answer.

What fails: every send attempt to the affected provider errors out. User impact: email notifications are delayed; push and SMS are unaffected because channel isolation keeps their queues and workers independent. Detection: elevated error rate and rising queue depth specifically on the email channel, plus provider health/status signals. Degradation: the email queue grows; notifications sit durably rather than being lost. Recovery: workers retry with exponential backoff and jitter as the provider recovers; the backlog drains at a paced rate rather than all at once.

Prevention/limits: a growing backlog is only safe within retention, storage, and freshness limits — a queue is not an unlimited buffer, and notifications with a real deadline (a one-time passcode, a time-sensitive fraud alert) should be expired rather than delivered stale once the provider recovers. A secondary provider may be available as a fallback, but only for notifications where a possible duplicate (if the primary's outcome was ambiguous rather than a clean rejection) is an acceptable trade-off — see Trade-offs → Provider Failover vs Retry Primary.

A major event fires a large volume of notifications simultaneously. The queue absorbs the burst. Workers drain at their controlled rate — as long as that rate is actually sustainable.

What fails: nothing, if the burst is genuinely temporary — the queue grows, then drains. User impact: delivery latency rises temporarily; producers are unaffected since they already returned. Detection: queue depth spike, correlated with a known traffic-generating event. Degradation: delivery of this burst's notifications lags behind newly arriving ones until the backlog clears.

Recovery math: a useful first question is the drain-time calculation — backlog size divided by sustainable delivery rate. A 500,000-message backlog draining at an illustrative 2,000/s clears in roughly 250 seconds; the same backlog at 200/s takes over 40 minutes. The hard limit: provider quota can be a hard ceiling — scaling out workers does not help once the bottleneck is the provider's own rate limit, not worker concurrency. Isolation: critical/transactional traffic should be isolated from campaign-style bursts (via priority tiers or channel-specific queues) so a promotional burst does not delay a concurrent security alert.

Provider recovers. All workers retry simultaneously. The synchronized load overwhelms the recovering provider.

What fails: the just-recovered provider gets hit with a large simultaneous spike of retries and fails again. User impact: delivery is further delayed, and the outage effectively extends. Detection: a second wave of errors immediately following an apparent recovery, tightly clustered in time. Degradation: the provider may throttle or reject the spike outright, compounding the original outage.

Without jitter: many workers waited the same backoff interval and all retry within the same narrow window. With jitter: each worker's wait includes a random offset, spreading retries across a window instead of a single instant, so the provider sees a manageable, gradually increasing retry rate rather than a spike.

Prevention: this preserves the earlier distinction between retryable/transient errors (worth retrying with backoff+jitter) and permanent errors (should not retry at all), and respecting the provider's own retry guidance — a Retry-After header after recovery is a strong signal of how much load the provider is actually ready to handle right now.

DLQ depth grows beyond a healthy baseline. Alert fires. Engineers inspect and act.

What fails: an increasing share of notifications is exhausting retries. User impact: those specific notifications are not delivered until someone intervenes. Detection: DLQ growth is best monitored by rate and by channel/provider/error-type breakdown, and by deviation from a rolling baseline — a spike specific to one provider or one error code points at a different root cause than a broad, even increase across the board. Oldest-message age in the DLQ is also worth tracking: a DLQ with very old, unaddressed messages indicates a process gap, not just a technical one.

Degradation: delivery for the affected slice of traffic stalls until root cause is fixed. Recovery: diagnose (provider API breaking change, malformed notification format, persistent provider outage, a bad template deploy), fix the underlying issue, then replay — paced, not all at once — and track replay volume as its own signal (a huge replay can itself look like a burst).

A worker crashes mid-processing and the queue redelivers the same notification. Whether this is safe or ambiguous depends entirely on exactly when the crash happened.

Case A — crash before the provider call

The worker crashes before it ever calls the provider. The queue redelivers; a new worker picks it up, sees no record of a completed send, and safely processes it from scratch. This case is safe — no external side effect happened yet, so reprocessing is simply normal at-least-once redelivery with no duplicate risk.

Case B — crash after the provider accepts, before recording success

The worker calls the provider, the provider accepts the request (and may already be in the process of delivering it), and the worker crashes or loses the response before it can record that success. The queue redelivers the same notification. The next worker checks dedup state, finds no record of success (because the crash happened before recording), and cannot know with certainty whether the first external side effect actually occurred. This case is genuinely ambiguous — retrying here can produce a duplicate delivery to the end user.

User impact: in Case A, none. In Case B, a possible (not certain) duplicate notification. Detection: persisted provider request/message IDs and reconciliation via provider callbacks can retroactively reveal whether Case B actually produced a duplicate. Mitigation: provider idempotency keys where supported prevent the provider itself from acting on a retried request twice, which closes much of this gap where available.

The message broker itself becomes unavailable — not the provider, the queue infrastructure sitting between ingestion and workers.

What fails: new notifications cannot be published to the queue; workers cannot receive new work. User impact: new notification requests may need to be rejected or degraded at the API layer if durable acceptance genuinely cannot be guaranteed — returning a fake 'accepted' during this window would violate the API's own contract. Detection: publish/consume errors, broker health checks failing, rising outbox-table size if a transactional outbox is in use (see Architecture Deep Dive → Durable Notification Ingestion) — since the outbox will accumulate unpublished rows while the publisher cannot reach the broker.

Degradation: if using a transactional outbox, the database write can still succeed and accumulate durably even while the broker is down — nothing is lost, but nothing is processed either until the broker returns. If not using an outbox, the API may have no safe way to accept new requests during the outage and should communicate this honestly (503-style degradation) rather than silently dropping. Recovery: once the broker returns, the outbox publisher (or direct producer) resumes publishing accumulated rows — paced, not all at once, since a large accumulated backlog can itself look like a traffic burst to downstream workers. Producer-side idempotency keys remain important here, since producers may have retried during the outage.

The provider starts returning 429s or otherwise signals it is throttling — a distinct scenario from a full outage.

What fails: a rising share of provider calls are rejected specifically for rate reasons, not connectivity or server errors. User impact: delivery slows for the throttled channel/provider; if adaptive rate limiting isn't in place, retries can make the throttling worse. Detection: 429 rate by provider/channel, and any Retry-After values the provider is returning.

Degradation and recovery: an adaptive rate limiter should reduce its assumed send rate in response to 429s (and honor Retry-After if provided) rather than continuing to retry at the prior rate. The queue absorbs the resulting backlog growth exactly as in a traffic burst. Critical/high-priority traffic should have protected capacity within the now-reduced rate budget, so a promotional-driven throttling event doesn't also delay security alerts.

The store backing idempotency/dedup checks (recording which notification IDs have already succeeded) becomes unavailable or degraded.

What fails: workers can no longer reliably check or record dedup state before/after a provider call. User impact: the risk of duplicate delivery increases, since the primary defense against redelivery-triggered duplicates is temporarily unavailable. Detection: elevated error rate specifically on dedup-store calls, distinct from provider-call errors.

Degradation options vary by notification class, and should be an explicit policy rather than a single universal fallback: for high-risk/high-consequence notification classes (e.g. anything financial or security-sensitive where a duplicate is costly), consider pausing sends until the dedup store recovers. For lower-risk classes, accepting a temporarily elevated duplicate-delivery risk and continuing to send may be acceptable. Where the provider itself supports idempotency keys, relying on provider-side idempotency as a fallback layer reduces exposure during this window, since it doesn't depend on this system's own dedup store being available.

The delivery-state store — the system tracking CREATED/QUEUED/SENT/DELIVERED and similar states — becomes unavailable or degraded.

What fails: state-transition writes (and status reads) fail or lag. User impact: notification delivery itself need not stop — the tracker is a visibility/observability layer, not (typically) on the critical delivery path in a well-isolated design. Customer support and dashboards temporarily lose accurate status visibility, which is a real but different kind of impact than delivery actually failing.

Detection: tracker write/read error rate; a growing gap between 'notifications processed by workers' and 'state transitions recorded.' Degradation: buffer state updates locally (in the worker, or in a durable local queue) and retry them against the tracker once it recovers, rather than blocking delivery on tracker availability. Preserve provider IDs/results during the outage window so they can still be reconciled once the tracker is healthy again — losing that data would create a permanent gap in delivery history, not just a temporary one.

Provider delivery-status callbacks arrive duplicated, delayed, or out of order — a normal, expected condition for webhook-based reconciliation, not an edge case.

What fails, concretely: a duplicate callback could otherwise apply the same state transition twice (usually harmless if idempotent, but worth designing for); an out-of-order callback (e.g. a stale 'sent' event arriving after a 'delivered' event already landed) could otherwise regress delivery state backward if applied naively; a missing callback leaves a notification's true final state unknown from callbacks alone.

Prevention: deduplicate on the provider's own callback/event ID before applying any transition. Make state transitions idempotent and monotonic — only allow forward progress through the state machine (e.g. never let a SENT callback move a record backward from an already-recorded DELIVERED), so an out-of-order arrival is naturally absorbed rather than corrupting the record. Handle terminal states explicitly (once a notification reaches a genuinely terminal state like DELIVERED, BOUNCED, or FAILED_PERMANENT, further callbacks should be reconciled/logged, not blindly re-applied). For missing callbacks, reconcile periodically against the provider's status/query API where one exists, rather than assuming silence means either success or failure.

High-priority traffic sustains high enough volume that lower-priority tiers effectively never get processed — not just delayed, but indefinitely stuck.

What fails: with a strict 'always serve high first, only touch normal/low when high is empty' policy, a sustained high-priority stream means normal/low messages age indefinitely in their queues. User impact: promotional and even some normal-priority transactional notifications may never be delivered, or delivered so late they've lost relevance. Detection: monitor queue age (oldest-message age), not just queue depth, broken out by priority tier — depth alone can look stable while age silently grows for the lower tiers.

Recovery/prevention: replace strict priority with weighted polling, reserved capacity, or aging (increasing a waiting message's effective priority the longer it waits, guaranteeing it eventually crosses the serve threshold). Where appropriate, expire genuinely stale low-value messages (e.g. an old promotional message that's no longer relevant) rather than letting them accumulate forever — but this must be a deliberate, product-scoped decision (which classes are safe to expire), not applied to everything uniformly.

A campaign targeting a very large recipient list (tens of millions) is submitted, and naive fanout floods every downstream queue at once.

What fails: if fanout enqueues the entire recipient list immediately, channel queues spike far beyond normal traffic, provider quota is instantly saturated by campaign traffic, and — without isolation — transactional/security traffic queued behind it is delayed. User impact: security alerts and other time-sensitive transactional notifications experience latency they should never experience, purely because of an unrelated marketing campaign. Detection: a sudden queue-depth spike correlated with a known campaign submission; queue age rising specifically for non-campaign traffic sharing the same queue.

Prevention/recovery: chunk recipient enumeration into bounded fanout jobs rather than enqueueing everything at once (see Architecture Deep Dive → Batch/Campaign Fanout). Pace chunk release to match sustainable delivery/provider capacity. Isolate campaign traffic from transactional traffic using separate queues and/or reserved capacity for critical traffic, so campaign volume physically cannot crowd it out. Support cancellation mid-fanout so a mistaken or urgent-to-stop campaign doesn't have to fully drain before it can be halted. Monitor queue age throughout the campaign, not just at submission.

A meaningful fraction of push sends fail because the device token is no longer valid — the app was uninstalled, the OS rotated the token, or the token was never valid to begin with.

What fails: the provider (FCM/APNs) returns a specific 'invalid/unregistered token' error for these sends. User impact: none directly to the affected user — but if not handled, the system wastes retry budget on sends that can never succeed, and dashboards/alerts can be skewed by a steady stream of a failure type that isn't actually actionable in the usual sense.

Detection: provider error taxonomy should classify this specific error as permanent, not transient. Recovery/prevention: treat invalid-token responses as non-retryable — deactivate that specific token immediately (see Architecture Deep Dive → Push Device Token Management) rather than retrying or DLQ-ing it as if it might succeed later. Since one user commonly has multiple devices, deactivating one invalid token should not affect delivery to that user's other active devices.

After an extended provider outage or broker outage, a very large backlog has accumulated. Releasing all of it at once on recovery recreates the original incident.

What fails: if every backlogged message is made available to workers simultaneously the moment the provider/broker recovers, the resulting flood can look identical to (or worse than) the original traffic burst or retry storm — potentially triggering a fresh round of provider throttling right as recovery was supposedly complete. User impact: the very notifications waiting in backlog are further delayed by a fresh self-inflicted overload, and some may now be stale enough that delivering them at all is no longer useful.

Detection: this is a predictable, plannable scenario, not a surprise — monitor backlog size continuously during any outage so the scale of the eventual recovery is known in advance. Recovery: release the backlog paced to match provider capacity (the same drain-time math as Traffic Burst), respect priority ordering so higher-priority backlogged messages are released first, and check each backlogged message's own expiry/deadline before releasing it — a stale time-sensitive notification should be expired, not delivered hours late just because the backlog finally cleared.

Triage map