Notification System Trade-offs & Decisions

Immediate delivery confirmation. Simple to reason about. But: producer latency becomes coupled to provider latency, provider slowness cascades upstream, there is no burst absorption, and a provider failure becomes an API failure.

Producer returns in milliseconds. Burst absorption. Provider isolation. Independent scaling. But: delivery is eventually consistent, and the producer cannot know the delivery outcome synchronously.

Simple. But during a provider failure, retries amplify load exactly when the provider is least able to absorb it — a retry storm.

Wait time roughly doubles per attempt; random jitter prevents synchronized retries across workers. A strong default for transient errors, though it must be paired with retryable/permanent error classification (see below) and an overall notification deadline so backoff doesn't quietly delay a time-sensitive message past the point of usefulness.

After N attempts, move to the DLQ. Prevents unbounded retry accumulation and makes failures observable rather than silently repeating forever.

Simple integration, one set of API semantics, one rate limit to reason about. But a provider outage becomes a full channel outage.

Primary/fallback routing gives higher resilience. But different APIs, different delivery semantics, and more complex monitoring/billing — and, critically, failover after an ambiguous primary-provider outcome can duplicate delivery (see Provider Failover vs Retry Primary below).

The queue delivers at-least-once; workers make internal state transitions idempotent, use idempotency keys at the producer/API and provider boundaries where available, and reconcile using provider callbacks/status APIs. This is a practical, achievable target for internal processing.

Cannot be guaranteed once an external provider is in the loop. The ambiguous-outcome race (provider accepts, worker crashes before recording success, message is redelivered) means a retry can produce a genuine duplicate delivery, no matter how good the internal idempotency machinery is.

Minimizes latency for critical traffic in the common case — the highest tier is always served first when anything is waiting. But if the top tier sustains enough volume to never fully empty, lower tiers can be starved indefinitely, not just delayed.

Guarantees bounded progress for every tier — no class waits forever — at the cost of a small amount of scheduler and capacity-planning complexity, and slightly less aggressive best-case latency for the top tier under sustained high-priority load.

Simpler topology, fewer moving pieces to operate. But weaker isolation — a slow or backed-up channel can compete for the same consumer capacity as a healthy one, and a provider-specific rate limit can't cleanly apply to just the traffic that needs it.

Independent scaling per channel, cleaner retry/backoff tuning per channel, and clean isolation of one provider's rate limit or outage from other channels. Trade-off: more queues, more consumer groups, and more operational surface to monitor.

The API writes to the database, then separately publishes to the broker. Simple to implement, but has two distinct partial-failure windows: the DB write succeeds and the publish fails (the notification is recorded but never processed), or the publish succeeds and the DB write fails (a worker processes something with no durable record).

The transactional outbox pattern: the notification and an outbox row are written in one database transaction; a separate publisher reads unpublished outbox rows and sends them to the broker, retrying until acknowledged. This closes the dual-write gap — but adds a publisher process, outbox table storage, and a cleanup/retention strategy for published rows as new operational surface.

A single shared counter (e.g. a token bucket in a fast shared store) gives an accurate, global view of remaining provider capacity across all workers. Cost: a network round-trip (and potential contention) on every send attempt.

Each worker enforces its own slice of the limit with no coordination. Simple and fast, but inherently inaccurate — the sum of independently-enforced local limits can still exceed the provider's actual ceiling, or leave capacity unused if traffic is unevenly distributed across workers.

A middle ground: workers periodically lease a slice of the total budget from a central coordinator and spend it locally between leases. Reduces round-trip frequency versus a fully centralized limiter, while staying more accurate than pure per-worker limiting — at the cost of some quota sitting idle in an underused lease at any given moment.

Improves apparent availability — the user gets a delivery attempt through a different path right away. But if the primary's outcome was ambiguous (a timeout, not a known rejection), the primary may have already accepted the send, and failing over risks a duplicate delivery to the end user.

Safer with respect to duplicate risk when the primary is only transiently degraded, but delays delivery further if the primary is genuinely down for an extended period, and it doesn't help at all if the primary's outage is sustained.

The rendered content (or a reference to the exact template version used) is fixed at creation time. Guarantees a delayed or retried notification sends the content the recipient expects based on when it was triggered — a template edit made afterward cannot silently change in-flight messages. Costs slightly more storage per notification and means a legitimate late content fix can't retroactively apply to already-enqueued messages.

Always uses the current template/variables at the moment of send. Lets a corrected template apply to everything not yet sent — useful for fixing a broken template mid-flight. Risk: a notification that sits in a retry queue for an extended period can be delivered with content that has since changed, which may not match the context the notification was originally triggered for.