Interview Perspective

Clarify scope before designing: which notification classes need different latency/SLA guarantees, whether recipients have preferences to honor, whether delivery tracking is required, and how promotional traffic differs from security/transactional traffic in policy terms. Then state the central decoupling principle before touching any component.

• Which classes need different latency/SLA? — Tests whether the candidate treats all notifications as equally urgent, or recognizes security/OTP traffic has a materially different latency requirement than a promotional campaign.
• How do promotional and security notifications differ in this design? — Tests whether the candidate connects this distinction to concrete mechanisms — priority tier, opt-out eligibility, retry/expiry policy — rather than treating it as a labeling exercise.

Yes in a realistic design — category enablement, channel allowance, opt-out state, and quiet hours should all be resolved before a channel worker acts, as part of routing. See Complete Design -> Multi-Channel Routing and Architecture Deep Dive -> Preferences & Channel Eligibility.

• What if preferences are cached and change while a notification is queued? — Tests awareness of the stale-preference risk — a cached read at enqueue time can be outdated by actual delivery time.

Because ingestion rate and delivery rate are fundamentally different and shouldn't be coupled — the producer needs to return fast regardless of provider health, and the queue is what makes that possible. See Complete Design -> Introduce the Queue.

• What if ingestion consistently exceeds delivery capacity, not just in a burst? — Tests whether the candidate recognizes a queue only absorbs temporary mismatch — sustained excess arrival needs a different answer (more provider capacity, shedding, reprioritization), not just a bigger queue.

Per-channel (and often per-priority) queues for isolation; most notifications don't need ordering at all, and where per-recipient ordering is genuinely required, partition by recipient ID — accepting the reduced parallelism a hot recipient key creates. See Architecture Deep Dive -> Queue Topology & Ordering.

• How do channels fail independently in this topology? — Tests whether the candidate can explain this concretely — separate queues, separate consumer groups, separate rate limits per channel — not just assert independence.

Separate priority tiers/queues, with workers preferentially serving the higher tier. But 'preferentially' matters — see the next question.

• What if high priority never fully empties? — Tests whether the candidate knows strict priority can starve lower tiers indefinitely under sustained load — this is the key follow-up in this category.
• How do you prevent starvation? — Tests knowledge of concrete mechanisms: weighted polling, reserved capacity, weighted fair scheduling, or aging. See Architecture Deep Dive -> Priority Scheduling & Starvation and Failure & Scale -> Priority Starvation.

This is the core ambiguous-outcome race: the provider may have already accepted the request, so the redelivered message might produce a duplicate send. This cannot be fully prevented with only a pre-send dedup check — it can only be reduced, via provider idempotency keys and reconciliation.

• Can you guarantee exactly-once delivery here? — Tests whether the candidate holds the line: no, not across an external provider, and explaining precisely why (the ambiguous-outcome window) is the strong answer.
• Where would you use idempotency keys? — Tests whether the candidate distinguishes the three boundaries: producer→API, queue→worker (idempotent processing), and worker→provider (provider idempotency keys where supported).

Then that specific worker→provider boundary has no protocol-level protection against a duplicate — the mitigation shifts to persisting the provider's request/message ID immediately and reconciling via the provider's status/callback mechanism where one exists, accepting that duplicate risk for this provider is higher than for one with native idempotency support. See Architecture Deep Dive -> Idempotency & Duplicate Suppression.

• Does this change your provider-failover posture? — Tests whether the candidate connects lack of idempotency support to being more conservative about failover, since duplicate risk there is already higher.

Rate-limit at the provider boundary so workers deliver at 10k/s regardless of their own capacity; the queue absorbs the resulting backlog. Scaling workers further does not help, since the bottleneck is the provider's quota, not worker throughput. See Architecture Deep Dive -> Distributed Provider Rate Limiting.

• How do distributed workers share that 10k/s quota accurately? — Tests knowledge of centralized shared limiter vs. leased/local quota trade-offs, and why naive per-worker limiting under- or over-shoots the real ceiling.

Treat it as authoritative feedback that the current send rate is too high — reduce the assumed rate (respecting a Retry-After header if supplied) rather than retrying blindly at the same rate.

• Why can adding more workers make a 429 situation worse? — Tests the same insight from a different angle: more workers hitting an already-throttled provider just generates more rejected requests and wasted retry traffic.

Pace the release to match the provider's sustainable rate rather than releasing everything at once, respect priority ordering, and check each message's expiry/deadline before delivering — some backlogged messages may no longer be worth sending. See Failure & Scale -> Backlog Recovery Storm.

• What's the risk of releasing the whole backlog at once? — Tests whether the candidate recognizes this can recreate the original outage/throttling condition right as recovery appeared complete.

Transient/retryable errors (timeouts, 5xx, temporary throttling) should retry with backoff; permanent errors (invalid destination, malformed request, permanently rejected, invalid push token) should not retry at all — they should go straight to a terminal failed state or DLQ.

• Why is retrying a permanent error wasteful beyond just delay? — Tests whether the candidate connects this to real cost — wasted retry budget, skewed monitoring, and (for a fixed retry ceiling) crowding out legitimate transient retries.

Backoff alone reduces retry rate over time but doesn't prevent many workers that failed at the same instant from retrying in lockstep. Jitter spreads that synchronized retry mass across a window, which is what actually prevents a thundering-herd spike on a recovering provider.

• When would you stop retrying? — Tests awareness of both a maximum attempt count and a notification's own deadline/expiry as independent stopping conditions.

Fix the root cause first, then replay at a controlled/paced rate rather than all at once, retaining attempt/error history, and verifying each message is still relevant (not expired, recipient hasn't since opted out) before replaying. See Complete Design -> Dead-Letter Queue.

• What if the notification has since expired? — Tests whether the candidate applies the deadline check specifically at replay time, not just at original send time.

The SMS queue backs up durably; push and email continue unaffected due to channel isolation; backoff+jitter governs retry behavior as the provider recovers, and recovery is paced rather than an instant backlog release. See Failure & Scale -> Provider Outage.

• Would you fail over to a secondary SMS provider immediately? — Tests the failover-vs-retry trade-off — not a fixed yes/no, but reasoning about urgency, duplicate tolerance, and whether the primary's failures are clean rejections or ambiguous timeouts.

Not automatically. A timeout doesn't confirm the primary failed to accept the request — it may have already succeeded. Immediate failover on ambiguous outcomes risks duplicate delivery. See Trade-offs -> Provider Failover vs Retry Primary.

• How would you protect a fallback provider from also being overwhelmed? — Tests whether the candidate considers that a fallback provider suddenly receiving all failed-over traffic needs its own rate limiting and health monitoring, not unlimited trust.

SENT (or PROVIDER_ACCEPTED) means the provider took the request; DELIVERED means confirmed receipt by the recipient/device, typically learned later via a callback or status query. Conflating the two overstates what the system actually knows.

• Does every channel support a real DELIVERED signal? — Tests honesty about channel differences — SMS in particular can often only be confirmed as far as carrier handoff, not true device receipt.

Deduplicate on the provider's own callback/event ID before applying any state transition, so a duplicate callback is a no-op rather than reapplying the same transition. See Architecture Deep Dive -> Provider Webhooks & Delivery Reconciliation.

• What if the webhooks arrive out of order? — Tests whether the candidate's state machine is monotonic (only allows forward progress) so an out-of-order callback can't regress an already-terminal state.

Reconcile via the provider's own status/query API on a reasonable polling cadence for notifications with no callback after some window, rather than assuming silence means success or failure. See Failure & Scale -> Duplicate/Out-of-Order Webhooks.

• Should this reconciliation run continuously or only on demand? — Tests operational thinking — a background reconciliation job is more robust than only checking when a support ticket comes in.

Resolved as part of routing, before a channel worker acts — notification intent -> recipient policy/preferences -> eligible channel(s) -> channel queue/worker. See Architecture Deep Dive -> Preferences & Channel Eligibility.

• What if the preference changes while the notification is already queued? — Tests whether the candidate re-checks eligibility close to actual delivery for delayed/retried notifications, rather than trusting a preference snapshot taken at enqueue time.

Track tokens per device (user_id -> device_id -> platform -> push_token -> status), select currently active tokens at send time, and deactivate any token the provider reports as invalid. See Architecture Deep Dive -> Push Device Token Management.

• What happens if one of the three tokens is invalid? — Tests whether the candidate treats this as a per-token, non-retryable failure that shouldn't block delivery to the user's other active devices.

Chunked, paced fanout for the campaign into the normal channel/priority queues, with transactional/security traffic isolated in its own priority tier or queue so campaign volume physically cannot crowd it out. See Architecture Deep Dive -> Batch/Campaign Fanout and Failure & Scale -> Massive Campaign Overload.

• How do you calculate how long the campaign will take to fully deliver? — Tests the drain-time calculation: recipient count divided by sustainable delivery rate (itself bounded by provider quota), not an assumption that more workers linearly speeds this up.

That's no longer a queue-absorption problem — it's a genuine capacity mismatch. The queue keeps growing regardless of tuning; the actual fix is provider-capacity increase, shedding/deprioritizing lower-value traffic, or accepting a longer delivery window for that traffic explicitly.

• Is provider quota or worker count usually the bottleneck here? — Tests whether the candidate defaults to 'add more workers' or correctly identifies provider quota as the likely hard ceiling.

This is the DB/queue dual-write problem: the notification exists in state but no worker will ever see it, unless something detects and repairs the gap. See Architecture Deep Dive -> Durable Notification Ingestion.

• How does the transactional outbox pattern help here? — Tests whether the candidate can explain the mechanism: write notification + outbox row in one transaction, then a separate publisher retries publishing until it succeeds.

Yes — if it publishes successfully but crashes before marking the row published, it may re-publish the same event on restart. This is exactly why consumers must still be idempotent even with an outbox in place.

• So does the outbox eliminate the need for consumer idempotency? — Tests a subtle but important point: no — the outbox solves the dual-write problem, not exactly-once delivery to the broker or beyond.

If durable acceptance can't be guaranteed (e.g. no outbox buffering the write), the API may need to degrade honestly rather than falsely claim acceptance. With a transactional outbox, the database write can still succeed and accumulate safely until the broker recovers. See Failure & Scale -> Queue/Broker Outage.

• What happens to notifications submitted during the outage once the broker recovers? — Tests whether the candidate paces the recovery publish rather than releasing an instant flood.

Depends on notification class: pause high-risk/high-consequence sends, or accept elevated duplicate risk for lower-risk classes, or lean on provider-side idempotency where supported as a fallback layer. See Failure & Scale -> Dedup/Idempotency Store Failure.

• Is there one universal right answer here? — Tests whether the candidate resists giving a single blanket answer and instead reasons per notification class.

It shouldn't have to — if the tracker is a downstream, best-effort-buffered consumer rather than a synchronous dependency, delivery can continue while status updates are buffered and retried once the tracker recovers. See Failure & Scale -> Delivery Tracker Failure.

• What data must be preserved during the outage to avoid a permanent gap? — Tests whether the candidate names provider IDs/results specifically as data that must not be lost even while buffered.

Not necessarily — it could be partial degradation rather than full unavailability. The response should be proportional: back off and reduce assumed capacity, but not necessarily fail over immediately, and keep monitoring whether the rate is climbing toward a full outage or stabilizing.

• How would you distinguish partial degradation from a full outage in your monitoring? — Tests whether the candidate thinks in terms of graduated signals (error rate trend, latency percentiles) rather than a binary up/down state.

Missed callbacks during that window should be recovered by later reconciling against the provider's own status/query API, if one exists, rather than treating the gap as permanently lost information. See Architecture Deep Dive -> Provider Webhooks & Delivery Reconciliation.

• Would the provider retry the webhook on its own? — Tests awareness that many providers do retry webhook delivery for a period, which is itself a source of the duplicate-callback scenario, not just a separate concern.

Per-tenant/per-producer rate limiting or quotas at the API layer, independent of the downstream provider rate limiting — otherwise one misbehaving or buggy producer can consume disproportionate queue and provider capacity that other tenants depend on.

• Is this the same mechanism as provider rate limiting? — Tests whether the candidate distinguishes tenant-level ingestion limiting (protecting fairness among producers) from provider-level rate limiting (protecting the external provider's quota) as two separate, complementary controls.