Tradeoffs & Design Decisions

"Exactly-once" is ambiguous until you say exactly-once what — queue delivery, execution attempt, or external business effect are three different guarantees, and conflating them is the single most common mistake in this discussion.

Queue-level exactly-once processing is achievable only inside a single broker's boundary (for example Kafka's idempotent producer and transactions for read-process-write between Kafka topics), and that guarantee stops where the broker ends. Execution attempt exactly-once (a worker runs a job's handler precisely once) additionally requires coordinating across the scheduler, the queue, and the worker — harder, and still bounded to your own systems. External business effect exactly-once (the payment is charged precisely once, on the provider's side) is the hardest of the three, because it depends on a third-party system that has no knowledge of your internal delivery guarantees at all.

It is not simply "use 2PC"

Distributed transactions or two-phase commit across your database and a third-party API are rarely available (most providers don't participate in your transaction protocol) and, where technically possible, add substantial latency and coordination overhead for a guarantee that a simpler pattern usually delivers just as well. In specific, bounded domains, an exactly-once effect can be approximated or genuinely achieved using: a transactional state change (the side effect and its completion marker commit atomically, when both live in your own database); an idempotent downstream API (many providers accept an idempotency key and deduplicate on their side); deduplication (a local dedup table keyed on a stable business id, checked before acting); transactional messaging (the outbox pattern, guaranteeing a message is published if and only if the local state change committed); and reconciliation (a periodic audit comparing your recorded state against the downstream system's actual record, resolving mismatches after the fact).

"Polling vs push" collapses three genuinely separate stages into one question — and the right answer differs at each stage.

A common confusion is treating "workers pulling from a queue" as the same category of decision as "database polling for due jobs" — they solve different problems. Worker-side pulling is about consumption rate control; scheduler-side due-work discovery is about how cheaply you can ask "what's ready." A production system typically ends up poll (discover) -> push (dispatch) -> pull (consume) all in the same pipeline, which is why framing this as one binary choice misses what's actually being decided at each stage.

"Use both" is the common production answer, but it is not the only correct architecture — and presenting it as the sole valid option skips a real trade-off worth understanding.

Database as source of truth

Strong queryability (filter by status, type, user, date range), straightforward updates/cancellation (a job is just a row you can mutate), and a natural audit history. The cost: you need an explicit bridge from "a row became due" to "a worker actually received it" — the durable dispatch / outbox mechanism described in the Architecture Deep Dive.

Durable log/queue as source of truth

If the queue itself is durable and replayable (e.g. a long-retention Kafka topic), you can treat the log as authoritative and skip building a separate database-backed job store — this can meaningfully simplify a purely event-driven flow. The cost: long-term future scheduling (a job due in 90 days) is awkward for a log designed around near-term delivery, arbitrary updates to an already-published message are difficult or impossible, and ad-hoc status queries ("show me all pending jobs for user X") are much harder against a log than against an indexed table.

Database + queue together

The common, practical choice — database for durability, queryability, and lifecycle management; queue for high-throughput, low-latency dispatch to workers. The cost of this combination is exactly the dual-write consistency problem covered in the Architecture Deep Dive: the two stores can disagree, and the design has to explicitly handle both directions of that disagreement rather than assuming "use both" is automatically safe.

This is a three-way trade-off, not a binary one — competing claimers (the SKIP LOCKED model) is a genuinely different point on the spectrum from either a single elected leader or sharded ownership, not just a variant of one of them.

"Financial jobs get strong consistency, notifications get eventual consistency" is a convenient rule of thumb, but it answers the wrong question — the real question is which specific piece of state actually needs strong consistency, independent of what kind of job it belongs to.

Several state transitions need strong consistency regardless of job type, because a race at exactly that point causes a correctness bug, not just a stale read: job creation idempotency (two near-simultaneous creation requests for the same logical job must not produce two job records); claim ownership (two workers must not both believe they hold the lease for the same job); the cancellation race (a cancel and a dispatch racing must resolve to one consistent outcome, not both); occurrence generation for recurring schedules (the same nominal fire time must not generate two occurrence records); and the completion transition (a job must not be marked both SUCCEEDED and, separately, re-queued for retry, from two different code paths).

Other state genuinely tolerates eventual consistency regardless of job type: analytics aggregates, dashboard metrics, and non-authoritative caches can lag by seconds without causing incorrect behavior, because nothing downstream makes a correctness-sensitive decision based on their exact current value.

The three most common due-time index implementations trade precision, durability, and operational complexity against each other — none dominates on every axis.

This comparison is qualitative: no throughput or latency figures are given because they depend on the implementation, hardware and workload. The timing-wheel structure originates in Varghese and Lauck, "Hashed and Hierarchical Timing Wheels: Data Structures for the Efficient Implementation of a Timer Facility" (SOSP 1987).

This is one of the most consequential production trade-offs in the entire design — it decides whether a crash between the database commit and the queue publish silently loses or duplicates dispatch.

Direct DB write, then queue publish

Simpler to build: transition the job's state in the database, then call the broker's publish API. The cost is exactly the dual-write problem covered in the Architecture Deep Dive — a crash between the two steps leaves the job durably marked as dispatched with no message ever published, and it will never be discovered without a separate reconciliation pass.

Transactional outbox

The job transition and an outbox record commit in the same database transaction; a separate relay publishes from the outbox and marks it published. This closes the silent-loss gap at the cost of an additional moving part (the relay process or CDC pipeline) and a small publish-latency tax (the relay has to notice and process the new outbox row, rather than publishing inline).

Where the worker lease lives — alongside job state in the database, or in a separate Redis structure — is a smaller-scale version of the same source-of-truth question as the due-time index.

A fixed-duration lease is simple to reason about but breaks down the moment job execution time is variable or genuinely long-running.

Fixed lease

The lease is set once, for a fixed TTL, and never extended. Simple: no renewal protocol to get wrong. Problematic the moment job durations vary meaningfully or some jobs run long — either the fixed TTL is set long enough to cover the slowest job (making crash recovery slow for the common case of fast jobs), or it's set short enough for fast jobs (causing false-positive reclaims of legitimately still-running long jobs).

Renewable lease

The lease starts short and the worker periodically renews it (the conditional check-then-extend pattern from the Architecture Deep Dive) as long as it's still making progress. This supports long-running and variable-duration work without picking one TTL that has to fit every job. The cost: a renewal/heartbeat protocol to implement correctly, and a genuine risk of false expiry if a worker is merely slow to renew (GC pause, network blip) rather than actually dead — the renewal interval and TTL have to be tuned with that trade-off in mind.

Whether the dispatcher starts moving jobs toward workers slightly before their exact due time is a real lever against thundering-herd bursts — but it changes what the queue and worker have to enforce.

Pre-dispatch

The dispatcher begins preparing/queuing a batch shortly before its exact due time, smoothing an instantaneous burst into a short window instead of a single spike. This requires the queue and worker to respect an eligible_at field distinct from the message's mere presence in the queue — a worker that blindly executes anything it dequeues will run jobs early, which silently breaks the scheduling contract. Pre-dispatch is a genuine correctness feature only if that enforcement is actually built; without it, it's just early execution.

Just-in-time dispatch

The dispatcher only ever queues a job once it's actually due. Simpler semantics — no eligible_at enforcement needed anywhere downstream — but it means the full weight of a synchronized burst (many jobs due at the same instant) lands on the dispatcher and queue at once, with no smoothing built in. This shifts the entire thundering-herd mitigation burden onto rate-limited dispatch and bounded batching rather than spreading it earlier in the pipeline.

When a recurring schedule missed one or more occurrences during an outage, skipping them and catching them up are both defensible defaults — the right one depends entirely on what the missed occurrence was for.

Skip missed occurrences

Resume from the next future occurrence and treat the missed ones as simply not run. Correct for schedules where a missed run has no lasting value once its window has passed — a "good morning" notification, an hourly metrics snapshot that's superseded by the next one.

Catch up missed occurrences

Run the missed occurrence(s), typically the most recent one, once the system recovers. Correct for schedules where the underlying work still needs to happen even if late — a daily billing run, a report that must exist for every day regardless of when it's generated.

If a recurring job's previous occurrence is still running when the next one comes due, the system needs an explicit policy — silently allowing unlimited overlap is rarely the right default.

Allow overlap

Simplest to implement — each occurrence runs independently regardless of whether prior occurrences are still in flight. Correct when occurrences are genuinely independent and cheap (a lightweight per-minute health check), but dangerous when occurrences share state or resources — unbounded overlap under a slow-running dependency can cascade into many concurrent instances competing for the same resource.

Prevent overlap (skip or queue)

Skip the new occurrence if the previous one is still running (correct when only the most recent state matters), or queue it to start immediately after the previous one finishes (correct when every occurrence's work must eventually happen, just not concurrently). Both require the schedule to track whether an occurrence is currently in flight, and both need a safety valve (a maximum queue depth, or a maximum skip streak) so a truly stuck previous occurrence doesn't block or silently drop an unbounded number of future ones.

Strict priority guarantees the most important work always goes first; fair scheduling guarantees every class of work eventually makes progress. A production scheduler usually needs a version of both, not a single global choice.

Strict priority

Always dispatch the highest-priority ready queue before touching any lower one. Simple, and gives the strongest possible latency guarantee to the top class. The risk: if high-priority volume is sustained and never lets up, lower-priority classes can be starved indefinitely — there is no built-in bound on how long a low-priority job can wait.

Fair scheduling (weighted, quota-based, or aging)

Guarantees every class makes measurable forward progress — via weighted capacity allocation, per-tenant quotas, or priority aging (a job's effective priority rises the longer it waits). The cost: the most urgent job no longer always jumps the entire queue instantly — its latency is now bounded by the fairness mechanism's parameters rather than being unconditionally first.

Storing a job's full payload inline is simple until the payload is large — at which point it quietly bloats every system that has to carry a copy of it.

Inline payload

The job payload lives directly in the database row, the Redis due-time index entry, and the queue message. Simple — no extra fetch step, no external storage dependency. The cost compounds across every system it touches: a large payload bloats the database's row size and index pages, bloats Redis's memory footprint per scheduled entry, bloats every queue message (and every retry of that message, since retries typically resend the full payload), and makes replaying a DLQ entry more expensive.

Payload reference

Store the actual payload as an immutable blob in object storage (or a dedicated payload store), and schedule only a lightweight reference (an object key or payload id) through the database, index, and queue. This keeps every layer of the scheduling path small and cheap regardless of payload size, at the cost of an extra fetch (and its own failure mode — what happens if the referenced payload is missing or expired by the time the job executes) and an additional storage system to manage retention for.

Multi-region scheduling is fundamentally a trade between availability during a region failure and the complexity of preventing duplicate dispatch across regions.

Single-region authority (with a replicated standby)

One region owns due-time decisions and dispatch for a given job at any time; its durable state is replicated to a standby region. Simple to reason about — there's only ever one place a dispatch decision can be made — at the cost of an availability gap during failover: jobs due while the home region is down, and no other region can act, either wait (bounded by RTO) or are handled by the misfire/catch-up mechanism once failover completes.

Active-active scheduling

Multiple regions can independently discover and dispatch due work, maximizing availability — no region failure stops dispatch entirely. The cost is substantial: without a global fencing mechanism and cross-region consensus on due-time ownership, two regions can genuinely dispatch the same job simultaneously, and preventing or safely tolerating that requires solving a harder coordination problem than anything else in this design.