Architecture Deep Dive: Distributed Task Scheduler

Every scheduling backend answers the same question, over and over, at whatever rate the workload demands: give me the next bounded batch of jobs eligible by time T. How you answer that query cheaply and correctly is the single most consequential decision in the whole system.

Five structures answer it, with very different cost curves. A B-tree/indexed database table (an index on (due_at, status)) is the simplest and gives you durability, transactions, and full queryability for free — the cost is read amplification under high polling frequency and index maintenance overhead as rows update. A sorted set (Redis ZADD/ZRANGEBYSCORE) trades durability for speed — an in-memory ordered structure keyed by due time is extremely fast to query but is not itself the source of truth. A time bucket (pre-grouping jobs into fixed windows, e.g. one bucket per minute) turns 'find due jobs' into 'read this bucket,' at the cost of coarser granularity and bucket-boundary edge cases. A timing wheel (as used by Kafka's purgatory and Netty's HashedWheelTimer) gives O(1) insert/expire but is in-memory and single-process, so it needs an external durable backstop. A broker-native delay mechanism (SQS delay queues, RabbitMQ delayed-message plugins) removes the need to build a due-time index at all, at the cost of a hard delay ceiling and no query/cancel capability once the message is in flight.

What actually matters when comparing these

Ordered retrieval — can the structure hand you jobs in ascending due-time order without a full scan? Bounded batches — can you ask for 'the next 500 due jobs' instead of 'all due jobs,' so a burst never floods downstream systems in one call? Partitioning — can the due-time space be split across multiple index shards so no single structure becomes a bottleneck? Updates — what happens when a job's due time changes after it's indexed (reschedule) or the job is cancelled — is that an O(1) removal or does it require a rebuild? Recovery/rebuild — if the index is lost or corrupted (a crashed Redis instance with no persistence, a corrupted timing wheel), can it be reconstructed from the durable source of truth without losing jobs?

SKIP LOCKED lets multiple scheduler or worker processes claim different due rows concurrently without blocking on each other — but the transaction that does the claiming must be short, and the actual work must happen outside it.

Why the claim and the dispatch must be separated

If you publish to the queue inside the same transaction that claims the rows, a slow broker call holds the row locks open for however long that network call takes — exactly the contention this pattern exists to avoid. Claim and commit quickly; do the slower work (publishing, calling out) after the transaction has released its locks. This is also why durable dispatch typically needs its own correctness mechanism — see the transactional outbox pattern below — because 'commit the claim, then publish separately' reintroduces a two-step, non-atomic sequence.

Lock scope, contention, and empty polling

FOR UPDATE SKIP LOCKED only skips rows currently locked by another transaction — it does not skip rows that simply don't match your WHERE clause, so the index on (status, due_at) still has to be selective enough that each poller isn't scanning large numbers of not-yet-due rows. Under high poller concurrency, many claimers hitting the same due window means most of them find nothing to claim (their candidate rows were taken by someone else) — that's expected and cheap, not a bug, but it does mean idle polling has a real cost at scale: wasted connections and CPU cycles scanning a mostly-already-claimed window. Widening the polling interval reduces this waste. PostgreSQL LISTEN/NOTIFY can wake pollers, but only when a session explicitly issues a NOTIFY (for example from a trigger when a job is inserted or rescheduled); it never fires because time passes, so a row becoming due at due_at generates no notification. Delivery is also best-effort: notifications are sent only to sessions currently listening, are not persisted for a disconnected listener, and are delivered only when the notifying transaction commits. Pollers therefore still need a timer or polling fallback, for example sleeping until the earliest known due_at and re-scanning periodically.

Lease recovery

A row claimed into DISPATCHING or RUNNING state whose lease_until has passed without a completion is not stuck forever — a recovery scan (WHERE status IN ('DISPATCHING','RUNNING') AND lease_until < now()) finds it and returns it to SCHEDULED for another claim attempt. This recovery scan is the DB-native equivalent of the worker heartbeat/reaper pattern, and it's what makes SKIP LOCKED safe to run unattended.

A Redis sorted set keyed by due-time score is a fast, memory-resident due-time index — but several commonly repeated claims about it are too strong, so the precise guarantees matter before the index is load-bearing in a design.

Encoding: not always 'skip list + hash table'

Redis sorted sets are commonly implemented using a hash table (member -> score lookup) plus a skip list (ordered traversal by score) once the set grows past a size/element-length threshold. Below that threshold, Redis may use a more compact listpack-style encoding instead, depending on Redis version and the zset-max-listpack-entries / zset-max-listpack-value configuration. The precise encoding is an implementation detail that can change between versions — don't present 'every sorted set is a skip list plus hash table' as an architectural fact; present it as 'the encoding Redis chooses depends on set size and configuration, and the complexity bounds below hold across that range.'

Complexity: logarithmic, not constant

Insert and update (ZADD) are typically O(log N). Range-by-score (ZRANGEBYSCORE) is typically O(log N + M) where M is the number of elements returned. About log2(N) comparisons for a 10-million-element set (roughly 23) is not 'effectively constant time' — 23 comparisons per operation is small in absolute terms, but the operation remains logarithmic, and that distinction matters once N grows by orders of magnitude or once you're comparing this structure honestly against a true O(1) alternative like a timing wheel.

Persistence and durability

Redis is not durable by default in the way a WAL-backed database is. appendfsync everysec bounds typical data loss to roughly the last second or two of writes under normal operation, but this is not an absolute guarantee — it depends on the underlying storage behaving as expected and on how the process actually terminates. If Redis is the only copy of due-time state (rather than an index derived from a durable store), that loss window is a real correctness gap, not just a performance footnote — see the durable-store-consistency section below for how to close it.

Once the database is authoritative for job state and Redis is 'just' the due-time index, the two can disagree — and the design has to say what happens when they do.

DB write succeeds, Redis write fails

The job exists durably in the database but never made it into the due-time index. Nothing will ever query for it, so it silently never fires — which is worse than a visible failure, because there's no error to alert on. Closing this gap needs one of: an outbox/CDC pipeline that reliably relays the write into Redis with retry; a periodic reconciliation scan of the database for jobs that are due but missing from the index; or treating the index as fully rebuildable and periodically regenerating it wholesale from the database for a bounded future window.

Redis write succeeds, DB write fails

The index now points at a job_id that doesn't durably exist (or exists in a stale, pre-transaction state). This is the safer failure direction: when the dispatcher pulls that entry, it validates against the source-of-truth database before dispatching, finds no matching durable record (or a record in a state that doesn't warrant dispatch), and simply discards or ignores the stale index entry. The dispatcher should never trust the index as authoritative — it's a work queue of candidates, validated against the database before anything downstream is triggered.

UPDATE job SET state='QUEUED'; publish(queue) looks like a single step but is actually two independent operations against two independent systems — and there is no atomic way to guarantee both happen, or neither happens.

If the process crashes between the UPDATE commit and the publish call, the database says QUEUED but the queue never got the message — the job is durably marked as dispatched but will never actually run unless something notices the gap. If the publish is retried for safety and the retry itself is delivered (the original attempt actually succeeded but the acknowledgment was lost), the queue now holds two messages for one dispatch — which is fine if downstream consumers are idempotent, but is a correctness gap if they aren't.

The transactional outbox pattern

Instead of writing to the job table and calling the broker as two separate operations, the same database transaction that transitions the job also inserts a row into an outbox table (dispatch record: job_id, payload, created_at, published_at = NULL). Both writes commit atomically — because they're the same transaction, in the same database. A separate relay process then: reads unpublished outbox rows; publishes each to the queue; marks it published (or deletes it) only after a confirmed publish. If the relay crashes after publishing but before marking the row published, it will publish the same row again on restart — which is why consumers must deduplicate on the outbox row's stable id, not merely react to arrival.

CDC as an outbox variant

Instead of a separate relay process polling the outbox table, change-data-capture (e.g. Debezium reading the Postgres WAL) can stream outbox inserts directly to the broker. This removes the polling relay but doesn't change the core guarantee: at-least-once delivery to the queue, with the same downstream deduplication requirement.

A leader-elected scheduler needs more than 'whoever grabs the lock first is the leader' — the lock must carry a unique owner token, and every renewal or release must verify that token, or a paused/slow leader can silently corrupt the lease it no longer owns.

A bare EXPIRE refresh without checking ownership first is unsafe: if the lease already expired and a new leader acquired it, the old leader's blind EXPIRE call extends the new leader's lease using the old leader's schedule, or a bare DEL release deletes the new leader's lock entirely — both let two nodes believe they're leader at once.

The stale-leader sequence

1. The current leader pauses (GC stop-the-world, CPU starvation, network partition) for longer than the lease TTL. 2. The lease expires. 3. A follower notices, acquires the lease, and becomes the new leader. 4. The original leader resumes running — from its perspective, nothing happened, and it still believes it's leader. 5. Without a further safeguard, both nodes now act as leader simultaneously, each dispatching jobs.

Fencing: making a stale leader's actions harmless even if it acts

A lease TTL reduces the window for split-brain but cannot eliminate it on its own — pauses of unpredictable length are exactly what make distributed lease timing unreliable. Fencing tokens close the gap: every time leadership changes, a monotonically increasing fencing epoch is issued. Every write the leader makes downstream (a claim, a dispatch) is tagged with its epoch. Downstream systems reject writes tagged with an epoch older than the highest epoch they've already seen. A conditional DB version (optimistic concurrency on the job row) or a shard-ownership generation number both serve the same purpose. Combined with idempotent dispatch consumers, a stale leader's stray actions become harmless duplicates rather than silent corruption.

A single leader dispatching everything is a throughput ceiling at large scale — sharding splits the due-time space across multiple independently-led partitions, each responsible for its own slice.

Ways to shard

Hash-range shards (hash the job_id or schedule_id into N buckets, one leader per bucket) give an even distribution with no natural locality. Time buckets (shard by due_at window, e.g. by hour) give natural locality for time-range queries but concentrate load on whichever bucket is currently 'hot' (the near-future window). Tenant or job-type shards (each large customer or job category owns a shard) give strong isolation — one noisy tenant can't starve another — but risk severe skew if one tenant or job type dominates total volume.

Ownership metadata, rebalancing, and hot shards

Each shard needs its own leader-election lease (the same mechanism as the single-leader case, replicated per shard) and an ownership record that says which node currently owns which shard range. Rebalancing — moving shard ownership when a node fails or when load is uneven — has to be done carefully: a shard boundary move mid-flight can momentarily leave a range of due jobs with no owner (a gap) or two owners (a duplicate-dispatch risk) unless the handoff is itself fenced with an ownership generation number. Detecting a hot shard requires per-shard observability (dispatch rate, backlog age, queue depth) — without it, an imbalanced sharding scheme just looks like 'the scheduler is sometimes slow' with no visible cause.

The lease/reaper pattern from the naive design is correct in spirit — a worker holds a time-bound claim on a job, and a reaper reclaims jobs whose lease expired without completion — but the fields on that lease, and where it lives, need to be more precise than 'a Redis lock.'

What the lease record needs

Conditional renewal and conditional completion

A renewal must check lease_owner == me AND lease_generation == what I last saw before extending — exactly the same check-then-act discipline as leader-election fencing. A completion write (marking the job SUCCEEDED) must carry the same generation check: if the generation has already moved on (the reaper reclaimed this job and gave it to another worker while this worker was still finishing), the late completion should be rejected or treated as a harmless duplicate signal, not allowed to silently overwrite a newer attempt's state.

Where the lease should live

A Redis lock plus a separate DB row for job state is two sources of truth that can disagree — exactly the durable-store/index-consistency problem discussed above, now applied to lease ownership instead of due-time indexing. If the database already owns job state, a DB-backed lease (the same columns as the table above, stored directly on the job/execution row, updated via conditional UPDATE ... WHERE lease_owner = ? AND lease_generation = ?) is often simpler: one system of record, one set of consistency guarantees, no cross-store reconciliation needed. Redis-based leases earn their complexity when lease-check throughput genuinely exceeds what the database can serve — not by default.

A job that runs for six hours breaks the assumption baked into a short, fixed lease TTL: you cannot simply set the TTL longer than the entire job, because that erases the whole point of having a short reclaim window for the common case of fast jobs.

The core techniques

Lease renewal: the worker periodically extends its own lease (the conditional renewal described above) while it's actively making progress, so the lease only needs to be as long as the heartbeat interval, not the whole job duration. Progress checkpointing: for jobs that can be resumed partway through (a batch export, a multi-stage pipeline), periodically persist how far the job has gotten, so a retry after a crash resumes from the last checkpoint instead of restarting from zero. Execution timeout: a separate, explicit ceiling on total job duration — distinct from the lease TTL — that triggers cancellation if a job runs pathologically long, independent of whether it's still heartbeating.

Process restart and cancellation

If the worker process itself restarts mid-job (deploy, crash, autoscale-down), the in-flight lease eventually lapses (no more heartbeats) and the reaper reclaims it — the job resumes from its last checkpoint on a new worker, or restarts from scratch if it wasn't checkpointable. Cancellation of a long-running job has to be checked cooperatively: the worker periodically polls 'has this job been cancelled?' at safe checkpoints, since there's no way to forcibly interrupt arbitrary running code from the outside without risking a corrupted intermediate state.

Downstream side effects

The longer a job runs, the more likely it has already caused partial downstream side effects (rows written, emails sent, files uploaded) before a crash or cancellation. This is exactly why idempotency has to be a first-class design concern rather than an afterthought for long-running work — see the next section.

The scenario that breaks every 'just add a lock' answer: a worker executes a side effect against an external system, crashes before recording success, and a retry follows. A scheduler-level lock alone cannot prevent the side effect from happening twice, because the lock only ever protected scheduling, not the external effect itself.

Concretely: a payment job comes due at 09:00. The scheduler's dispatch record commits in the database. The queue publish is retried for safety and both attempts are delivered, producing two messages for the same dispatch. Worker A picks up the job, acquires the lease, calls the payment provider's charge API, and the charge succeeds — but Worker A crashes before it writes 'SUCCEEDED' anywhere. The lease expires. Worker B (from the duplicate message, or via the reaper reclaiming the lease) picks up the same job and, having no record that Worker A already succeeded, calls the charge API again.

Why a scheduler-level lock alone is not enough

The lease and the idempotency lock both live in the scheduler's own storage — they know nothing about the state of the external payment provider. Worker A's lease expiring tells you nothing about whether Worker A's charge attempt actually completed on the provider's side before it crashed. The job was never lost (the database still has the durable record); the duplicate dispatch was harmless on the scheduler's side (two messages for one logical unit of work is a normal at-least-once artifact); but the lease alone is insufficient to guarantee the payment itself is exactly-once.

The patterns that actually close the gap

Transactional local mutation: if the side effect is entirely within your own database (e.g. decrementing an internal balance), wrap the check-and-act in the same transaction as the state update, so a crash mid-way rolls back cleanly. Unique business key: derive a natural idempotency key from the business operation itself (e.g. invoice_id or order_id + attempt_window), not from the scheduler's internal job_id, so that even a completely different retry path converges on the same key. Downstream idempotency key: pass an idempotency key to the external provider's API (most payment processors support this natively) so the provider itself deduplicates the charge even if your worker calls it twice. State machine + reconciliation: record 'charge initiated' before calling out, then run a periodic reconciliation job that checks the provider's actual transaction log against your recorded state and resolves any 'initiated but unknown outcome' rows. Saga/compensation: where the side effect can't be made idempotent directly (e.g. it triggers a chain of other systems), design an explicit compensating action instead of assuming the forward path can be made safe to repeat.

A recurring schedule ("every day at 9am", "every Monday") is not a job — it's a template that generates a stream of individual job occurrences, and getting that generation step wrong is how you get duplicate or missing runs.

Schedule data model

Conceptually, a schedule record needs: schedule_id; the recurrence rule itself (cron expression or an equivalent structured rule); the timezone the rule is evaluated in (see the next section — this is not optional); next_fire_at, the precomputed next occurrence so the due-time index doesn't need to re-parse the recurrence rule on every poll; a misfire policy (what to do if the scheduler was down when this should have fired); an overlap policy (what to do if the previous occurrence is still running when the next one comes due); a version number (for safe concurrent updates to the schedule itself); and a paused flag.

Idempotent occurrence generation

Each fire of the schedule produces one job occurrence, and that occurrence's identity should be derived deterministically: occurrence_id = f(schedule_id, nominal_fire_time). If the occurrence-generation step runs twice for the same nominal fire time (a retry, a duplicate scan, a failover replaying recent work), inserting with that deterministic id and an ON CONFLICT DO NOTHING (or equivalent) makes the second attempt a harmless no-op instead of a duplicate job.

Advancing safely to the next occurrence

After generating occurrence N, computing next_fire_at for occurrence N+1 has to happen atomically alongside marking occurrence N as generated — otherwise a crash between 'generate occurrence N' and 'advance next_fire_at' either regenerates N again (caught by the idempotent occurrence id above) or, worse, never advances and the schedule appears to have stopped firing entirely.

A recurring schedule expressed as "9am" is a wall-clock concept tied to a specific timezone — and that's a fundamentally different kind of time than the absolute UTC instant the due-time index actually stores and compares against.

Three different notions of time in the same system

Wall-clock recurring schedule: what the user configured — 'every day at 9am America/New_York.' This has to be re-resolved to an absolute instant for each occurrence, because the UTC offset for 9am New York changes across a DST transition. Absolute UTC instant: what the due-time index actually stores and compares — a single point on the timeline, unambiguous, DST-agnostic once computed. Monotonic elapsed time inside a process: used for timeouts and lease durations within a single worker's lifetime — never for cross-node scheduling decisions, because it resets on restart and isn't comparable across machines.

DST transitions

On a 'spring forward' transition, the wall-clock time 2:30am may not exist at all that day — a naive recurrence calculation can produce an invalid or ambiguous instant. On a 'fall back' transition, 1:30am occurs twice — a schedule computed against the wrong occurrence fires an hour early or late, or (worse) twice. Resolving the recurrence rule against a real timezone database (not a fixed UTC offset baked in at creation time) and re-resolving it fresh for every occurrence, rather than caching a computed offset, is what keeps this correct across transitions.

Clock skew across nodes

Distributed scheduling decisions ultimately compare wall-clock timestamps across different machines — the scheduler's notion of 'now' and the worker's notion of 'now' are not guaranteed to be identical. Use synchronized clocks (NTP) as a baseline and build in a small safety margin (a few seconds) wherever a comparison is time-sensitive (lease expiry, misfire detection) rather than assuming every node's clock agrees to the millisecond.

After any outage long enough that jobs became overdue, the system faces a choice that's easy to get catastrophically wrong: how much of the backlog does it try to run, and how fast?

Discovery

On recovery, the scheduler needs to find two different kinds of backlog: overdue one-time jobs (their due_at already passed) and recurring schedules that missed one or more occurrences entirely (their next_fire_at is in the past, possibly by several missed cycles).

Misfire policy for recurring schedules

A recurring schedule that missed occurrences during the outage needs an explicit policy, not a default assumption: fire once immediately for the most recent missed occurrence and skip the rest; fire once for every missed occurrence (dangerous for anything other than genuinely idempotent, cheap jobs); or skip all missed occurrences entirely and simply resume from the next future occurrence. Which is correct depends entirely on the job's semantics — a daily report that was missed twice should probably run once, not twice; a payment reminder should probably not double-fire.

Avoiding instant unbounded catch-up

Whatever the discovered backlog is, releasing all of it back into the dispatch pipeline in one burst recreates the exact thundering-herd failure mode described elsewhere in this design — except now it's self-inflicted by the recovery process itself. Mitigate with: bounded batches (dispatch the backlog in fixed-size chunks, not all at once); priority (time-sensitive job classes drain before lower-priority ones); a catch-up rate limit (cap dispatch throughput during recovery below the normal steady-state ceiling, so downstream systems aren't hit by a second spike on top of whatever caused the outage); coalescing (multiple missed occurrences of the same recurring schedule collapse into a single catch-up run, per the misfire policy above); and an expiration/deadline (a job so overdue that running it late no longer has value should be marked expired rather than dispatched at all).

A retry formula (base * 2^attempt + jitter) is necessary but not sufficient — a production retry engine needs explicit state per attempt and a classification of why the job failed, because not every failure deserves the same treatment.

Classifying failures before deciding how to retry

Transient (a timeout, a connection reset) — retry with standard backoff. Throttling (a 429 or explicit rate-limit signal from downstream) — retry, but honor any Retry-After hint and back off more aggressively than the default curve. Permanent (a 4xx validation error, a business-rule rejection) — do not retry at all; route straight to the DLQ, because retrying an inherently-invalid request just wastes attempts and delays visibility into a real bug. Unknown/ambiguous (the call timed out with no confirmation the side effect didn't happen) — this is the hardest case, and it's exactly where the idempotency patterns from earlier in this deep dive matter: retry, but only because the downstream operation is idempotent, not because the failure was clearly safe to repeat.

Capped exponential backoff with jitter

Backoff should grow exponentially up to a capped maximum delay (unbounded exponential backoff eventually schedules retries days or weeks out, which is rarely the intent), and jitter should be added to avoid every job that failed at the same instant retrying at the same instant again — the same thundering-herd concern that applies to initial dispatch applies to synchronized retries.

The dead letter queue is a forensic record, not a graveyard — its value comes from what you can safely do with the jobs sitting in it, not just from the fact that they're preserved.

Beyond preservation: operational controls

Replay authorization — not every engineer with database access should be able to trigger a bulk replay of DLQ jobs; replay is itself a production action and needs the same access control as any other. Replay rate limit — replaying an entire DLQ backlog at once recreates the thundering-herd problem, so replay should be dispatched through the same bounded, rate-limited path as normal catch-up traffic. Payload/schema compatibility — a job that's been sitting in the DLQ for weeks may reference a payload schema or a code path that no longer exists; replay tooling should validate compatibility before re-dispatching, not assume the original handler still accepts the original payload shape. Retain original job identity — replaying a DLQ entry should preserve its original job_id for traceability, so its full history (original attempts, failure reasons, replay events) stays linked. New attempt id — while the job identity is preserved, the replay itself gets a fresh attempt_id, so it's distinguishable from the attempts that originally failed. Audit log — who replayed what, when, and why is itself operationally important data, especially for anything touching money or user-facing state. Poison-job quarantine — a job that fails the same way on every replay attempt (a genuinely broken payload, not a transient downstream issue) should be flagged and excluded from further automatic replay attempts rather than cycling indefinitely between DLQ and retry.

A cancel request racing against an in-flight dispatch is a three-way race: the cancel request, the scheduler's dispatch, and the worker actually starting execution — and the job's version has to be the tiebreaker, not arrival order.

Concretely: a user cancels a job at the same moment the scheduler has already claimed it and queued a dispatch message. The message may already be in flight when the cancellation is recorded. Without a check, the worker picks up the stale message and executes a job the caller believes was cancelled.

Version-checked execution

Every job carries a version number that increments on any meaningful state change (cancellation, reschedule). A dispatch message carries the version it was dispatched with. Before executing, the worker re-reads the job's current state and version from the source of truth: if the current version doesn't match the dispatched version, or the current status is CANCELLED, the worker discards the message instead of executing it. This is the same check-then-act discipline used for lease renewal and leader fencing, applied to execution itself.

Rescheduling

A reschedule (changing due_at on an existing job) needs to: increment the version; invalidate the old due-index entry (remove or mark stale the entry keyed to the old due time, so the due-time index doesn't fire the job at its original time as well as its new one); and add a new due-index entry at the new due time. Any dispatch message already in flight against the old version is rejected by the version check above when a worker attempts to execute it, closing the same race the cancellation case does.

Strict priority (always dispatch the highest-priority queue first, full stop) solves the 'critical jobs shouldn't wait behind noise' problem and creates a new one: it can starve lower-priority classes indefinitely if high-priority volume never lets up.

Approaches, compared

Separate queues per priority class, drained in strict order — simple, but exactly the starvation risk above if the top queue is never empty. Weighted fair scheduling — allocate worker capacity across priority classes proportionally (e.g. 70% of workers on high priority, 20% on medium, 10% on low) so every class makes forward progress even under sustained high-priority load. Reserved worker capacity — carve out a fixed pool of workers that only ever pull from a specific class (e.g. payments always has N dedicated workers), guaranteeing a floor of throughput regardless of what else is happening. Per-tenant quotas — cap how much total capacity any single tenant's jobs can consume, independent of job priority, so one large tenant can't monopolize the pool even if all their jobs are legitimately high priority. Aging — a job's effective priority increases the longer it waits, so even low-priority work eventually crosses the threshold to get dispatched, bounding worst-case wait time without abandoning priority ordering entirely.

A queue between the dispatcher and the workers absorbs a burst — temporarily. If the true bottleneck is downstream of the workers (a rate-limited third-party API, a database that can't take the write volume) and that bottleneck's capacity stays below the arrival rate for long enough, the queue just grows without bound until something else breaks first.

The backpressure chain runs: due jobs -> dispatcher -> queue -> workers -> downstream dependencies. A queue can smooth a short spike, but it cannot manufacture downstream capacity that doesn't exist. If downstream throughput is permanently lower than the sustained arrival rate, the queue depth (and the age of its oldest message) grows linearly forever — the system needs either to slow the arrival rate at the source, add real downstream capacity, or shed load deliberately, because the queue alone will not fix an indefinite capacity mismatch.

What to actually monitor

Due backlog (jobs that are due but not yet dispatched) and queue depth (dispatched but not yet consumed) are different numbers and both matter — a due backlog with a shallow queue points at a dispatcher problem; a shallow due backlog with a deep queue points at a worker/downstream problem. Oldest job age answers 'how stale is the worst-case job right now' in a way a raw count can't. Dispatch lag (time between due_at and actual dispatch) and execution lag (time between dispatch and execution start) separate scheduler-side delay from worker-side delay. Downstream throttling signals (429s, explicit rate-limit responses) are the earliest indicator that the real bottleneck has moved past the workers entirely — catching this before queue depth balloons is the difference between a graceful backpressure response and a multi-hour backlog.

50 million jobs all due at 9:00:00am is the scheduler world's signature failure mode — the mitigation strategy below has to be correctness-aware, not just throughput-aware, because some of the intuitive fixes quietly break the guarantee that a job doesn't run before it's due.

Correct mitigations

Pre-stage without early execution — the dispatcher can begin reading and preparing the due batch slightly ahead of the exact due time, but must never actually dispatch a job before its due_at. 'Start moving jobs at 8:59:00' is only safe if that means queuing internal work, not making the job eligible to execute a minute early — conflating the two silently breaks the scheduling contract. Bounded batch retrieval — pull due jobs in fixed-size pages (the LIMIT 500 pattern from the SKIP LOCKED section) rather than one unbounded query for the whole due set. Partition the due set — split the burst across shards (per the sharding section above) so no single dispatcher instance absorbs the full 50M-row spike alone. Rate-limited dispatch — cap outbound dispatch throughput to what downstream systems (queue brokers, workers, third-party APIs) can actually sustain, spreading the burst over a window even though every job is still technically 'due' at the same instant. Allowed jitter windows — where the job's semantics tolerate it (a reminder, a report — not a hard-deadline payment), add a small random offset at creation time so 50M independently-created 9am schedules don't all resolve to the exact same second in the first place. Autoscale ahead of a known burst — if the burst is predictable (a fixed daily time), pre-warm worker capacity before it hits rather than reacting after queue depth spikes. Downstream quotas and priority/fairness — apply the same per-tenant and priority mechanisms from the sections above so the burst doesn't let one job class or tenant crowd out everything else.

Kafka, SQS, and a Redis-backed queue are not interchangeable — treating 'push jobs to a queue' as a single undifferentiated choice hides differences that directly affect correctness and operational behavior.

Queue depth alone tells you almost nothing about whether a task scheduler is healthy — it doesn't distinguish a brief absorbed spike from a slow leak, and it says nothing about correctness metrics like duplicate execution.

Metrics worth tracking

Scheduling lag (time between due_at and the job entering the dispatch pipeline), dispatch lag (time between claim and actual publish to the queue), queue wait (time a message sits in the queue before a worker picks it up), execution duration, and end-to-end completion latency (due_at to SUCCEEDED) together decompose 'is this slow' into which stage is actually slow. Overdue job count (due jobs not yet dispatched) and retry rate flag emerging backlog and instability respectively. Duplicate attempt rate and lease expiry rate are correctness signals, not just performance ones — a rising duplicate rate is early warning that idempotency assumptions are being exercised more than expected, and a rising lease expiry rate often precedes a wave of duplicate attempts. DLQ ingress rate and oldest DLQ age together show whether failures are accelerating and whether anyone is actually working the backlog. Per-tenant backlog surfaces the priority/fairness failure mode before it becomes a support ticket. Scheduler shard ownership (which node owns which shard right now, and for how long) makes rebalancing and hot-shard problems visible instead of invisible. Outbox publication lag (time between an outbox row being written and actually published) is the single metric most likely to catch a stuck or crashed relay before it becomes 'jobs are silently not running.'

Multi-region task scheduling is an advanced extension, not a default requirement — and the questions worth asking are mostly about where authority lives, not about topology diagrams.

The questions that actually matter

Where is schedule write authority — can a job be created or modified from any region, or does it have a home region? Where is due-time ownership — which region's scheduler instance is actually responsible for deciding this specific job is due and dispatching it? Can two regions dispatch the same job — and if the honest answer is 'yes, under some failure mode,' what makes that safe (idempotent consumers, a global fencing mechanism) rather than merely rare? How is failover fenced — when a region fails and another region takes over its jobs, what prevents the original region from resuming and dispatching the same jobs a second time if it recovers unexpectedly? What is the RPO/RTO — how much committed schedule data can be lost in a region failure, and how long can dispatch reasonably be paused during failover? What happens to jobs due during the actual failover window — are they queued and caught up afterward (see misfire/catch-up above), or does the design accept a bounded gap?

Prefer a clear baseline before active-active

A single home region (or a single home shard-authority) per job, with durably replicated state to a standby region, is a far simpler starting point than true active-active dispatch across regions — and it directly reuses the fencing and leader-election mechanisms already built for the single-region case, just applied at region granularity. Active-active dispatch (multiple regions independently able to decide a job is due and dispatch it) requires solving global fencing and cross-region consensus on due-time ownership, which is a substantially harder problem than anything else in this design — introduce it only once the single-home-region-plus-failover baseline has a proven, specific limitation it can't meet.