Designing a Distributed Task Scheduler

Every production system needs to run code at a specific point in the future. Send a password reset email in 15 minutes. Generate a weekly report every Monday. Run payroll on the last day of the month. These deferred, time-sensitive jobs are what a task scheduler exists to handle.

The core loop appears simple: store a job with a future execution time, detect when it's ready, hand it to a worker, mark it done. But at scale — millions of jobs, distributed execution, hardware failures, network partitions, and at-least-once delivery guarantees — every naive approach collapses.

Why cron breaks at scale

Cron was designed for single-machine, single-process operation. It has no awareness of distributed systems, no retry logic, no duplicate-execution prevention, no job history, and no horizontal scaling. Whether a crashed cron daemon actually loses jobs scheduled during its downtime depends on the design: a cron process that merely triggers work with no durable backing table genuinely does lose that work. But a cron process that just scans a durable table for run_at <= NOW() will pick up overdue rows again on restart — the real limitations of single-cron are single point of dispatch, coarse (typically minute-level) granularity, no coordination across multiple instances, and hard horizontal scaling, not an inherent guarantee of silent job loss.

Real-world examples

Before designing anything, establish what the system must do and what guarantees it must actually provide — and be precise about what "run at 09:00:00" means, because a distributed scheduler cannot promise the exact instant.

What "scheduled at 09:00:00" actually means

A practical distributed scheduler does not promise execution at the exact timestamp. The precise contract is: the job becomes eligible for execution at or after 09:00:00, and the system targets a bounded scheduling delay. It's worth separating four distinct points in time that are easy to conflate: the scheduled/eligible time (when the job becomes runnable), the dispatch time (when the scheduler hands it to the queue), the worker start time (when execution actually begins), and the completion time. The gap between the first two is dispatch_lag = dispatched_at - scheduled_at, and the gap to actual execution start is start_lag = started_at - scheduled_at — both are SLOs the system should measure and bound, not zero-cost guarantees.

Functional Requirements

Non-Functional Requirements

It's worth being explicit up front about what the design does and does not promise: jobs may run somewhat late under load (bounded by the scheduling-lag SLO); jobs may run more than once under retry/crash scenarios (idempotency handles this); overlapping recurring executions and missed-recurrence handling are governed by explicit, per-schedule policies rather than one fixed system-wide rule; and cancellation of an already-started execution is best-effort/cooperative, not a hard guarantee.

Sizing a task scheduler requires understanding job ingestion rate, job storage size, polling frequency, and worker throughput. Every number below is an illustrative assumption to reason with, not a universal threshold that determines which architecture is "correct."

Job storage

As one illustrative scenario: 10 million active scheduled jobs, each record roughly 8 bytes (job_id) + 8 bytes (scheduled_at) + 8 bytes (status) plus metadata — call it a few hundred bytes — if the due-time index stores only job_id + scheduling metadata rather than the full task payload. The larger payload (which can be kilobytes) belongs in the durable job store or an object store, referenced by id, not duplicated into every scheduling index entry or Redis member — storing full payloads in the due-time index is one of the most common ways to make that index unnecessarily expensive to hold in memory or scan.

Throughput

Illustrative numbers to reason with, not fixed boundaries: a moderate schedule-creation rate (well under the write capacity of a single primary database), a peak due-job rate that can spike far above the steady-state average (the classic '9 AM burst' where many independently-created schedules resolve to the same nominal time), and a retry amplification factor (what fraction of attempts fail and re-enter the pipeline) that multiplies effective load beyond the raw schedule-creation rate.

Polling and dispatch cost

Naive polling — SELECT * FROM jobs WHERE scheduled_at <= NOW() AND status = 'pending' on every tick — is a full table scan without the right composite index. With an index on (status, scheduled_at), a bounded-batch query (LIMIT 500) is inexpensive under normal load, but a burst where a large fraction of the table becomes due at once still stresses the index and I/O regardless of indexing quality — this is a genuine scaling consideration, not something indexing alone solves.

Worker sizing and burstiness

Worker capacity should be sized against the peak due-job rate, not the average — a workload with high burstiness (many jobs due in the same second) needs either enough standing worker capacity to drain that burst within the scheduling-lag SLO, or a deliberate, bounded, rate-limited dispatch strategy that spreads the burst over a longer window at the cost of some added lateness. Which trade-off is right depends entirely on the product's tolerance for lateness — there's no universal 'right' number of workers independent of that tolerance.

The simplest possible task scheduler: a cron daemon polls a database table every minute. When it finds jobs past their execution time, it runs them.

How it works

1. Application inserts a row into a scheduled_jobs table with a future run_at timestamp. 2. A cron job runs every minute: SELECT * FROM scheduled_jobs WHERE run_at <= NOW() AND status = 'pending'. 3. For each job, the cron process executes the task handler function inline. 4. It updates the job status to 'completed' or 'failed'.

When this works

A single-server cron approach works fine for: fewer than roughly 10,000 total scheduled jobs (a rough, illustrative threshold), job execution times under the cron interval (1 minute), single server deployments, low-stakes jobs where occasional missed executions are acceptable.

When it breaks

At scale, five failure modes emerge simultaneously.

The first real improvement: multiple workers claiming rows from the database, using PostgreSQL's SKIP LOCKED to prevent duplicate claims without a separate coordination service.

The key insight: pessimistic row-level locking

PostgreSQL supports SELECT ... FOR UPDATE SKIP LOCKED. When multiple workers execute this query concurrently, each one atomically claims a different set of rows; workers that encounter an already-locked row skip it instead of blocking. Within the claiming transaction itself, no two workers pick the same job.

Why it works, and what it doesn't guarantee

No separate lock manager, no ZooKeeper, no Redis is required — the database transaction itself is the coordination mechanism during the claim. Workers can be added or removed freely, and SKIP LOCKED prevents two workers from claiming the same locked row during that claim transaction. It does not, on its own, provide exactly-once execution after a worker crash: a worker that claims a row and then dies mid-execution has still claimed it, and recovering that abandoned claim (via lease expiry) can hand it to a second worker — which is why idempotent execution and lease-based recovery, not the claim step alone, are what make this safe across retries.

Retry and lease recovery

A job in 'claimed' or 'running' state that never completes (worker crash) needs recovery: any row with a status of 'claimed'/'running' and a lease_until in the past is considered abandoned. A background reaper resets it to 'pending' so it can be claimed again. This requires the lease/heartbeat columns shown above, not just a bare status flag.

When Approach 1 fails

Replace the database polling loop with a Redis Sorted Set — using job execution timestamps as scores. Efficient range queries replace expensive table scans, at the cost of taking on a second stateful system.

The data structure

A Redis Sorted Set is a collection where every member has a score; members are retrievable in score order. ZRANGEBYSCORE returns all members within a score range in typically O(log N + M) time — well suited to 'give me all jobs due before NOW()'.

Atomic claim pattern

The claim-and-remove must be atomic to avoid two pollers claiming the same entry. A Lua script (executed atomically by Redis) can: find due jobs via ZRANGEBYSCORE, remove the claimed jobs from the pending set, and add them to a 'running' set with a lease expiry. Concurrent pollers each get disjoint sets of jobs.

Performance characteristics — stated carefully

Insert/update is typically O(log N); range query is typically O(log N + M). At large N this remains logarithmic, not constant — a set of 10 million entries costs roughly log2(10M) ~ 23 comparisons per operation, which is small in absolute terms but is still a logarithmic bound, not 'effectively constant time.' Avoid stating fixed throughput numbers (e.g. '1M+ ops/sec' or '100,000 jobs/sec') as guarantees — actual throughput depends heavily on payload size, network round-trip, and persistence configuration.

What this pattern is, and isn't

A Redis sorted set is one common delayed-scheduling primitive, not universally 'the' industry-standard solution. Several job libraries across ecosystems use a similar approach; it is a widely used, well-understood pattern for exactly this kind of due-time indexing.

Durability depends on configuration — and on what Redis is being asked to be

Redis persistence and failover behavior depend entirely on configuration (RDB snapshots, AOF, appendfsync policy, replica topology) — appendfsync everysec bounds typical loss to roughly the last second of writes under normal conditions, but that is not a universal guarantee under every failure topology. This matters differently depending on Redis's role: if PostgreSQL is the source of truth and Redis is only a due-time index, Redis needs to be treated as rebuildable and the system needs a reconciliation path so a missing Redis entry never permanently loses a durably-recorded job — see Durable Store + Redis Index Consistency for the full mechanics. If Redis itself is authoritative for job existence, the durability requirements on it become much stronger.

When Approach 2 fails

For fault tolerance with a single coordinated dispatcher, run multiple scheduler nodes with leader election: only the leader dispatches jobs, and followers watch for leader failure and take over. This is one valid coordination model, not the only way to scale a scheduler — competing DB claimers (Approach 1), sharded ownership, time-bucket ownership, and broker-native delayed delivery are all alternatives with different trade-offs, covered in Trade-offs & Decisions.

Leader election mechanism — done correctly

A bare EXPIRE scheduler:leader ... refresh with no ownership check is unsafe: if the lease already expired and a different node acquired it, the original leader's blind EXPIRE call extends the new leader's lease using the old leader's schedule — both nodes can end up believing they're leader. Every renewal (and every release) must atomically verify current value == my token before acting, exactly as shown above.

Leader election alone does not prevent split-brain

A lease TTL bounds how long a stale leader can go undetected — it does not prevent that stale leader from acting during the window before it notices its lease expired. The failure sequence: the leader pauses (GC stall, CPU starvation, network partition) longer than the TTL; the lease expires; a follower acquires it and becomes leader; the original leader resumes and, unaware anything happened, continues believing it's still leader; without a further safeguard, both nodes can act as leader simultaneously.

Fencing closes the gap

A fencing/epoch token — a monotonically increasing number issued on every leadership change — tags every downstream write the leader makes. Downstream systems reject any write tagged with an epoch older than the highest one they've already seen. This, combined with idempotent dispatch on the receiving end, makes a stale leader's stray actions harmless duplicates rather than silent corruption, even in the rare window where two nodes briefly both believe they're leader.

Why one leader for dispatch (when you choose this model)

Dispatching from a single leader eliminates the race of multiple nodes claiming the same job at once. Workers remain fully distributed — the leader only decides what to enqueue; workers execute independently.

Sharded leadership

At very high scale, a single leader is a throughput bottleneck. Solution: shard by job type, customer ID, or hash range. Each shard has its own leader. Illustratively, if each leader sustained 10,000 jobs/sec (a workload-dependent assumption, not a benchmark), 10 shards would give 100,000 jobs/sec total without any single leader becoming hot.

Worker pool

Workers are stateless and horizontally scalable. They pull jobs from a queue (Kafka, SQS, Redis Queue). Each worker maintains a heartbeat. The scheduler monitors heartbeats and re-enqueues jobs from workers that stop heartbeating.

Retry and DLQ

Failed jobs enter a retry queue with exponential backoff: 1s, 2s, 4s, 8s, ... up to a maximum delay. After N failures (e.g., 5), the job moves to the Dead Letter Queue. DLQ jobs are never automatically retried — they require manual intervention or a separate recovery process.

The production-grade distributed task scheduler combines all three approaches into a layered architecture: a durable job store, a fast dispatch layer, horizontally scaled workers, and full observability.

Component breakdown

This is the architecture the rest of this page builds toward — the Architecture Deep Dive covers each of these components' internal mechanics (due-time indexing, durable dispatch, leader fencing, worker leases, idempotency) in far more depth.

A production job lifecycle needs to separate two things the naive model conflates: the logical job (or schedule)'s state, and the state of each individual execution attempt against it. A job can survive several failed attempts without the job itself ever being 'failed.'

Logical job / schedule states

SCHEDULED (eligible to fire, or waiting for its next occurrence), CANCELLED, COMPLETED, PAUSED (recurring schedules only), and DEAD/TERMINAL_FAILED (retries exhausted or a permanent failure). These describe the job's overall lifecycle, not any single execution attempt.

Execution attempt states

READY/DISPATCH_PENDING (claimed, not yet enqueued), QUEUED (published to the dispatch queue), LEASED/RUNNING (a worker holds the lease and is executing), SUCCEEDED, FAILED_RETRYABLE, FAILED_PERMANENT, and TIMED_OUT. Every attempt has its own row and its own history — attempt 3 failing doesn't erase the record of attempts 1 and 2.

The orphaned attempt problem, precisely

A worker whose process crashes cannot mark its own attempt FAILED — it's gone. What actually happens: the worker publishes a heartbeat periodically while it holds the lease; if heartbeats stop, a separate reaper component observes that the lease has expired (not that the worker "failed" — the reaper has no idea what happened to the worker) and reclaims the attempt, creating or permitting a new attempt rather than mutating the dead one. This is a meaningful distinction: the system is inferring failure from absence, not receiving an authoritative failure signal, and everything downstream (idempotency, dedup) has to account for that uncertainty.

Durable job creation and idempotent requests

A client that creates a job can time out after the database commits but before the response arrives, and reasonably retry the creation request. Without an idempotency key, that retry creates a second logical job for the same intended work. The fix is the same one used broadly for idempotent APIs: require a stable client-supplied request/idempotency key on job creation, so one logical scheduling request — retried any number of times — results in exactly one logical job or schedule record.

The dual-write problem: job store to queue

Marking a job SCHEDULED -> QUEUED in the database and publishing to the dispatch queue (Kafka/SQS) are two separate operations against two separate systems — there is no way to make them atomic by default. If the DB commit succeeds and the process crashes before publishing, the job is durably marked QUEUED but no message was ever sent — it's stranded unless something notices. If the queue publish succeeds and the process crashes before the DB records the dispatch, a retry of the whole operation can publish a second message for the same logical dispatch.

The baseline fix is the transactional outbox pattern: in one database transaction, claim the due job, write a durable outbox/dispatch record, and update the attempt state — all three commit together or not at all. A separate relay process reads unpublished outbox rows, publishes them to the queue, and marks them published. The relay's own publish step can still be repeated (a crash between publish and mark-published), which is exactly why consumers must remain idempotent regardless — the outbox closes the silent-stranding gap, it does not manufacture atomicity across the DB and an arbitrary external queue. Calling this combination "atomic" requires naming the mechanism that makes it safe.

Worker lease semantics

A worker lease means: this worker currently owns the right to make progress on this attempt. It does not mean no duplicate external side effect can occur — those are different guarantees, and conflating them is a common design mistake. A lease record needs a unique attempt identity, a lease owner, a lease expiry, a generation/fencing token (so a stale lease-holder's late actions can be detected and rejected), and periodic heartbeat renewal that atomically checks current ownership before extending.

Avoid the common but imprecise advice that lease TTL should always be "significantly longer than worst-case execution time" — for long-running or unbounded-duration jobs, that's impractical (see Long-Running Jobs in the Deep Dive). The better pattern: keep the lease comfortably longer than the heartbeat interval and expected transient pauses, renew it periodically while work continues, detect failure based on missed renewals rather than total elapsed time, and apply a separate maximum execution timeout only where the product genuinely needs one.

Worker idempotency, precisely

"Check before acting" is not, by itself, safe advice — a check and the subsequent act can race (two workers both check, both see "not yet done," both act), and it offers no protection for side effects against arbitrary external systems regardless of how carefully it's implemented locally. What actually works depends on where the side effect lives: for a local database mutation, use a unique constraint, a transaction, and an atomic upsert or conditional update — the database itself enforces the invariant, not a race-prone check. For an external API that supports an idempotency key, pass one and let the provider deduplicate. For an external API with no idempotency support, the honest options are durable operation state plus reconciliation, business-specific deduplication, or accepting that an exactly-once effect may not be fully achievable — a scheduler-level lock (Redis or otherwise) is not a universal exactly-once mechanism for effects outside the scheduler's own storage.

Cancellation semantics, precisely

"Running jobs cannot be cancelled mid-flight" oversimplifies three genuinely different cases. A pending/scheduled job (not yet dispatched) can usually be cancelled strongly and immediately. A queued-but-not-yet-started job is race-prone: the cancellation and the worker picking up the message can interleave, so the worker must re-check the job's current state and version before executing, and discard a stale message for an already-cancelled job. A running execution may support cooperative cancellation — the worker periodically checks whether it's been asked to stop at safe points — but cannot be forcibly halted mid-instruction without risking a corrupted partial side effect, and any external side effects already performed cannot necessarily be undone. Rescheduling follows the same version-check discipline: bump the job's version, invalidate the old due-index entry, add a new one at the new time, and let the version check reject any stale in-flight dispatch against the old time.

There are several real mechanisms for implementing "run job X at time T," and none of them is universally best — each trades precision, persistence, complexity, and recovery behavior differently, and the right choice depends on scale and the specific correctness requirements of the workload.

Cancellation and update behavior differ sharply too: a database row can simply be updated or deleted; a Redis sorted-set member can be removed or re-scored in O(log N); a timing wheel entry may require locating its current slot; and a message already inside a broker's native delay queue generally cannot be updated or cancelled at all once enqueued. Burst handling also differs: DB/Redis/time-bucket approaches can apply bounded-batch retrieval and rate-limited dispatch to smooth a spike; a timing wheel fires everything scheduled for the same tick essentially at once, which needs an explicit dispatch-rate limiter downstream of it.

Recurring schedules: definition vs. individual occurrence

A recurring schedule ("every day at 9am") is not itself a job — it's a template that generates a stream of individual occurrences over time. weekly-report schedule -> execution for 2026-09-21 -> execution for 2026-09-28, and so on. Conflating the schedule with any one occurrence is how systems end up either duplicating or silently dropping runs across restarts and failovers.

Each occurrence needs a deterministic identity, typically occurrence_id = f(schedule_id, nominal_fire_time), so that a scheduler retry, a failover, or a recovery scan that re-evaluates the schedule produces the same occurrence identity rather than a new logical duplicate — inserting with that deterministic id under an idempotent constraint turns a repeated generation attempt into a harmless no-op. A schedule record needs, conceptually: the recurrence rule itself (cron expression or interval), an explicit IANA timezone, a precomputed next-fire-time, a misfire policy, an overlap policy, a version (for safe concurrent updates), and a paused flag.

Time zones and DST

For a schedule like "every day at 09:00 Europe/London," the system must store the recurrence rule and the IANA timezone together, and compute each occurrence's absolute instant fresh, according to that timezone's rules, at generation time — not convert the wall-clock time to a fixed UTC offset once at creation and reuse it forever, because the UTC offset for 9am London changes across the DST boundary. Two DST edge cases need an explicit product policy: the "spring forward" transition can make a local time like 2:30am not exist at all on that day, and the "fall back" transition makes a local time like 1:30am occur twice — the schedule needs to define which occurrence it means, rather than leaving it to whatever a naive time library happens to resolve. For one-time absolute jobs (not recurring), storing a normalized instant such as UTC from the start avoids this class of problem entirely.

Misfire and catch-up policy

When the scheduler was down and recovers, recurring schedules may have missed one or more occurrences. This needs an explicit, per-schedule policy rather than one system-wide default: run every missed occurrence (dangerous unless occurrences are cheap and fully idempotent), run only the most recent missed occurrence, skip all missed occurrences and resume from the next future one, or coalesce missed work into a single catch-up execution. Automatically dispatching every missed recurrence after a long outage is rarely the right default — it can recreate a self-inflicted thundering herd, and for some job types (a daily reminder run twice) it's simply wrong.

Overlap and concurrency policy

A separate and distinct problem from duplicate execution: recurring execution N may still be running when occurrence N+1 becomes due. The available policies — allow overlap (fine for cheap, independent occurrences; dangerous for anything sharing state or resources), skip the new occurrence, queue the next occurrence to start once the current one finishes, replace/cancel the prior attempt where that's safely supported, or serialize execution per schedule key — again need to be chosen per schedule based on what the occurrences actually do, not applied uniformly.

Retry logic is where most task scheduler implementations have subtle bugs. Getting it right requires classifying why a job failed, not just counting how many times it's been tried.

Exponential backoff with jitter, capped

Retry delay formula: base_delay * 2^attempt + random_jitter, capped at a maximum delay so backoff doesn't grow unbounded into days or weeks. Without jitter, every job that failed at the same instant retries at the same instant again — trading one thundering herd for a synchronized retry storm.

Classify the failure before deciding how to retry

Transient (timeout, connection reset) — retry with standard backoff. Throttling (an explicit rate-limit or 429 response) — retry, but honor any Retry-After guidance and back off more aggressively than the default curve. Permanent (validation error, business-rule rejection, auth failure) — do not retry; route straight to the DLQ, since retrying an inherently invalid request wastes attempts and delays visibility into an actual bug. Unknown/ambiguous (a timeout with no confirmation the side effect didn't happen) — retry only because the operation is idempotent, not because the failure looked safe.

What the retry engine needs to track, beyond a counter

Attempt count, next_retry_at, last error class, first-failure time (for enforcing an overall deadline independent of attempt count), an explicit deadline past which no further retry happens regardless of remaining budget, and the retry policy version that applied — so a later policy change doesn't reinterpret already-recorded history incorrectly.

Retry budgets and dependency-wide protection

A per-job retry limit isn't the only safeguard worth having: a retry budget (a cap on total retry volume across a time window, independent of any single job) and a circuit breaker or rate limiter in front of a failing downstream dependency prevent a single struggling dependency from being hammered by every affected job's independent retry schedule simultaneously — which can turn a partial outage into a complete one.

Dead Letter Queue and safe replay

A DLQ preserves the original job payload, all failure reasons, and full attempt history for jobs that exhausted retries or failed permanently. It must trigger an alert as the DLQ grows — illustrative thresholds like "more than 10 jobs/min" or "more than 1,000 jobs" are examples to calibrate for your own workload, not universal numbers. Replay needs its own safety rules: a replay creates a new attempt while preserving the original logical job identity (so the full history stays linked); bulk replay must be rate-limited, for exactly the same reason initial dispatch is — to avoid a self-inflicted burst; before replaying, verify the original root cause is actually fixed, not just that time has passed; attempt and error history from before the replay must be preserved, not overwritten; and operator replay/discard actions need authorization and an audit log, since they're production actions with real consequences, especially for anything touching money or user-facing state.

Task schedulers scale in stages, but the trigger for moving to the next stage should be a measured bottleneck, not a fixed jobs-per-day threshold — two teams at the same job volume can have very different bottlenecks depending on job size, burstiness, and downstream capacity.

What actually triggers moving to the next stage

Measure, don't guess: query load and lock contention on the claiming query, dispatch lag against the SLO, queue depth and oldest-message age, due-bucket size at peak, worker utilization, downstream throttling signals, and (once sharded) per-shard/leader saturation. A team that never sees these metrics approach their limits has no reason to move to the next stage, regardless of raw job volume.

Understanding how each component fails — and how the system recovers — separates a reliable scheduler from a fragile one. See Failure & Scale Scenarios for a much deeper treatment of these and other failure modes.

Every architectural decision in a task scheduler involves an explicit trade-off. The four below are a quick-reference summary — the full set of fifteen, with deeper analysis of each, lives on the dedicated Trade-offs & Decisions page.