Failure Scenarios & Scale Challenges

A worker process dies (OOM, hardware fault, deploy) while holding the lease on a RUNNING attempt.

If undetected, the job silently never completes — worse for the caller than a visible failure, since there's no error to react to.

The worker's heartbeat stops. A separate reaper process periodically scans for attempts whose lease has expired without a completion signal.

The reaper reclaims the lease and permits a new attempt — it does not, and cannot, know whether the original worker's side effect actually completed before it died.

A new attempt executes via the normal retry path, subject to the job's idempotency guarantees.

Keep the lease comfortably longer than the heartbeat interval (not a universal 30s/90s pair — tune both to your actual execution-time distribution and false-positive tolerance), and make the underlying operation idempotent so a possible duplicate attempt is harmless.

The scheduler process (or the current leader, in an elected setup) stops running.

New dispatch halts. Jobs due during the outage accumulate as overdue but are not lost, provided their state lives durably in the job store rather than only in the crashed process's memory.

Without leader election: no automatic detection — the gap is noticed via monitoring (rising scheduling lag, growing overdue count) or a human report. With leader election: a follower detects the absence once the lease expires.

On recovery, do not blindly dispatch every overdue job at once — apply the job or schedule's misfire policy (run latest, skip, or coalesce for recurring schedules; dispatch with normal priority for overdue one-time jobs) and release the backlog through the same bounded, rate-limited path as any other burst.

Without election: manual restart, then a bounded catch-up scan. With election: a follower becomes leader once its lease acquisition succeeds, then performs the same bounded catch-up.

Leader election with fencing bounds detection time to roughly the lease TTL, but actual failover time also depends on how quickly the new leader's catch-up scan completes — it is influenced by, not universally equal to, the TTL alone.

The database primary becomes unreachable or fails outright.

Job creation requests fail (should return an explicit error, not silently succeed). Workers can often drain jobs already dispatched before the outage, but cannot record status updates — creating a window where execution and durable status can disagree.

Standard database health checks and replication-lag monitoring; application-level write failures surface immediately to callers.

Reject new job-creation requests explicitly (503, or an equivalent clear failure) rather than silently accepting them into a non-durable buffer. A client that times out and retries should use the durable job-creation idempotency key from the Architecture Deep Dive, so a retry after recovery doesn't create a duplicate job.

Promote a replica to primary. With properly acknowledged synchronous replication, committed-transaction loss can be zero for the configured synchronous failure model — at the cost of higher write latency and availability sensitivity during normal operation. With asynchronous replication, the actual data loss (RPO) is determined by the real replication lag at the moment of failure, not a fixed 'typically under a second' figure — it can be more under load or during a slow link.

Avoid recommending local Redis buffering of job-creation requests as a generic high-availability fix — a buffer that isn't itself durable and reconciled just moves the same failure window one layer down. A safer baseline is explicit rejection plus client retry with an idempotency key, or a separately designed durable ingress path built specifically to tolerate primary unavailability.

The dispatcher's DB transaction (claim + state transition) commits, but the process crashes, times out, or the broker rejects the call before the message is actually published.

The job is durably recorded as dispatched but no worker will ever receive it — silently stranded, unless something notices.

A transactional outbox (or equivalent durable dispatch record) with a monitored publication lag metric — an outbox row sitting unpublished past a threshold is directly observable, unlike a bare DB-then-publish design where there's no record to check.

Never treat 'DB says dispatched' as equivalent to 'a worker will receive this' without a durable record of the publish step itself.

The outbox relay retries publishing the unpublished record until it succeeds, using the queue's own retry/idempotent-publish semantics.

Build the transactional outbox pattern from the start for anything where silent stranding is unacceptable — the job/attempt state transition and the durable dispatch record commit in the same database transaction, and a separate relay (or CDC pipeline) is solely responsible for actually publishing it, with retry.

The publish call to the broker succeeds, but the process crashes or the transaction fails before the database records the dispatch as complete.

If the scheduler retries the entire dispatch operation (because, from its perspective, the DB write never confirmed), the job may be published to the queue a second time — two messages for one logical dispatch.

A rising duplicate-attempt-rate metric, or two attempt records observed for what should be a single dispatch of the same job.

Every dispatched message should carry a stable dispatch/attempt ID. A duplicate message for the same attempt ID is not a new unit of work — it's a redelivery.

The worker (or a dedup layer in front of it) recognizes the repeated attempt ID and either discards the duplicate outright or executes it against the same idempotency guarantees as any other retry.

Idempotent workers plus a stable attempt ID handle this by design — the fix isn't preventing the duplicate publish (which is hard to guarantee across DB + broker), it's making sure a duplicate is always safely recognizable and harmless downstream. Periodic reconciliation between recorded dispatch attempts and observed executions catches anything that slips through.

A successful side effect's confirmation signal (queue ACK or durable completion write) is lost, independent of the effect itself succeeding.

From the scheduler's perspective alone, this is indistinguishable from the effect never having happened — so without further protection, a retry appears identical to a legitimate first attempt.

There is no local signal that distinguishes this case — detection has to happen at the effect's own boundary: a downstream idempotency key rejecting the second charge, or a reconciliation pass comparing recorded attempts against the provider's actual transaction log.

Never assume 'no ACK' means 'no effect' — the two are decoupled, and treating a lost ACK as proof of failure is how a system ends up performing the effect twice with full confidence it only ran once.

Worker B's retry calls the payment provider with the same downstream idempotency key Worker A used (or an equivalent stable business key). The provider recognizes the key and returns the original result instead of charging again — this is what actually converges the two attempts to one logical successful outcome, not the scheduler's own lease or ACK bookkeeping.

The job was never silently lost (the durable record persisted through both attempts); the duplicate dispatch was harmless on the scheduler's side (a normal at-least-once artifact); the lease reduced, but did not eliminate, concurrent execution; and the downstream idempotency key is what actually prevented a second real-world charge. This is the complete reasoning chain the scheduler's design has to support end to end.

A worker that's genuinely still making progress fails to renew its lease in time. Common causes: a GC pause or CPU starvation delaying the heartbeat thread, a network partition between the worker and the lease store, the heartbeat thread itself failing independent of the main work, or an outage of the lease store (Redis or the database) preventing any renewal from landing regardless of the worker's own health.

Two workers may now be executing the same logical attempt concurrently — the exact scenario idempotency has to handle, not a rare edge case.

A rising lease-expiry rate combined with a rising duplicate-attempt rate is the signal this is happening at meaningful volume, not a one-off.

A lease expiry does not imply the original worker is dead — it implies only that renewal didn't happen in time, which has several possible causes besides death.

Where the resource supports a fencing/generation token, the second worker's completion write is accepted and the first worker's late completion (if it arrives after the generation has moved on) is rejected or treated as a harmless duplicate signal rather than allowed to silently overwrite newer state. Idempotent side effects are required regardless of whether fencing is available, since fencing only protects the scheduler's own bookkeeping, not an external system the first worker may have already reached.

Size the lease relative to the heartbeat interval and realistic pause durations for your runtime (GC behavior, typical network jitter) rather than an arbitrary fixed number, and treat idempotency as mandatory rather than optional insurance against this specific race.

A leader experiences a long pause (GC stop-the-world, CPU starvation, network partition) exceeding the lease TTL, a follower acquires leadership during that window, and the original leader later resumes execution unaware anything changed.

Without a further safeguard, both nodes may act as leader simultaneously — potentially double-dispatching the same due jobs from two independent dispatch loops.

A fencing epoch mismatch is directly detectable: any write tagged with an epoch older than the highest one already observed is identifiably stale, which is a much stronger signal than trying to infer split-brain from dispatch symptoms alone.

Never assume a lease TTL alone prevents both nodes from acting — it only bounds how long the stale leader can go undetected, not whether it can still act during that window.

The new leader (with a strictly higher fencing epoch) continues normally. The stale leader's actions, tagged with an old epoch, are rejected by any fencing-aware downstream consumer; combined with idempotent dispatch, its actions become harmless duplicates rather than corrupting state.

Fencing/epoch tokens on every leader-originated write, conditional (not blind) job-state transitions that check current version before mutating, and idempotent dispatch consumers throughout — these three together are what make a stale leader's resumed activity safe rather than merely rare.

The Redis instance or cluster serving as the due-time index loses its data — through a crash without persistence, a botched failover, or an operator error.

Jobs may become late (nothing is querying for them via the now-empty index) but should not disappear, provided the database remains the authoritative record of their existence and due time.

Redis health/availability monitoring, plus a rising gap between the database's count of due jobs and Redis's index size.

Never treat the durable database record as lost just because its fast-path index is empty — the index is a rebuildable optimization, not the source of truth, and every consumer of it should already validate against the database before acting (see the Architecture Deep Dive's index-consistency section).

Rebuild the index from the database, typically prioritizing jobs already overdue or due soonest first. A temporary fallback to direct database polling for the affected window (if the design supports it) keeps dispatch functioning, at reduced throughput, while the rebuild completes. A reconciliation scan afterward confirms no due job was missed during the gap.

Design the index as fully and cheaply rebuildable from day one — treating Redis as disposable state rather than irreplaceable state is what turns this from an incident into a routine, bounded degradation.

The queue/broker (Kafka, SQS, or equivalent) is unreachable or degraded.

New dispatch to workers halts. Jobs remain durably due but undelivered — the outage should be visible as growing dispatch lag, not as lost jobs.

Broker health checks, and a directly measurable outbox/dispatch-lag metric (time between a job being claimed/ready and actually published) rising past normal bounds.

The dispatcher must not mark a job as irreversibly dispatched without durable evidence the publish actually succeeded — doing so risks the same silent-stranding failure as the DB-commit-succeeds-queue-fails scenario, just triggered by a longer outage instead of a single transient failure.

Once the broker recovers, the outbox relay (or equivalent) resumes publishing the backlog of unpublished dispatch records — ideally rate-limited, since a long outage can produce a large backlog that would otherwise recreate a catch-up storm.

Avoid unbounded in-memory buffering of jobs awaiting publish as a way to 'ride out' the outage — an in-memory buffer is exactly the kind of state that disappears if the dispatcher process itself restarts during the outage. Rely on the durable outbox record, not an in-memory queue, as the thing that survives.

A downstream dependency that job execution relies on becomes unavailable or starts failing at a high rate.

Every affected job fails and, without coordination, independently retries against the same struggling dependency — a retry storm that can prevent the dependency from ever recovering, and can also exhaust worker capacity that unrelated job classes need.

A spike in failure rate concentrated on jobs calling a specific dependency, plus explicit throttling/error signals (429s, timeouts) from that dependency.

Do not let every failing job retry on its own independent schedule against a dependency that's already struggling — that's precisely what turns a temporary dependency blip into a sustained outage.

Backoff with jitter (as always) plus a circuit breaker that stops sending traffic to the dependency once its failure rate crosses a threshold, giving it room to recover; a shared retry budget across all jobs hitting that dependency prevents them from collectively overwhelming it even with individually reasonable backoff.

Rate limiting and a retry budget scoped to the dependency (not just per-job), the ability to pause the affected job class specifically while preserving unrelated capacity for everything else, and alerting that distinguishes 'a downstream dependency is down' from 'our scheduler is broken' — these are different incidents needing different responses.

An extended scheduler or leader outage during which a very large number of jobs become due.

Without controlled recovery, the sudden release of the entire backlog can overwhelm the database, queue, workers, and downstream dependencies all at once — a worse spike than the original outage.

Overdue-job count and oldest-due-job age rising sharply is the leading indicator during the outage; the recovery scan itself should be observable (backlog size, dispatch rate during catch-up) so operators can see it's proceeding safely.

Never dispatch the entire discovered backlog in one unbounded burst just because the scheduler is finally back online.

Discover the backlog via a bounded scan (not a single unbounded query), apply each job or schedule's misfire policy (skip, catch-up-latest, or coalesce for recurring schedules), and release the resulting backlog through rate-limited dispatch, respecting priority so time-sensitive classes drain first, with an explicit deadline/expiry for jobs so overdue that running them late no longer has value.

Pre-provisioned autoscaling that can react to a detected catch-up event, downstream quotas that remain in force during recovery (not just steady-state), and treating the catch-up path as a first-class, tested piece of the design rather than something that's only ever exercised during a real incident.

Clocks across scheduler/worker nodes drift apart, even slightly, due to normal NTP jitter, virtualization effects, or a genuinely misconfigured clock source.

Time-sensitive decisions — lease expiry checks, due-time comparisons, misfire detection — can disagree slightly between nodes, occasionally causing a job to be treated as due, or a lease to be treated as expired, a few seconds earlier or later than another node would conclude.

Clock-drift monitoring (comparing node clocks against a reference, or against each other) and NTP sync-health checks.

Never assume all nodes agree on 'now' to sub-second precision — build a small safety margin into any comparison where being off by a second or two would cause an incorrect decision (lease expiry, misfire detection).

Standard NTP-based synchronization keeps skew small in practice; where a single authoritative clock matters more than distributed agreement (e.g. a single shard's due-time ownership), preferring database or single-server time for that shard's decisions removes cross-node comparison from the critical path entirely.

Tolerate small skew explicitly in your SLOs rather than assuming it away, use synchronized clocks (NTP) as a baseline, and never build a mechanism whose correctness depends on wall clocks being perfectly identical across nodes.

A recurring schedule's wall-clock fire time crosses a DST transition boundary in its configured timezone.

On a 'spring forward' transition, the configured local time may not exist that day at all — a naive implementation can produce an invalid instant or simply fail to fire. On 'fall back,' the configured local time occurs twice — a naive implementation may fire twice, or fire at the wrong one of the two instances.

Recurring schedules whose fire times don't match expectations around known DST transition dates for their configured timezone — worth explicitly testing against, not just discovering in production.

Never resolve a recurring wall-clock schedule to a single fixed UTC offset once and reuse it forever — the offset itself changes across the DST boundary, so each occurrence has to be resolved fresh, against the actual timezone rules, at generation time.

Apply the schedule's explicit, product-defined DST policy (e.g. 'on the nonexistent-time case, fire at the nearest valid time after the gap'; 'on the repeated-time case, fire only on the first occurrence') consistently, rather than however a particular time library happens to resolve the ambiguity by default.

Store the recurrence rule together with an explicit IANA timezone (never a fixed offset), define the DST edge-case policy as an explicit product decision up front, and use a normalized absolute instant (UTC) for one-time (non-recurring) jobs so this class of problem doesn't apply to them at all.

A cancellation request is recorded at approximately the same time the job has already been claimed and its dispatch message is in flight to (or already sitting in) the queue.

Without a safeguard, a worker can pick up the stale message and execute a job the caller reasonably believes was cancelled.

There's no failure signal here in the traditional sense — this is a design correctness question, verified by testing the race directly (cancel immediately after dispatch, confirm the worker declines to execute) rather than discovered via monitoring.

Every dispatch message carries the job's version at claim time; before executing, the worker re-reads the job's current state and version from the source of truth. If the current state is CANCELLED, or the version doesn't match what was dispatched, the worker discards the message instead of executing it.

No special recovery needed — a correctly discarded stale message simply results in the job staying cancelled, which is the desired outcome.

Build the version/state check into the worker's execution path as a mandatory pre-condition, not an optional safety net — and be explicit with users that an already-running execution's external side effects (an email already sent, a charge already made) may not be reversible even though the job is now marked cancelled.

A single tenant (or job class) creates or triggers an unusually large volume of due work concentrated in a short window, far exceeding typical per-tenant volume.

Other tenants' jobs experience elevated scheduling lag or dispatch delay, even though their own volume didn't change — they're being crowded out by a neighbor's burst, not by overall system load.

Per-tenant backlog and dispatch-lag metrics (not just system-wide aggregates) surfacing one tenant's share disproportionately — this is invisible in system-wide metrics alone, which is exactly why per-tenant observability matters.

Never let system-wide capacity be allocated purely first-come-first-served across tenants — a single large tenant will then set the experience for every smaller one during its own bursts.

Apply the tenant's rate limit or quota (throttling their burst to a bounded rate rather than rejecting it outright, where the product allows), and rely on reserved capacity or weighted fair scheduling to guarantee other tenants' jobs continue to make progress during the burst.

Per-tenant quotas and rate limits enforced at admission (not just at dispatch), reserved worker capacity carved out independent of any single tenant's volume, and partitioning (shard by tenant, or route large tenants to dedicated capacity) so one tenant's scale characteristics can't structurally affect another's.

An operator (or an automated recovery trigger) initiates a bulk replay of a large DLQ backlog after believing the original root cause is resolved.

If replay is unbounded, it can overwhelm the now-recovering downstream dependency a second time, potentially causing it to fail again — a replay-induced re-outage.

Replay progress and its effect on the target dependency's error rate should be directly observable in real time, not discovered after the fact.

Never treat 'the root cause is fixed' as sufficient justification to replay an entire DLQ backlog at unlimited speed — the volume itself is a burst risk independent of whether the underlying bug is fixed.

Replay through the same rate-limited, bounded-batch path used for any other backlog release, prioritized (time-sensitive job classes replay first), and idempotent (each replayed job gets a new attempt ID but is still subject to the same idempotency guarantees as any other attempt, in case some DLQ entries actually did partially succeed before failing).

Require authorization for bulk replay actions, log them for audit, rate-limit replay throughput by default rather than making it an opt-in safeguard, and verify the specific root cause is actually resolved (not just that time has passed) before triggering a large replay.

A full regional outage (infrastructure failure, network partition, provider-wide incident) affecting the region that currently holds schedule write authority and due-time ownership for a given set of jobs.

Jobs owned by the failed region stop being discovered and dispatched until failover completes. Jobs due during the failover window are delayed by however long failover and any subsequent misfire/catch-up handling takes.

Region-level health checks and cross-region monitoring; this needs to be distinguished clearly from a single-node or single-shard failure, since the response and expected recovery time are very different.

Failover to a standby region must be fenced exactly like single-region leader failover — a resumed original region must not be able to dispatch the same jobs the standby region has already taken over, which requires the same epoch/fencing discipline applied at region granularity.

The standby region assumes due-time ownership (using its replicated durable state) and begins dispatch, applying the misfire/catch-up policy for jobs that became due during the outage. State replication lag at the moment of failure determines the actual RPO for that specific incident.

Define, in advance, as explicit design targets rather than after-the-fact promises: where write authority lives under normal operation, how failover is fenced, what RPO and RTO the design is actually built to achieve (not merely hoped for), and what happens specifically to jobs due during the failover window. Exact recovery-time numbers are meaningful only when stated as targets the design is built and tested against.