Interview Perspective: Distributed Task Scheduler

Strong answer direction: it decides when an independently executable job becomes eligible to run and gets it durably dispatched — it is not a workflow/DAG orchestrator (it doesn't manage dependency graphs or multi-step durable workflow state), and it is not a message broker (it may use one, but its job is scheduling, not general pub/sub). What interviewer is testing: whether you can scope the problem precisely instead of describing a general-purpose distributed system. Read deeper: Complete Design overview.

• Would you build this differently for one-time jobs only, versus also supporting recurring schedules? — Recurring adds occurrence identity, misfire policy, and overlap policy — meaningfully more surface area than one-time jobs alone.

Strong answer direction: yes to all three under specific, named conditions — late is bounded by the dispatch-lag SLO, running twice is the expected shape of at-least-once delivery (made safe via idempotency), and overlap is a distinct policy question for recurring schedules (allow, skip, or queue the next occurrence). Treating these as one undifferentiated 'reliability' question is a common shortcut worth avoiding. What interviewer is testing: precision about which guarantee applies to which failure mode. Read deeper: Job Lifecycle and Overlap and Concurrency Policy.

• What's the maximum execution duration you'd support? — This determines whether you need renewable leases and progress checkpointing, or a fixed-lease model is sufficient.

Strong answer direction: it means the job becomes eligible for execution at or after 09:00, with a bounded scheduling-delay SLO (dispatch_lag = dispatched_at - scheduled_at) — not a promise of execution at the exact instant. Separating scheduled time, dispatch time, worker start time, and completion time avoids conflating four different points into one ambiguous guarantee. What interviewer is testing: whether you understand that a distributed scheduler cannot promise exact-instant execution, and can define what it promises instead. Read deeper: Requirements.

• What happens if the scheduler is down at exactly 09:00? — The job becomes late, not lost, provided its state lives durably — covered by misfire/catch-up policy on recovery.
• What if a job expires before it actually runs? — The design needs an explicit deadline concept so an extremely late job can be marked expired rather than dispatched with no remaining value.

Strong answer direction: an index on (status, due_at) with bounded-batch retrieval (LIMIT N, ORDER BY due_at) — whether backed by a database, a Redis sorted set, or a timing wheel, the core requirement is ordered retrieval of a bounded slice, not a full scan. What interviewer is testing: whether you reach for indexing and batching as the default, rather than either a full scan or an unbounded query. Read deeper: Due-Time Index Design.

• What happens with 50 million jobs sharing the same due timestamp? — Bounded batches and partitioning across index shards prevent one query from trying to return all 50M rows at once — see Thundering Herd.

Strong answer direction: either SKIP LOCKED-style competing claims (each claimer atomically skips rows another has locked) or leader election with fencing (only the leader claims, with a token preventing a stale leader's claims from being trusted). Both are valid; the trade-off is contention-under-scale versus election/fencing complexity. What interviewer is testing: awareness that this is a real trade-off with more than one right answer, not a single fixed pattern. Read deeper: Single Leader vs Competing Claimers vs Sharded Ownership.

• Which would you pick for a system with 10 scheduler nodes? — Depends on measured contention — competing claimers if contention is low, sharding if a single leader/claim point is a proven bottleneck.

Strong answer direction: without an outbox, the job is durably marked dispatched but no message was ever sent — silently stranded. The fix is a transactional outbox: the state transition and a durable dispatch record commit in the same DB transaction, and a separate relay retries publishing until it succeeds, tracked via a publication-lag metric. What interviewer is testing: recognition that DB-then-publish is not atomic by default, and knowledge of the actual production pattern that closes the gap. Read deeper: Durable Dispatch / Transactional Outbox.

• Can the outbox relay itself publish a duplicate? — Yes — if it crashes between publishing and marking the row published, it will republish on restart. This is why consumers must still be idempotent regardless of the outbox.

Strong answer direction: the database and the broker are two independent systems with no shared transaction coordinator between them — a crash between the two calls leaves them in disagreement, in either direction (DB updated but not published, or published but not recorded). Using the word 'atomic' here without naming the actual mechanism (outbox, CDC, or a genuinely transactional integration where one exists) is a red flag, not a correct shortcut. What interviewer is testing: whether you understand dual-write as a real correctness problem, not a detail to hand-wave past. Read deeper: Durable Dispatch / Transactional Outbox.

• Are there systems where this actually can be transactional? — Some databases support transactional outbox via native CDC (e.g. Debezium reading the WAL), which removes the separate relay process but doesn't remove the need for idempotent consumers.

Strong answer direction: the worker's heartbeat stops; a separate reaper process detects the lease has expired without a completion signal and reclaims the attempt, permitting a new one via retry — idempotency is what makes that retry safe, since the reaper cannot know whether the original side effect actually completed before the crash. What interviewer is testing: the full mechanical chain (heartbeat -> lease expiry -> reaper -> retry -> idempotency), not just 'it gets retried.' Read deeper: Worker Lease Protocol.

• Why can't the crashed worker just mark itself FAILED? — It's gone — there's no process left to write that state. Failure has to be inferred from lease expiry by a separate, still-running component.
• How do you handle a job that legitimately runs six hours? — Renewable leases (periodic renewal while making progress) plus optional progress checkpointing, not one fixed long TTL.

Strong answer direction: this is genuinely ambiguous from the scheduler's own perspective — a lost ACK looks identical to a job that never executed, so the queue or reaper will trigger a retry. The retry's safety comes entirely from downstream idempotency (a stable business key at the effect boundary), not from the scheduler correctly guessing what happened. What interviewer is testing: whether you treat ACK loss as decoupled from actual effect success, rather than assuming one implies the other. Read deeper: ACK Lost After Successful Execution.

• Is this the same scenario as the classic payment example? — Yes — this is exactly the case where a lease and a scheduler-level lock reduce concurrent execution but the downstream idempotency key is what actually prevents a second charge.

Strong answer direction: it depends on where the effect lives. A local DB mutation: unique constraint plus one atomic transaction covering both the mutation and its completion marker. An external API with idempotency-key support: pass a stable key derived from the business operation, and let the provider deduplicate. An external API without one: durable operation state plus reconciliation against the provider's actual record, accepting that a fully exactly-once guarantee may not be achievable — not distributed transactions as a default fix. What interviewer is testing: whether you can name the specific mechanism per boundary rather than one blanket answer. Read deeper: Idempotency and External Side Effects.

• Why isn't a Redis lock on the job enough by itself? — The lock only protects scheduling-level coordination — it knows nothing about whether the external side effect actually completed before a crash.

Strong answer direction: if the worker inserts the row and then crashes before performing the actual side effect, a retry sees the row, assumes the work is done, and skips it — the job is now silently lost, which is worse than the duplicate this was meant to prevent. The fix is modeling state with enough granularity (pending / effect-identity / completed) plus reconciliation, not a single flag written ahead of the real effect. What interviewer is testing: recognition of this specific, subtle correctness bug, which is one of the most common mistakes in scheduler idempotency design. Read deeper: Duplicate Execution.

• What's the correct order for a local DB effect? — The mutation and its completion marker commit together, atomically, in the same transaction — there's no window where one happened without the other.

Strong answer direction: separate the schedule definition (recurrence rule, timezone, next-fire-time, misfire/overlap policy) from each individual execution occurrence, and derive a deterministic occurrence identity as f(schedule_id, nominal_fire_time) — an idempotent insert on that id means a scheduler retry or failover replaying the same generation step is a harmless no-op instead of a duplicate. What interviewer is testing: whether you understand recurring jobs as a stream of occurrences generated from a template, not literally 'the same job re-queued.' Read deeper: Recurring Schedule Engine.

• What if the previous occurrence is still running when the next comes due? — That's the separate overlap policy question — allow, skip, or queue the next occurrence, chosen per schedule based on whether concurrent runs are safe.

Strong answer direction: apply an explicit, per-schedule misfire policy — run the latest missed occurrence, skip all missed occurrences, or coalesce them into one catch-up run — rather than automatically firing every missed occurrence, which can both be semantically wrong (a daily reminder firing three times) and recreate a self-inflicted burst. What interviewer is testing: whether you treat 'catch up' as a policy decision rather than an automatic default. Read deeper: Misfire and Catch-Up.

• Does this policy need to be configurable per schedule, or can it be system-wide? — Per-schedule is more correct — a billing run and a 'good morning' notification have very different answers to 'is late output still valuable?'

Strong answer direction: the recurrence rule together with an explicit IANA timezone — never a fixed UTC offset computed once at creation time, because that offset itself changes across DST transitions. Each occurrence's absolute instant is computed fresh, against the real timezone rules, at generation time. What interviewer is testing: whether you know that 'convert to UTC once and store that' is a subtle correctness bug for recurring wall-clock schedules specifically (one-time absolute jobs are fine to normalize to UTC immediately). Read deeper: Time Zones, DST and Clock Semantics.

• What about the DST spring-forward gap, where the local time doesn't exist that day? — Needs an explicit product policy — e.g. fire at the nearest valid time after the gap — rather than letting a time library's default behavior decide silently.
• And the fall-back case where the local time occurs twice? — Also needs an explicit policy — e.g. fire only on the first occurrence — stated up front rather than discovered in production.

Strong answer direction: a SCHEDULED (not yet dispatched) job can usually be cancelled cleanly and immediately. A QUEUED job is race-prone — the cancel and the worker picking up the message can interleave, so the worker must re-check current job state/version before executing and discard a stale message for an already-cancelled job. A RUNNING job may support cooperative cancellation (the worker periodically checks for a cancellation signal at safe points) but can't be forcibly halted mid-instruction, and any external side effect already performed may not be reversible. What interviewer is testing: whether 'running jobs cannot be cancelled' gets replaced with the more precise, staged answer. Read deeper: Cancellation and Rescheduling.

• How do you stop a stale queued message from executing after cancellation? — Version-check discipline — the dispatch message carries the version at claim time, and the worker rejects it if the current state/version has since changed.

Strong answer direction: no, not necessarily — competing DB claimers (SKIP LOCKED), sharded ownership, time-bucket ownership, and broker-native delayed delivery are all valid alternative coordination models. Leader election is one useful pattern, not a mandatory ingredient of every scalable scheduler. What interviewer is testing: whether you present leader election as a default reflex or as one deliberate choice among several. Read deeper: Single Leader vs Competing Claimers vs Sharded Ownership.

• If you do use leader election, what's the biggest gap in a naive implementation? — A bare TTL-based lock without fencing — a paused leader can resume after losing its lease and act again, which the TTL alone doesn't prevent.

Strong answer direction: a monotonically increasing epoch issued on every leadership change, attached to every write the leader makes downstream. Consumers reject any write tagged with an epoch older than the highest one already seen — this is what makes a stale, resumed leader's actions harmless rather than merely rare, since the lease TTL alone only bounds detection time, not whether the stale leader can still act during that window. What interviewer is testing: precise understanding of why TTL alone is insufficient for split-brain safety. Read deeper: Leader Election, Leases and Fencing.

• Is fencing always available? — Not for every downstream system — where it isn't, idempotent dispatch consumers are the fallback that makes a stale leader's actions safe.

Strong answer direction: no — they differ meaningfully in redelivery model (consumer offsets vs. visibility timeout vs. explicit ack), ordering guarantees, retention/replay capability, native delayed-delivery support, and poison-message handling. Treating them as interchangeable 'a queue' glosses over real design consequences — e.g. SQS gives you visibility-timeout redelivery and DLQ redrive natively, while Kafka gives you replay and partition ordering but expects you to build retry/redelivery semantics yourself. What interviewer is testing: whether you understand the specific mechanics of whichever queue you propose, not just that 'a queue' sits between dispatch and execution. Read deeper: Queue Semantics.

• Which would you pick for this design, and why? — Depends on whether replay/ordering (Kafka) or native visibility-timeout redelivery and DLQ redrive (SQS) matters more for the specific failure modes you're prioritizing.

Strong answer direction: classify failures — transient retries with standard capped backoff and jitter; throttling retries but honors any Retry-After guidance; permanent (validation, auth, business-rule) skips retry entirely and routes to DLQ; ambiguous retries only because the underlying operation is idempotent. To prevent a retry storm, add a circuit breaker and a shared retry budget scoped to the failing dependency, not just per-job backoff. What interviewer is testing: whether retry logic is failure-aware, and whether you understand retry storms as a distinct problem from single-job retry correctness. Read deeper: Retry Scheduler.

• How do you safely replay a large DLQ backlog after the dependency recovers? — Rate-limited, authorized, audited, prioritized, and idempotent replay — never an unbounded bulk replay, which can re-trigger the same outage it followed.

Strong answer direction: first ask what lateness is acceptable — the system doesn't need to execute all 50M within seconds just because they share a due_at. Then: bounded batch retrieval and partitioned due-time discovery avoid one query trying to return everything at once; rate-limited dispatch caps throughput to what downstream systems can sustain; jitter (only for job classes whose semantics allow it) spreads load at creation time; autoscaling reacts to a known, predictable burst ahead of time. What interviewer is testing: whether you name the actual bottleneck-specific mitigations rather than asserting a single universal fix (e.g. 'just use Redis'). Read deeper: Thundering Herd.

• Is it safe to start dispatching at 8:59 to spread the load? — Only if the queue or worker enforces eligible_at and never executes early — otherwise this silently breaks the scheduling contract rather than fixing the burst.

Strong answer direction: rate-limited dispatch caps outbound throughput to that dependency's actual sustained capacity, accepting that the excess backlog simply takes longer to drain within the accepted lateness SLO — a queue absorbs a temporary spike, but it cannot manufacture downstream capacity that doesn't exist, so a sustained mismatch needs either real added capacity or an explicit, accepted backlog-drain time. What interviewer is testing: whether backpressure is understood as fundamentally limited by the true bottleneck, not just something a bigger queue fixes. Read deeper: Backpressure.

• How would you detect this is happening before it becomes a multi-hour backlog? — Track dispatch lag, queue depth, oldest-job age, and explicit downstream-throttling signals — not queue depth alone.

Strong answer direction: start from a single home-region (or home-shard) authority per job with durably replicated state to a standby region — this reuses the same leader-election-plus-fencing mechanism already built for single-region failover, applied at region granularity. Active-active dispatch (multiple regions independently deciding a job is due) is a substantially harder problem requiring global fencing and cross-region consensus on due-time ownership, and should be introduced only once the simpler baseline has a proven, specific limitation. What interviewer is testing: whether you reach for the simplest baseline first rather than jumping to active-active as a default 'more available' answer. Read deeper: Multi-Region Scheduling.

• What happens to jobs due during the actual failover window? — Handled by the same misfire/catch-up mechanism used for any scheduler downtime, bounded by the design's stated RTO.

Strong answer direction: reject new job-creation requests explicitly (not silently) while the primary is down; workers can often drain jobs already dispatched before the outage; on promotion of a replica, RPO depends on the actual replication mode (near-zero for properly acknowledged synchronous replication, non-zero and workload-dependent for asynchronous) — avoid claiming a fixed 'typically under a second' figure as a universal guarantee. What interviewer is testing: precise handling of a core infrastructure failure, and honesty about what replication mode actually guarantees. Read deeper: Database Primary Failure.

• Would you buffer job creation requests locally during the outage? — Generally no, unless that buffer is itself durable and reconciled — an in-memory buffer just relocates the same failure window rather than closing it.

Strong answer direction: never assume all nodes agree on 'now' to sub-second precision — build a small safety margin into any time-sensitive comparison (lease expiry, misfire detection), rely on NTP for a reasonable baseline, and where a single authoritative clock matters more than cross-node agreement, prefer a single server's or the database's own clock for that specific decision rather than comparing independently-observed wall clocks. What interviewer is testing: whether you treat 'perfectly synchronized clocks' as a false assumption to design around, not a given. Read deeper: Clock Skew.

• Does this affect leader election too? — Yes — lease expiry detection in leader election has exactly the same skew-tolerance requirement as lease expiry detection for worker attempts.