Job Processing

Background job processing runs a unit of work outside the request or user interaction that asked for it. The work might be submitted through an API, triggered by an event, or fired by a schedule. A job system is more than a queue: it also records what the job is supposed to do, which worker currently owns it, how progress and failure are tracked, and what happens when that worker disappears halfway through.

Typical jobs are generating a PDF report, reconciling a payment, importing a CSV file, rebuilding a search index, or sending a scheduled reminder. They differ in how much a repeat costs. A report can usually be recomputed. A payment settlement cannot be repeated casually and needs idempotency and reconciliation. So the lifecycle you design has to reflect the business operation, not whatever status enum a library ships with.

The mental model: a job is a durable row with an owner and a deadline. The row says what must happen and what has happened so far. The owner is a worker holding a time-limited lease, so ownership is something that can be taken away when the worker stops responding. The deadline is the lease expiry: the point where the system decides the owner is gone and somebody else may try. Almost every design question in this lesson is a question about those three things: what the record says, who may change it, and what happens when time runs out.

This lesson owns the lifecycle above the transport. How a message physically travels from producer to consumer, what at-least-once delivery means, and how acknowledgements and visibility timeouts behave belong to Queues & Workers. What a failed job becomes after retries are exhausted, and how it is replayed, is covered in Dead-Letter Queues.

A user uploads a video. Transcoding to two resolutions, a thumbnail, a virus scan, and a CDN upload take 35, 45, 8, 15, and 20 seconds in this illustration: 123 seconds of sequential work. Done inside the request, the user watches a spinner for two minutes, a proxy timeout may cut the connection first, and one slow step fails the whole request. Done as a job, the API validates the upload, stores a job record, and answers in a few hundred milliseconds that the work was accepted. What the caller needs is an acknowledgement and a way to find the result later, not every side effect finished.

The test for each step is whether the caller needs its outcome to decide what to do next. In an order flow, charging the payment, reserving inventory, and writing the order row are synchronous because the response is "your order is confirmed" or "it failed". The confirmation email, the invoice PDF, the warehouse notification, the analytics event, and the loyalty points can happen seconds later without changing what the customer sees. Payment authorization is usually kept in the request path for exactly this reason, with a tight timeout. Authentication and search results are the same: the next screen depends on them.

The accepted-then-poll contract

An asynchronous API has to say what "accepted" means. The conventional shape is 202 Accepted with a Location header pointing at a job resource, which the client polls or which pushes completion over a webhook or a stream. The job resource reports a state, not a guess: queued, running, succeeded with a link to the result, or failed with an error code the client can act on. See REST and API Design for resource and status-code conventions, and Pagination if clients list their jobs.

Creating the job atomically with the business change

The order row and the jobs it spawns should either both exist or neither. When the job table lives in the same PostgreSQL database as the orders, one transaction can insert both, and no job is created for an order that rolled back or lost for one that committed. That atomicity is a property of sharing a database. If the jobs are published to a separate broker such as SQS, RabbitMQ, or Kafka, no transaction spans the commit and the publish, and the two can disagree after a crash between them. The Transactional Outbox exists for that case: write the intent to a table in the business transaction, and have a relay publish it afterwards.

A minimal lifecycle is QUEUED → RUNNING → SUCCEEDED, with failure branches RUNNING → RETRY_WAIT (retried later) and RUNNING → FAILED (given up). Real systems add CANCEL_REQUESTED, CANCELLED, and sometimes TIMED_OUT. Each state needs a documented meaning and a list of legal transitions, because the transitions are what the database will enforce and what operators will rely on at 3 a.m. The one definition worth being strict about is RUNNING: it should mean a worker currently holds a valid claim, not merely that some worker set a flag hours ago and may have died since.

Read the figure by what moves a job. A claim moves it into RUNNING and raises the lease generation. A worker's outcome report (success, error, cancellation acknowledged) moves it out. A sweeper moves it out when the lease expires without any report. A cancel request from the outside moves it only to CANCEL_REQUESTED if it is running, because the worker has to stop first. Everything that leaves RUNNING is conditional on the lease the writer holds; the next sections make that precise.

A durable job record

The schema below is illustrative, a starting point for PostgreSQL rather than a drop-in table. It adds a check constraint so only the listed states exist, and another so a lease only exists while the job is running. Constraints like these are cheap and turn a bad code path into an error instead of a silently corrupt row.

CREATE TABLE jobs (
  job_id            uuid        PRIMARY KEY,
  job_type          text        NOT NULL,
  state             text        NOT NULL DEFAULT 'QUEUED'
    CHECK (state IN ('QUEUED','RUNNING','RETRY_WAIT','SUCCEEDED',
                     'FAILED','CANCEL_REQUESTED','CANCELLED')),
  attempt_count     integer     NOT NULL DEFAULT 0,
  max_attempts      integer     NOT NULL DEFAULT 5,
  run_after         timestamptz NOT NULL DEFAULT now(),
  lease_owner       text,
  lease_expires_at  timestamptz,
  lease_generation  bigint      NOT NULL DEFAULT 0,
  payload_ref       text        NOT NULL,
  last_error_code   text,
  created_at        timestamptz NOT NULL DEFAULT now(),
  updated_at        timestamptz NOT NULL DEFAULT now(),
  CHECK ((state IN ('RUNNING','CANCEL_REQUESTED'))
         = (lease_owner IS NOT NULL AND lease_expires_at IS NOT NULL))
);

-- Keep the claim query cheap: index only the rows a worker could take.
CREATE INDEX jobs_claimable_idx ON jobs (run_after, created_at)
  WHERE state IN ('QUEUED','RETRY_WAIT');

Three details deserve a decision rather than a default. First, payload_ref points at the input instead of embedding it, so a large or sensitive payload is stored once under its own access rules and the hot table stays narrow. Second, attempt_count must mean one specific thing. Incrementing at claim time counts attempts that started, including ones that crashed; incrementing on reported failure counts only attempts that ended in an error. This lesson increments at claim, because a worker that dies mid-job still consumed an attempt and a job that crashes every worker must eventually stop. If reporting needs the other number, store it separately. Third, last_error_code keeps the reason for the previous failure on the row, so a job that exhausted its attempts remains diagnosable after the transport has forgotten it.

The lease idea is not specific to a job table. Every queueing system has a way to say "this one consumer has the work for now", and a way to take it back. The differences matter because they decide what a stale worker can still do, and the vocabulary below is what to look for in whichever system you run.

• PostgreSQL job table (this lesson): the lease is columns on the row. You own the state machine, the claim query, and the fencing predicate, and you can query everything. You also own the cost: polling load, bloat from frequent updates, and no built-in fan-out.
• Amazon SQS: the lease is the visibility timeout. A received message is hidden for that period (30 seconds by default, up to 12 hours) and reappears if it is not deleted. A consumer can extend the timeout for a message it still holds, and the receive count lets a redrive policy move repeat offenders to a dead-letter queue. A consumer whose timeout expired may still be able to delete the message until someone else receives it again, and there is no generation number you control, so it is a lease without a fencing token.
• RabbitMQ: a delivered but unacknowledged message belongs to its channel. If the channel or connection closes, the broker requeues it. A delivery acknowledgement timeout (consumer_timeout, 30 minutes by default in current versions) closes the channel of a consumer that holds a delivery too long. Prefetch bounds how many unacknowledged messages one consumer may hold at once.
• Kafka: ownership is of a partition, not a message. A consumer group assigns each partition to one member, and a member that stops polling within max.poll.interval.ms or stops sending heartbeats within the session timeout is removed, which triggers a rebalance. Offset commits from a member of an earlier group generation are rejected, a built-in form of fencing for the offset. There are no per-message retries or delays: one slow or failing record holds up everything behind it in its partition, which is why job-style retry and scheduling are usually built with a table or retry topics beside Kafka.

Many production systems combine them: a broker moves job IDs quickly, while a durable job table remains the source of truth for state, attempts, and ownership. Then the broker's redelivery is only a wake-up call, and the table's conditional update decides whether a worker may proceed. The delivery semantics of each broker, and when messages repeat, are worked through in Queues & Workers.

The naive claim is a SELECT for a queued job followed by an UPDATE that marks it running. Between the two statements another worker can run the same SELECT and see the same row as QUEUED. Both then update it and both start the work. Wrapping the pair in a transaction does not fix this by itself under the default READ COMMITTED isolation, because plain reads take no locks.

PostgreSQL offers SELECT … FOR UPDATE SKIP LOCKED. The selecting statement locks the rows it returns, and any other transaction running the same statement skips locked rows instead of waiting for them. Combined with an UPDATE in one statement, each worker gets a different job, and nobody blocks behind a busy claimant. It is available from PostgreSQL 9.5.

-- :worker_id is a bound parameter (your driver's placeholder syntax applies).
WITH picked AS (
  SELECT job_id
  FROM jobs
  WHERE state IN ('QUEUED','RETRY_WAIT')
    AND run_after <= now()
  ORDER BY run_after, created_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
UPDATE jobs AS j
SET state            = 'RUNNING',
    lease_owner      = :worker_id,
    lease_expires_at = now() + interval '60 seconds',
    lease_generation = j.lease_generation + 1,
    attempt_count    = j.attempt_count + 1,
    updated_at       = now()
FROM picked
WHERE j.job_id = picked.job_id
RETURNING j.job_id, j.lease_generation, j.lease_expires_at, j.payload_ref;

The sequence shows the point of the clause. Worker 1 locks job 41 and has not yet committed. Worker 2 arrives, sees 41 locked, skips it, and takes job 42. Without SKIP LOCKED Worker 2 would wait for Worker 1's transaction to finish and then find 41 already running. Once both commit, a later claim no longer selects either row, because they are RUNNING and the claim only looks at claimable states. Every timestamp, including the lease expiry, comes from the database clock through now(), so workers with skewed clocks cannot disagree about when a lease ends.

Know what the clause does not give you. It is not fair or strictly ordered: skipped rows are simply passed over and a job being held by a slow claimant can be overtaken. PostgreSQL's documentation describes the result as an inconsistent view of the table, which is acceptable for a queue and wrong for most other uses. It also does nothing about a worker that crashes after the commit. After the commit the row lock is gone, and the persisted lease is the only record of ownership. That is deliberate. Keep the claim transaction short and never hold a row lock, or a connection, across minutes of external processing: a long transaction pins a pooled connection, delays vacuum cleanup, and still would not survive the worker's own crash in a useful way.

To claim several jobs at once raise LIMIT and process the returned rows, which cuts round trips under load. Keep the batch small relative to lease length: every claimed job's lease is already running while you work through the earlier ones.

A lease has an expiry. Suppose Worker A claims a job for 60 seconds and then freezes for 90: a long stop-the-world garbage collection, a virtual machine paused for migration, a laptop lid closing, a container starved of CPU. The lease expires, the sweeper requeues the job, and Worker B claims it. Then A wakes up with no idea any time has passed and carries on. Both workers now believe they should finish the job. Checking the clock before the write does not close the gap, because the pause can happen between the check and the write. A lease on its own cannot stop A from performing a stale write after expiry.

A fencing token fixes this. It is a number that increases on every claim, here lease_generation. The worker receives its generation from the claim, sends it with every write to shared state, and the store rejects any write carrying an older number. The check has to be done by the thing being protected, in the same atomic operation as the write; a worker comparing numbers in its own memory proves nothing.

UPDATE jobs
SET state            = 'SUCCEEDED',
    lease_owner      = NULL,
    lease_expires_at = NULL,
    updated_at       = now()
WHERE job_id           = :job_id
  AND state IN ('RUNNING','CANCEL_REQUESTED')
  AND lease_owner      = :worker_id
  AND lease_generation = :generation
  AND lease_expires_at > now();

The timeline is worth reading line by line. At 66 seconds the sweeper sees a RUNNING row whose lease has ended and requeues it; at 68 seconds Worker B claims it and the generation becomes 13. At 95 seconds Worker A wakes and runs the completion update with generation 12. At that moment the row belongs to B and B's lease is still valid until 128 seconds, so the lease_expires_at > now() condition would not reject A. What rejects A is lease_owner = 'A' failing, and more robustly lease_generation = 12 failing. The generation check is the one that survives a worker reclaiming its own expired job: there the owner name matches, only the number differs, and a stale thread of the same process is still rejected.

The expiry predicate is a design choice, not a safety requirement. If nobody has claimed the job, a worker that finishes slightly late still holds the current generation, and rejecting it throws away finished work for a rule that protects nothing. Some systems drop the predicate and accept any completion that matches owner and generation; the version above is the stricter one, and either is safe because generation changes on every claim.

What fencing does not cover

The token fences the job-state update. It does not reach back and cancel something Worker A already did. If A, before freezing, sent a charge to a payment provider, that call is gone. If A wakes and sends the charge after B already did, the provider has no idea a job table exists and will process both unless the request carries something the provider itself enforces: an idempotency key derived from the job, or a version or fencing token on the protected resource. Fencing in the general sense, as Kleppmann describes for locks, only works when the resource checks the token. Do not claim a fenced job record means the side effects are safe. For the TTL-and-ownership idea in a general-purpose lock service, see Distributed Locks.

A job that can run longer than a comfortable lease should renew it. The worker, or a thread beside it, sends a heartbeat: an update that pushes lease_expires_at forward. The heartbeat has the same fence as the completion. Without the generation check, a stale worker's heartbeat could revive an expired claim after another worker already took it.

UPDATE jobs
SET lease_expires_at = now() + interval '60 seconds',
    updated_at       = now()
WHERE job_id           = :job_id
  AND state IN ('RUNNING','CANCEL_REQUESTED')
  AND lease_owner      = :worker_id
  AND lease_generation = :generation
  AND lease_expires_at > now()
RETURNING state;

Choosing the numbers is a trade between detection speed and write load. After a crash the job sits owned by a dead worker until the lease runs out, so a shorter lease recovers faster. But the interval must leave room for a delayed network call or a busy database, or healthy workers lose their jobs. A common rule is a heartbeat interval of about a third of the lease, which tolerates one lost heartbeat. Note what the lease is not: it is not sized to the job's duration. A lease of twice the P99 runtime without heartbeats would make crash recovery as slow as the longest job. With heartbeats, the lease only needs to cover the heartbeat delay.

Try a lease shorter than the heartbeat interval: the lease lapses between renewals and a perfectly healthy worker loses its job, which is a bug that only appears under load. Every heartbeat is also a database write, so 1,000 concurrent jobs at one heartbeat every 20 seconds is 50 writes per second just to stay alive. Frequent updates to the same row also create dead row versions. If you index lease_expires_at for the sweeper, each heartbeat touches an indexed column and PostgreSQL can no longer apply its cheaper in-page (HOT) update. When only a small share of rows are RUNNING, scanning them without that index is often the better trade.

The sweeper

Something has to notice leases that expired without a report. A sweeper runs periodically and returns those jobs to a claimable state, subject to the attempt limit. This is the only place a lease turns into a retry, so it must not act without checking attempts: a job that crashes its worker every time would otherwise loop forever.

UPDATE jobs
SET state = CASE
      WHEN state = 'CANCEL_REQUESTED'     THEN 'CANCELLED'
      WHEN attempt_count >= max_attempts  THEN 'FAILED'
      ELSE 'RETRY_WAIT'
    END,
    run_after        = now(),
    lease_owner      = NULL,
    lease_expires_at = NULL,
    last_error_code  = 'LEASE_EXPIRED',
    updated_at       = now()
WHERE state IN ('RUNNING','CANCEL_REQUESTED')
  AND lease_expires_at <= now()
RETURNING job_id, state, attempt_count;

Whether an expired lease should be retried automatically depends on the job. If the job is safe to repeat, as a report is, requeueing is right. If the worker may have already performed an external side effect, the sweeper cannot know. Mark such jobs for reconciliation, or let the handler start with a check of what the previous attempt did, before the job runs again.

Five timeouts, five different questions

Systems routinely use one word, "timeout", for separate mechanisms. They should be configured and monitored separately, and each one is answered by a different part of the design.

One trap lives between the last three. A job that exceeds its execution timeout may still have an outstanding request at a remote system. Cancelling locally is not proof the remote operation did not commit, so a timed-out job should be treated as an ambiguous failure, not a clean one. The next section handles that case.

The first question when an attempt fails is whether trying again can change the outcome. Network timeouts, temporary rate limits, and short dependency outages are candidates for bounded retries with exponential backoff and jitter. Invalid input, an unsupported schema version, or a missing authorization grant need a correction, and repeating them immediately only adds load. The third category is the dangerous one: some errors are ambiguous. A payment call that times out may mean the provider charged the card and the response was lost. The right action is not a blind retry but a lookup by operation ID to learn what happened.

The flow ends in three places, each with a different meaning for operators. FAILED with diagnostics means a human or a code change is needed. RETRY_WAIT means the system expects to recover by itself. SUCCEEDED after reconciliation means the earlier attempt had in fact completed, so repeating it would have duplicated the effect. A final failure must remain visible on the job record with an actionable error code and the attempt history, not disappear because the transport message was acknowledged and deleted. Jobs that exhaust their attempts are the input of Dead-Letter Queues, which cover quarantine and replay.

Backoff for jobs

For attempt n, a common schedule is delay = min(maxDelay, baseDelay × 2^(n−1)) followed by jitter, where n is the number of attempts already made (so the first retry uses n = 1). State which convention you use; off-by-one in the exponent changes every delay by a factor of two. With a 5 second base and a 15 minute cap the pre-jitter delays are 5, 10, 20, 40, 80, 160, 320, 640 seconds, then 900. Do not let thousands of failed jobs retry in the same second after a shared dependency recovers. Randomising each delay spreads them out. The derivation of the schedule and the jitter variants is in Retry & Backoff; what a job adds is that the result is stored, as run_after, instead of held in memory:

-- :delay_seconds is computed in the worker (capped exponential plus jitter).
UPDATE jobs
SET state = CASE WHEN attempt_count >= max_attempts
                 THEN 'FAILED' ELSE 'RETRY_WAIT' END,
    run_after        = now() + make_interval(secs => :delay_seconds),
    lease_owner      = NULL,
    lease_expires_at = NULL,
    last_error_code  = :error_code,
    updated_at       = now()
WHERE job_id           = :job_id
  AND state            = 'RUNNING'
  AND lease_owner      = :worker_id
  AND lease_generation = :generation;

Two failure shapes from production show up repeatedly. An endless retry loop happens when there is no attempt ceiling: a job that fails every time stays in the system forever and its alerts never fire, so max_attempts belongs in the schema, not in a code comment. A poison job is one whose payload makes the handler fail deterministically, or worse, crashes the worker. In a job table it only consumes its own attempts, because other rows are still claimable. On a strictly ordered stream such as a single Kafka partition or one SQS FIFO message group, the same job blocks everything behind it, which is why ordered consumption needs an explicit skip or quarantine rule.

Everything above makes sure a job is not lost. None of it makes the side effects happen once. A worker that sends an email and then crashes before reporting success leaves a job that will be retried and a customer who receives two emails. Lease expiry, a visibility timeout, a connection drop, and a deploy that kills workers mid-job all produce the same outcome. A system can retry reliably and still produce a wrong result when the handler is not idempotent. This is a statement about the handler, so the job system cannot fix it for you.

Techniques, roughly from strongest to weakest:

• Make the effect itself idempotent. Write the result under a key derived from the job: reports/{job_id}/report.csv overwritten on each attempt, an INSERT … ON CONFLICT DO NOTHING on a natural key, or a conditional update ("set status to shipped where status is packed").
• Pass an idempotency key to the remote system, derived from the job and the specific operation inside it, stable across attempts. The provider then returns the original result for a repeat. Details in Idempotency.
• Record the effect in the same transaction as the job update when both live in one database, so success and its evidence commit together.
• Reconcile before retrying when the remote system supports lookup by operation ID.

A common wrong fix is to check "was this already sent?" at the start of the handler. The check and the send are two steps, so two overlapping attempts can both see "no" and both send. The check works only when the claim of the effect is atomic, for example inserting a row with a unique key and sending only if the insert succeeded, and even then a crash between the insert and the send needs a recovery rule.

Report progress only when it means something. A bar at 65% is misleading if the remaining step can take 30 minutes. Stage-based status (UPLOADING, VALIDATING, GENERATING, PUBLISHING) is honest when step lengths are uneven. A processed-over-total counter is good when the denominator is stable, as with rows in an import. A status endpoint should also separate a temporarily lost worker from a permanent failure: "running, last heartbeat 12 seconds ago" and "waiting to retry at 14:05 after a timeout" are very different from "failed", and clients that treat them alike either give up too early or poll forever. Once a job succeeds, the status resource should carry the result location, or a stable link the client can fetch from.

Cancellation is cooperative in most real systems. The requester sets the job to a request-to-stop state, the worker notices at a safe checkpoint, stops, records any partial effects, and only then does the job become CANCELLED. Killing the process instead just triggers the crash-recovery path and may leave partial work with no record. A cancelled email job cannot retract a message already delivered, and a cancelled transfer might need compensation. Decide per job type whether retry, cancel, and manual replay are allowed, and what "cancelled" promises.

-- Requester: an unclaimed job is cancelled outright; a running job is asked to stop.
UPDATE jobs
SET state = CASE state WHEN 'RUNNING' THEN 'CANCEL_REQUESTED' ELSE 'CANCELLED' END,
    updated_at = now()
WHERE job_id = :job_id
  AND state IN ('QUEUED','RETRY_WAIT','RUNNING')
RETURNING state;

-- Worker, at a checkpoint after it has stopped and recorded partial effects:
UPDATE jobs
SET state = 'CANCELLED', lease_owner = NULL, lease_expires_at = NULL, updated_at = now()
WHERE job_id = :job_id AND state = 'CANCEL_REQUESTED'
  AND lease_owner = :worker_id AND lease_generation = :generation;

The worker-side half is a loop that checks at points where stopping is safe. Pseudocode, not runnable:

for batch in batches(job.input):
    process(batch)                       # idempotent per batch
    state = heartbeat(job, generation)   # returns None if the lease was lost
    if state is None:
        return LOST_LEASE                # stop; do not report anything
    if state == "CANCEL_REQUESTED":
        record_partial_effects(job)
        return acknowledge_cancel(job, generation)
complete(job, generation)

A recurring job needs a stable occurrence ID that identifies one specific firing: the schedule plus the instant it was meant to run. Without it, two scheduler instances that both tick at 02:00, or a leader that fails over just after firing, each create a job and the business operation runs twice. With it, creating the job is a uniqueness problem the database can solve, not a coordination problem between schedulers.

The same mechanism covers submissions from users and APIs. When several requests refer to the same logical operation, a business key is a better identity than a random message ID. Monthly invoice generation can use (account_id, billing_month). Two submissions with the same key collapse into one job.

ALTER TABLE jobs ADD COLUMN dedup_key text;
CREATE UNIQUE INDEX jobs_dedup_idx ON jobs (job_type, dedup_key)
  WHERE dedup_key IS NOT NULL;

-- A scheduler firing the 2026-03-01 02:00 UTC occurrence. A second scheduler
-- running the same statement inserts nothing.
INSERT INTO jobs (job_id, job_type, payload_ref, dedup_key)
VALUES (gen_random_uuid(), 'daily_settlement', :payload_ref,
        'daily_settlement@2026-03-01T02:00:00Z')
ON CONFLICT (job_type, dedup_key) WHERE dedup_key IS NOT NULL
DO NOTHING
RETURNING job_id;

Three scheduling policies need deciding up front. Missed occurrences: if the scheduler was down from 01:00 to 04:00, should the 02:00 run happen late (catch up), be skipped, or be merged with the next one (coalesce)? An hourly metrics rollup might catch up each hour; a "send daily digest" should probably send once. Time zones: "every day at 02:00" and "every 24 hours" differ. On the day clocks go forward, local 02:00 may not exist, and on the day they go back, 1:30 occurs twice, so a calendar schedule needs an explicit rule for both. Separation: keep the decision of when a job exists apart from the execution of the job, so a slow worker pool never delays the scheduler and a scheduler bug never executes work directly.

Workers scale separately from the API, which is a large part of why jobs exist. Capacity starts as arithmetic: the average number of jobs in progress equals the arrival rate times the average duration (Little's law). If 50 jobs arrive per second and each takes 2 seconds, about 100 run at once, so you need at least that many worker slots plus headroom for bursts. Scaling is not guaranteed linear. The job table, the database, or a rate-limited downstream becomes the new ceiling as workers are added, so find the bottleneck before promising "more workers, more throughput".

The scaling signal should be how long the oldest runnable job has waited, not only how many jobs are queued. Depth alone misleads. 10,000 tiny jobs may clear in seconds, while 50 two-hour jobs behind a full worker pool are a real incident. Alert on the age of the oldest runnable job against a latency target, and treat depth as supporting detail.

Isolation is the other half. Run separate worker pools per job type: CPU-heavy video work and light email sending should not compete for the same slots, and the Bulkhead lesson covers the principle. Within a pool, cap concurrent jobs per tenant so one customer's backlog cannot starve another's. Without such a cap a single import of 200,000 rows by one tenant occupies every slot while everyone else waits.

A producer bug that enqueues millions of jobs in seconds is the usual queue explosion. Bound it where jobs are created: apply rate limiting to enqueue, and apply backpressure so producers learn that the system is full instead of silently piling up work. When demand exceeds what can ever be processed, shedding low-priority work early (load shedding) beats a backlog that grows until every job is late.

An API receives a request for a quarterly report. It creates a job with a stable ID and a reference to an input snapshot, so a rerun reads the same data, and replies 202 Accepted with Location: /reports/jobs/{job_id}. A worker claims the job and writes output to a temporary object under a job-specific key. It validates totals against the snapshot, then publishes the artifact pointer in the same transaction that completes the job, carrying its lease generation:

BEGIN;

UPDATE jobs
SET state = 'SUCCEEDED', lease_owner = NULL, lease_expires_at = NULL, updated_at = now()
WHERE job_id = :job_id AND state = 'RUNNING'
  AND lease_owner = :worker_id AND lease_generation = :generation;
-- Application: if this changed 0 rows, ROLLBACK and stop.

UPDATE reports
SET artifact_uri = :final_uri, published_at = now()
WHERE report_id = :report_id;

COMMIT;

Suppose the worker crashes after uploading part of the file and before publishing. The row stays RUNNING until the last heartbeat plus the lease length passes; the sweeper then requeues it, and the next worker overwrites the deterministic artifact key and publishes with the next generation. Nothing the first attempt wrote is visible to users because the pointer was never updated, and an orphan cleanup job deletes unused temporary objects after a retention window. The timeline also shows why the user-facing number is the 240 seconds from submission, not the 90 seconds the successful attempt ran: queue wait and recovery time are part of the experience.

What to watch on a dashboard for this system: queue wait and execution time at p50, p95, and p99 (separately); attempts per successful job; the number of expired leases; jobs stuck in RUNNING; scheduler delay; cancellation latency; and the age of the oldest runnable job. Measure end to end from the user's submission, because the last attempt's runtime hides everything before it.

The mechanism in this lesson promises specific, limited things. At most one worker holds a given generation of a job, because the claim is atomic. A stale worker cannot change the job's recorded state, because every write carries and is checked against its generation. A job whose worker disappears is eventually reclaimed, because the lease expires and the sweeper acts on it. A job that keeps failing eventually stops, because attempts are bounded and the last error remains on the row.

It does not promise that the work runs exactly once, that external side effects performed by a stale worker are undone, that jobs run in submission order (claim order is by run_after and creation time, and retries and skipped rows reorder it), or that the result is correct. Execution is at least once with fenced bookkeeping. Correctness of the outcome comes from idempotent handlers and reconciliation, never from the lease. Ordering that matters has to be designed explicitly, for example one running job per entity key.

For a single independent task, a queue plus a durable job table is usually enough. The task has one outcome, and the retry and lease rules above cover its failures. A workflow is different: many dependent steps, conditional branches, approvals that take days, long-lived timers, and compensation when a later step fails. Encoding that as one job that enqueues the next job, which enqueues the next, spreads the business process across handlers and makes its state invisible. Use an explicit workflow state machine or an orchestration engine that persists the workflow's history, and let jobs be the individual steps.

When a multi-step process must undo earlier steps if a later one fails, the Saga Pattern describes compensating transactions. Choose jobs for independent units of deferred work, and a workflow engine when the relationships between the steps are themselves the business logic.

The transport underneath jobs, with acknowledgements, visibility, and at-least-once delivery compared across brokers, is in Queues & Workers. Jobs that run out of attempts end up in Dead-Letter Queues, and getting a job into the system reliably alongside a database change is the job of the Transactional Outbox. When one event should start several different kinds of work, Pub/Sub fans it out. For the retry mathematics see Retry & Backoff, and for the idempotency techniques that make repeats safe, Idempotency.