Queues & Workers

A message queue holds work that a producer has submitted and no consumer has finished. A worker is a process that receives or claims a message and does the work outside the request that created it. Together they decouple when work is accepted from when it is executed, which is useful whenever the work is slow, bursty, failure-prone, or simply not needed before the client gets a response.

Take an image-upload API. Receiving the file, validating it, and storing the original takes a few hundred milliseconds. Thumbnails, moderation, metadata extraction, and downstream notifications can take seconds or minutes. If the HTTP handler performs all of it, the endpoint inherits the latency and the availability of every dependency it touches. Instead the handler stores the original, records a job, enqueues a message that points at it, and returns 202 Accepted with a job ID. Workers do the slow part later, at a pace the fleet can sustain.

The mental model that keeps designs honest: a queue is a durable promise of intent, not a record of completion. The producer’s promise is “this work is owed”. Nothing in the queue says it has been done, only that nobody has yet told the broker it was. Everything hard in this lesson follows from the gap between those two statements: a worker can finish the work and die before saying so, or say so and be wrong about whether the work really committed. Two things are consequently non-negotiable. The path from “request accepted” to “job durably recorded” has to be a correctness boundary, and the work has to be safe to run more than once.

A message passes through a few phases: produce → persist → make available → receive or claim → execute → acknowledge or delete, with retries and eventual quarantine as side exits. Each hand-off fails differently, and the two most important facts are negative. A worker receiving a message is not evidence that its side effect committed. A worker completing its side effect is not evidence that the broker heard about it.

Follow one job, img_42:

1. The API stores the original image in object storage, creates a stable job ID, and publishes a message that references the object. The queue carries a pointer, not megabytes of pixels.
2. The broker makes the message available. Worker A receives it and starts working.
3. Worker A writes the thumbnail to object storage and marks the job SUCCEEDED in the job database.
4. Worker A acknowledges the message. Only now may the broker drop it from the active delivery path.

If Worker A dies after step 3 and before step 4, the broker never learns the work finished. After whatever timeout or connection-loss detection the broker uses, it hands img_42 to Worker B. That is normal at-least-once delivery, not a broker defect, and the figure walks through exactly that branch.

Read the figure by lifeline. Messages 1 and 2 are the producer-side boundary: the publish is only safe to rely on once the broker confirms it stored the message. Message 3 is the first delivery; messages 4 and 5 are the worker’s side effects, ordered so that the job row records the outcome. Message 6 is the crash; nothing is sent to the broker. Message 7 is the broker acting on silence, by timer in this SQS-style picture. Message 8 is the redelivery; messages 9 and 10 are the idempotency check: Worker B consults durable job state, finds SUCCEEDED, and skips the work. Message 11 is the acknowledgment that ends the message’s life. Had the crash come between messages 4 and 5, Worker B would have found the job unfinished and redone the thumbnail. That is also safe, because the output key is derived from the job ID, so the second write replaces the first with identical bytes.

Where a crash leaves things

If the worker acknowledged before doing the work, the second row turns into permanent loss instead of redelivery. Acknowledge-first is at-most-once delivery by construction, which is why the default discipline in this lesson is to acknowledge last.

Returning before the work is done moves a question from the server to the client: how will I find out how it went? The conventional answer is the asynchronous request-reply shape that the REST lesson describes: a 202 Accepted response, a job identifier, and a status resource the client can poll or be called back on.

POST /v1/images HTTP/1.1
Idempotency-Key: 7c1f6e0a-3b52-4d0e-9a41-5f2d6c1b8e90
Content-Type: image/jpeg

HTTP/1.1 202 Accepted
Location: /v1/jobs/img_42
Retry-After: 2

{ "job_id": "img_42", "status": "QUEUED" }

GET /v1/jobs/img_42     ->  { "job_id": "img_42", "status": "RUNNING" }
GET /v1/jobs/img_42     ->  { "job_id": "img_42", "status": "SUCCEEDED",
                              "result": "/v1/images/img_42/thumbnail" }

Two details carry the correctness. First, the status endpoint reads the job table, never the queue. Queue state can answer “is it waiting”; it cannot answer “did it succeed”, and a deleted message is as consistent with success as with a worker that acknowledged and then lost its write. Second, the client-supplied Idempotency-Key lets a client that timed out retry the upload without creating a second logical job; the mechanics are in the Idempotency lesson.

The harder boundary is internal. A queue is not automatically durable: whether a published message survives a broker crash depends on how the broker is configured and on what its acknowledgment means. SQS acknowledges a send after storing the message redundantly. RabbitMQ needs a durable queue, persistent messages, and publisher confirms before “the broker has it” is true; quorum queues confirm after a majority of replicas have the message. Kafka needs acks=all with a sensible min.insync.replicas. In every case the producer must wait for that confirmation, or it is returning 202 for work that exists only in memory.

That leaves the classic dual-write problem. If the handler inserts the job row in the database and then publishes to the broker, a crash between the two leaves a job that is QUEUED forever and no message to run it. Reversing the order leaves a message with no job row. A database transaction cannot span the broker, so the fix is to make one of them the source of truth and derive the other, which is the subject of the Transactional Outbox lesson. A reconciler that re-publishes jobs stuck in QUEUED is the cheap stop-gap, and it is safe only because the consumer is idempotent.

In SQS, receiving a message does not remove it; it makes the message invisible to other receivers for the visibility timeout (30 seconds by default, up to 12 hours). If the worker deletes the message before the timer runs out, the job is finished. If it crashes, or simply fails to delete in time, the timer expires and the message becomes receivable again. Each receive increments the message’s receive count and hands out a fresh receipt handle.

The figure shows a message that is Ready, becomes In flight on receive, and leaves In flight in one of two ways. Deletion ends it. Timeout or an explicit return sends it back to Ready. A message whose next receive would take the receive count past the queue’s redrive limit (maxReceiveCount) goes from Ready to a dead-letter queue instead of being delivered; what to do there is covered in the Dead-Letter Queues lesson.

A timeout is a lease on visibility, not a lock on the work

The visibility timeout tells the broker when to stop hiding the message. It does not stop the original worker. A worker that outlives its timeout keeps running, and a second worker, correctly following the queue’s rules, starts the same job. For some stretch of time two processes hold the “same” work. The queue cannot prevent this; only the handler can tolerate it, by idempotency and, when stale writers must be rejected rather than merely tolerated, by fencing. The lease-and-fencing machinery, including how a stale worker is kept from overwriting a newer one, belongs to the Job Processing lesson.

Experiment with it below. A job that takes 45 seconds against the default 30-second timeout is delivered twice, because the message reappears at second 30 while the first worker is still busy. Turn on extension and the duplicate disappears, at the price of the worker having to keep calling the broker.

Sizing the timeout

The timeout trades two failures against each other. Too short, and healthy slow jobs get duplicated; the duplicates consume capacity, which makes more jobs slow, which creates more duplicates. Too long, and a crash leaves the message invisible for the whole timeout: with a 15-minute timeout, a crash at second 2 costs nearly 15 minutes of latency for that job, and the figure’s “crash recovery delay” readout is that number.

Two practical rules follow. Size the timeout from the tail of processing time, not the mean; a common starting point is a small multiple of the p99 duration, to be revisited with real data. And for jobs whose duration is genuinely variable, prefer a short timeout plus periodic extension (ChangeMessageVisibility sets the timer to a new value counted from now, with a ceiling of 12 hours since the message was first received). Extension turns a crashed worker back into a short wait, because the heartbeats stop. It also adds a failure mode of its own: a worker that is alive but stuck keeps extending forever, so extension needs a hard cap on total processing time.

One more SQS detail that bites in practice. Every receive gives a different receipt handle, and deletion should use the most recent one. Deleting with an older handle can return success without removing the message, which looks exactly like a duplicate that “came back”.

The question “when is a message done and who owns it meanwhile” has different answers in different brokers, and mixing the vocabularies is a common source of production bugs.

RabbitMQ: acknowledgments on a channel

RabbitMQ pushes messages to consumers over a channel. Each delivery stays unacknowledged until the consumer sends basic.ack; there is no per-message visibility timer that hands it to another consumer. Instead, when the channel or connection closes with deliveries still unacknowledged, the broker requeues them, and redelivered messages carry a redelivered flag. A consumer can also nack or reject with or without requeue. The failure that a visibility timeout would catch, a worker that is connected but stuck, is handled differently: RabbitMQ has a delivery acknowledgement timeout (30 minutes by default in current versions) after which the channel is closed and the messages are requeued. Quorum queues add a delivery limit so a message that keeps coming back is dead-lettered instead of looping.

Kafka: offsets and partition ownership

Kafka is a log, not a queue in the SQS sense. Messages are never deleted by consumption. Within a consumer group each partition is assigned to at most one consumer, and progress is a single committed offset per partition: the position of the next record to read. There is no per-message ack, no nack, and no visibility timeout. To get at-least-once behaviour you commit the offset after the work is done; commit before the work and a crash loses the records in between. Ownership is a lease of a different kind: a consumer that stops polling for longer than max.poll.interval.ms, or stops heartbeating for longer than the session timeout, is removed from the group and its partitions are reassigned, and the new owner resumes from the last committed offset, reprocessing anything the old owner had done but not committed.

Because consumption does not delete, replay is a first-class operation: seek the group back to an earlier offset and read again, for as long as retention keeps the data (time- or size-based, seven days by default, or forever for a compacted topic, which keeps the latest record per key). Parallelism is capped by the partition count; a seventh consumer in a six-partition topic sits idle. And since one slow or poisonous record blocks everything behind it in the same partition, retry topics and dead-letter topics are patterns you build, not features you switch on.

Note what “replay” means in the “after success” row. Redelivery of an unacknowledged message is recovery; re-reading history is replay. SQS and RabbitMQ do the first only. Once a message is deleted or acked, the only copy is whatever you stored yourself.

Whatever the mechanism, a design should answer four questions in writing. When is a message considered done? What causes redelivery? How long can two workers legitimately overlap? Where is the durable source of truth for job state? If the answer to the last is “the queue”, the system cannot say whether a job succeeded.

At-most-once means a message may be lost but is not deliberately redelivered. It is produced by acknowledging before doing the work, or by sending without confirmation, and it suits disposable telemetry where a lost sample is cheaper than a duplicate. At-least-once means a committed message will be delivered one or more times; it is what you get by acknowledging after the work and the default for nearly every production queue. The price is duplicates, which are the application’s problem.

Exactly-once is only meaningful with its scope attached. Kafka can provide exactly-once within Kafka: an idempotent producer prevents duplicates from its own retries, and a transactional consume-process-produce loop commits output records and the consumed offsets atomically, visible to consumers reading with isolation.level=read_committed. SQS FIFO queues deduplicate sends that carry the same deduplication ID within a five-minute window, but a consumer can still receive a message twice after a visibility timeout. Neither reaches out and makes your HTTP charge, your email, or your S3 write atomic with the acknowledgment. Unless the side effect participates in the same transaction or protocol as the acknowledgment, no broker setting can give you that.

So the practical target for business effects is effectively-once: at-least-once delivery plus handlers whose repeated execution has no additional effect. That is built from durable state, stable operation IDs, and database constraints, and it is what the next section implements. The Idempotency lesson covers the request-level version of the same idea.

The simplest idempotent handler is one whose effect is naturally repeatable: writing a thumbnail to a key derived from the job ID, or setting a status to SUCCEEDED. When the effect is not naturally repeatable, record a unique operation ID and make that record commit in the same database transaction as the effect. Run in PostgreSQL, the following is complete as written (the literals stand in for parameters your driver would bind).

CREATE TABLE jobs (
  id          text PRIMARY KEY,
  status      text NOT NULL CHECK (status IN ('QUEUED','RUNNING','SUCCEEDED','FAILED')),
  object_key  text NOT NULL,
  output_key  text,
  created_at  timestamptz NOT NULL DEFAULT now(),
  started_at  timestamptz,
  finished_at timestamptz
);

CREATE TABLE processed_operations (
  consumer_name text NOT NULL,
  operation_id  text NOT NULL,
  processed_at  timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (consumer_name, operation_id)
);

INSERT INTO jobs (id, status, object_key) VALUES ('img_42', 'QUEUED', 'originals/img_42.jpg');

-- One statement, so the marker and the mutation commit or roll back together.
-- Affected rows = 1: first execution applied. 0: duplicate, nothing changed.
WITH claim AS (
  INSERT INTO processed_operations (consumer_name, operation_id)
  VALUES ('thumbnail-worker', 'img_42')
  ON CONFLICT DO NOTHING
  RETURNING 1
)
UPDATE jobs
   SET status = 'SUCCEEDED', output_key = 'thumbs/img_42.jpg', finished_at = now()
 WHERE id = 'img_42'
   AND EXISTS (SELECT 1 FROM claim);

When the mutation spans several statements, use an explicit BEGIN, do the marker insert first, read its affected-row count in application code, and run the rest of the mutation only if the count is 1, then COMMIT. The one rule that cannot bend is that the marker and the effect share a transaction. A marker committed before the work, and a crash after it, produces a job that every later delivery considers “already handled” and nobody ever finished.

Often you do not need the separate table at all. If the job row has terminal states, the guard is just a conditional update: UPDATE jobs SET status = 'SUCCEEDED' … WHERE id = $1 AND status <> 'SUCCEEDED'. A dedupe table earns its keep when there is no natural state column to guard, or when one consumer applies many different effects. Dedupe rows should be pruned on a schedule longer than the longest period a message can reappear, which is bounded by queue retention.

Be exact about what this does not protect. The transaction covers rows in one database. It does not make the thumbnail upload, the HTTP call to a payment provider, or the acknowledgment to the broker atomic with the commit. For those, either use a naturally idempotent write (deterministic object keys), pass a stable idempotency key to the external system so it deduplicates on its side, or record intent before the call and reconcile uncertain outcomes afterwards. The worker loop that ties these together looks like this.

loop:
  msg = broker.receive(wait = 20s)
  if msg is none: continue

  job = db.get(msg.job_id)
  if job is none or job.status == SUCCEEDED:
      broker.ack(msg)                       # duplicate or stale: just finish the message
      continue

  try:
      out_key = "thumbs/" + job.id + ".jpg"  # deterministic: a re-run overwrites itself
      storage.put(out_key, make_thumbnail(storage.get(job.object_key)))
      db.complete_job(job.id, out_key)       # the transactional statement above
      broker.ack(msg)                        # last, always
  except PermanentError as e:
      db.fail_job(job.id, reason = e)        # record why, then stop retrying
      broker.ack(msg)                        # or route to the DLQ, never loop forever
  except TransientError:
      broker.release(msg)                    # let the broker redeliver; backoff is its job

A queue smooths short-term differences between how fast work arrives and how fast it is completed. It does nothing for the sustained difference. Let λ be the arrival rate and μ the rate at which the fleet completes jobs. When λ stays above μ the backlog grows by about λ − μ jobs per second. At 120 arrivals per second against 100 completions per second, that is 20 jobs per second, or roughly 1,200 more waiting jobs every minute, until something gives: a retention limit, memory, patience, or an incident.

Explore the dynamics below. The defaults reproduce that example for a two-minute spike on top of a baseline of 60 jobs per second: the backlog peaks near 2,400 jobs and drains about a minute after the spike ends. Then lower the downstream ceiling to simulate a shared database or a third-party quota, and add workers. Past the ceiling, extra workers change nothing.

That last behaviour is why autoscaling on queue depth alone is a trap. Workers multiply load on whatever they call. When the limit is the database, the quota of an external API, or a lock, more workers raise latency for everyone and can tip the dependency over. Cap fleet size at what the dependency sustains, apply rate limiting on the worker side of that call, and use bulkheads so one slow integration cannot consume every worker.

Measure age, not just depth

A depth of 10,000 means little without the job duration. Ten thousand jobs that take 20 ms each drain in minutes on a modest fleet; ten thousand that take an hour each represent days of work. The signal users actually feel is age of the oldest message (SQS publishes it as ApproximateAgeOfOldestMessage; Kafka’s analogue is consumer lag, best translated into time). For a FIFO queue the age at any instant is roughly the backlog divided by the completion rate, which is how the figure derives its “oldest job at peak” readout.

Little’s Law, L = λW, relates the average number of jobs in a stable system to the arrival rate and the average time each spends there. It is a good sanity check: 100 jobs per second at 0.2 seconds each means about 20 jobs in service at once. It does not apply to a backlog that is growing, because a growing queue has no steady state to average over, and it should not be used to predict wait times there.

Deciding what happens when you cannot keep up

Pick the policy before the incident. State a maximum acceptable age for each class of job and decide what happens beyond it. Some work should be refused at submission when the system already cannot meet its promise, a decision that belongs to load shedding and to backpressure propagated to producers. Some can be delayed or coalesced: a thousand “recompute this user’s cache” messages need one computation. Some expire: a notification that is two hours late is worse than none, so give it a TTL or a deadline the worker checks on receipt. Priority queues need an explicit starvation control, usually separate queues with weighted polling, or low-priority work never runs under sustained high-priority load.

Throughput per worker depends on service time, CPU and memory, I/O waits, and the limits of whatever it calls. For I/O-bound work, more concurrent workers raise throughput until the dependency saturates. For CPU-bound work, adding workers beyond the cores on a host stops helping and begins to hurt through context switching and memory pressure. Concurrency inside a worker (threads, async tasks) and the number of worker processes multiply, so decide which dimension you are tuning.

For a first capacity estimate, divide the target throughput by what one worker completes. If a worker finishes 5 jobs per second and the target is 100 jobs per second, 20 equally capable workers is the floor. That floor assumes perfectly even load. At a target utilization of 70% to leave headroom for variance, failures, and deploys, the same target needs 100 / (5 × 0.7) ≈ 29 workers. Use measured completion rates and tail durations, not a single average, and check the result against the downstream ceiling from the previous section.

Prefetch: reserving messages you cannot yet process

A worker that takes many messages at once reserves them: they are in flight and invisible to everyone else while sitting in its local buffer. This is deliberate for throughput, because it hides network round trips when jobs are tiny. For slow or uneven jobs it is a fairness problem. In RabbitMQ the setting is the consumer’s prefetch count (basic.qos), the maximum number of unacknowledged deliveries it may hold. SQS has the same effect through batch receive (up to 10 messages per call), all of which share the timer. In Kafka the analogue is the batch returned by one poll.

The figure shows the failure mode. Three workers share 40 messages; two take one second each, one takes eight seconds. With prefetch 1, the slow worker takes a message only when it is free, so the fast workers absorb most of the work and the queue empties in 24 seconds. At prefetch 10, the broker has already handed the slow worker ten messages before it finished the first, and the queue is not empty until it works through all ten: 80 seconds. Nothing was lost or failed; messages were simply committed to a worker that cannot get to them. They are also exposed to its visibility timeout or channel loss while they wait.

A workable rule: use a low prefetch (often 1) for long or variable jobs, and raise it only as far as needed to keep fast, uniform jobs from being limited by round-trip latency. Wherever you land, make sure the time a buffered message may wait fits inside the timeout that protects it.

A competing-consumer queue gives no global processing order. Even when messages leave the broker in order, parallel workers, retries, and uneven durations change the order in which work finishes. A “balance updated to 50” message that is retried can complete after a later “updated to 80”, leaving the stale value in the database. RabbitMQ requeues try to restore a message’s original position but with several consumers or a prefetch above 1 the effect is still reordering; SQS standard queues make no ordering promise at all.

Most systems do not need a total order, only an order per entity: all events for one customer, order, or account. There are three ways to get it, and they combine.

• Partition by key and serialize within a partition. Kafka’s key-to-partition mapping, SQS FIFO message groups, or one queue per shard of the key space put all events for one entity on one lane, consumed by one worker at a time.
• Enforce the order at the state store. Carry a version or sequence number and make the write conditional: apply event n only if the entity is at n−1. Out-of-order or duplicate events are rejected or parked rather than applied, which also defends against redelivery.
• Keep a single consumer, accepting that a globally ordered queue becomes the throughput bottleneck of the system (RabbitMQ’s single active consumer is one way to run it).

The figure shows the first approach: a router hashes the key to a lane, each lane has one active worker, so events for one customer are serial while different customers proceed in parallel. The cost of per-key ordering is head-of-line blocking. If an event at the front of a lane keeps failing, everything for that key (and everything else hashed to the lane) waits behind it. That is the strongest argument for bounded retries and a way to park a failing message, the next topic. Also note that SQS FIFO and Kafka bound the parallelism you can get: many hot keys on few lanes limits throughput, and a few keys with all the traffic cannot be spread out.

A poison message fails every time: malformed data, a schema the consumer does not understand, or a deterministic bug in the handler. Retrying it forever burns capacity, spams logs, and in an ordered lane blocks everything behind it. Contrast the transient failure, a timeout or a briefly unavailable dependency, which a retry usually fixes.

The handler’s job is to classify. Permanent failures (validation errors, unknown schema version, business rules that say no) should stop immediately and record why. Transient failures should be retried a bounded number of times, ideally with growing delay and jitter, as set out in the Retry & Backoff lesson. A queue offers delay mechanisms for this: SQS delay queues and per-message delays (up to 15 minutes), a visibility timeout set deliberately on release, RabbitMQ’s TTL-plus-dead-letter patterns, and, on Kafka, separate retry topics with increasing delays.

Retries must end somewhere. When attempts are exhausted, move the message, with its failure reason and attempt count, to a dead-letter queue and alert on its volume. SQS does this by a redrive policy on receive count; RabbitMQ through dead-letter exchanges, and a delivery limit on quorum queues; Kafka through a producer you write. What to inspect, whether to fix the code or the data, and how to replay safely is the business of the Dead-Letter Queues lesson.

The basic shape is the work queue: each message goes to exactly one worker out of a pool, and adding workers raises throughput. Variations adjust one property. A delayed queuemakes messages visible only after an interval, which serves scheduled work and retry-after windows. A priority arrangement serves urgent work ahead of bulk work, normally as separate queues with weighted polling to avoid starvation. A dead-letter queue parks messages that exhausted their attempts, as above.

The shape to keep separate is fan-out. A work queue delivers each message to one consumer on purpose. If several independent systems each need every event (billing, search indexing, and analytics all reacting to “order placed”), consuming from one shared queue gives each of them a fraction of the events, not all of them. That is a publish/subscribe problem, with a subscription per consumer, and it is the subject of the Pub/Sub lesson.

A visibility timeout shorter than the job

The timeout is 30 seconds. After a dependency slowdown, jobs take 45 seconds. Maximum receive count is 3.

1. At 0 s, Worker A receives job J (receive count 1) and starts working.
2. At 30 s, J becomes visible again. An idle Worker B receives it (count 2) and repeats the work, side effects included.
3. At 45 s, A finishes and tries to delete J with its original receipt handle. That handle is stale, so the message may remain.
4. At 60 s, B’s timer expires with B at 30 s of 45. Worker C receives J (count 3).
5. At 90 s, the timer expires again, a receive would exceed the limit, and J goes to the DLQ, successfully processed and quarantined as a failure.

Every job costs up to three times the work, so the fleet has less spare capacity, so more jobs queue and slow, so more exceed the timeout. The fix is on three levels: size the timeout from tail latency, extend it while working, and make the handler idempotent so the duplicates are harmless.

Autoscaling into a bottleneck

1. A backlog of 50,000 jobs appears after an upstream batch. The autoscaler goes from 20 workers to 200.
2. Each job does one database write. The database comfortably handled 100 writes per second from 20 workers; 200 workers now ask for about 1,000.
3. Database latency climbs, job durations rise, timeouts expire, and the duplicates add even more writes.
4. The backlog is bigger than before the scale-out, and now the database is down too.

Scale to the dependency’s limit, not the queue’s depth, and let the queue hold the rest. A bounded fleet with a long backlog is an inconvenience; an unbounded fleet that takes the database down is an outage.

Accepting work that was never enqueued

1. The handler inserts a job row (QUEUED), commits, and then publishes to the broker.
2. The process is killed between the commit and the publish during a deploy.
3. The client has its job ID and polls forever; the job exists only as a row no worker knows about.

The durable fix is the outbox; the operational fix is a reconciler that finds jobs QUEUED for longer than expected and re-publishes them. Either way, a status of QUEUED older than the expected wait is an alarm, not a state.

The API stores the original in object storage and inserts a job with status QUEUED. A publisher emits ImageProcessingRequested(job_id, object_key, schema_version) only after that durable acceptance boundary, using an outbox if the job table and the broker are separate systems. Workers receive the message, load the job, and return early if it is already terminal. Otherwise they write outputs under deterministic keys and atomically mark the job complete, as in the earlier statement, then acknowledge. GET /jobs/{id} reports QUEUED, RUNNING, SUCCEEDED, or FAILED from the table.

A reconciler runs on a schedule and closes the gaps that no broker can. For jobs that never got published:

-- Jobs accepted more than 5 minutes ago that no worker has started.
SELECT id, object_key
  FROM jobs
 WHERE status = 'QUEUED'
   AND started_at IS NULL
   AND created_at < now() - interval '5 minutes';
-- Re-publish a message for each row. Safe only because the consumer is idempotent.

Jobs stuck in RUNNING need more care, since a slow worker and a dead one look identical from the table. Deciding which one it is, and rejecting writes from the loser, is what leases and fencing tokens are for, covered in Job Processing.

What to watch

A green worker CPU chart does not show that the pipeline meets its promise. The signals that do:

• Enqueue failures at the producer, and the age of the oldest QUEUED job that has no message.
• Age of the oldest message and depth: for SQS, ApproximateAgeOfOldestMessage and the visible versus not-visible counts; for RabbitMQ, ready versus unacknowledged messages; for Kafka, consumer lag.
• Completion throughput and p95 or p99 duration, per job type, against the visibility timeout.
• Redelivery: receive counts above 1, and the redelivered flag in RabbitMQ. A rising rate usually means timeouts too short or workers dying.
• Retry rate, DLQ volume and age, with an alert on the first poison message, not the thousandth.
• End-to-end latency from job creation to SUCCEEDED, the number the client actually experiences.

Use a queue when the work can finish asynchronously, bursts need absorbing, retries are acceptable, and completion can be observed by another route (a status endpoint, a webhook, or a later notification). Slow media processing, report generation, email and webhook delivery, and analytics events all fit. In a checkout the same split applies at a finer grain: the payment decision the customer is waiting for stays in the request, while the confirmation email, inventory notification, and analytics move behind a queue.

Avoid it when the caller needs a transactionally consistent answer in the same request, such as a login check, a fast read, or an authorization the user must see before leaving the page, unless you have deliberately designed an asynchronous protocol for it. A queue also adds a dead-letter queue, retry policy, idempotent handlers, job tracking, and monitoring. That is a good trade for long-running or unreliable work and a poor one for a 10 ms operation.

Between “do it inline” and “run a broker” sits the database job table: a jobs table that workers claim with SELECT … FOR UPDATE SKIP LOCKED. For modest volume it is often enough, and it brings real transactions, so enqueuing can happen in the same commit as the business change. It needs deliberate claiming, indexes on the claim path, and retention so the table does not bloat, and it gives up the independent scaling and delivery features of a broker. The claim query and its trade-offs are examined in Job Processing and, from the publishing side, Transactional Outbox.

Read the flow top to bottom. The first question separates work the caller is waiting on from work it is not. The second recognizes that several independent consumers per event is a pub/sub problem. The third refuses to introduce a queue before the handler can survive a duplicate, because at-least-once delivery will produce one. Only then does volume and where your state lives decide between a table and a broker.

• Accept work only after durable publication: return 202 with a job id once the message is safely enqueued, and never accept work that was not enqueued.
• A visibility timeout is a lease on visibility, not a lock on the work. Size it above the longest realistic job or extend it with heartbeats, or a second worker will run the same message.
• Acknowledgment rules differ per broker: SQS uses a visibility timeout, RabbitMQ requeues on channel close and has an acknowledgement timeout, and Kafka tracks partition offsets.
• Delivery is at-least-once, so write the dedupe marker in the same transaction as the effect and know what that transaction does not cover.
• A queue absorbs bursts but creates no capacity: watch the age of the oldest message, not just depth, and decide up front whether to shed, delay or scale when workers cannot keep up.
• Bound retries and route messages that can never succeed to a dead-letter queue; autoscaling workers into a shared bottleneck makes the incident worse.

Redelivery is only safe with Idempotency, and the delay between attempts is the subject of Retry & Backoff. When producers outrun workers, Backpressure is how the pressure travels upstream. For jobs that outlive a visibility timeout, see Job Processing for leases and fencing; for messages that cannot succeed, Dead-Letter Queues; for several consumers per event, Pub/Sub; and for publishing reliably from a database transaction, the Transactional Outbox.