Backpressure

Backpressure is a feedback mechanism that stops a producer from sending work faster than a consumer can finish it. When the consumer slows down or the queue in front of it nears its limit, the system tells upstream components to pause, slow down, or send less. It is about controlling the rate of work entering a bottleneck, not about storing the excess somewhere.

The arithmetic is the whole motivation. Suppose a service receives 2,000 events per second and its workers sustain 1,200. The backlog grows by 800 per second. After one minute that is about 48,000 events, assuming constant rates and no drops. At 4 KiB each, payload storage alone is 48,000 × 4 KiB = 187.5 MiB, before object headers, indexes, replication, or runtime bookkeeping. A bigger queue only moves the day the process falls over; it does not change the 800-per-second imbalance.

The mental model is a pipe feeding a narrow valve. Pressure builds behind the valve, and a well-designed system lets that pressure travel back up the pipe so the source eases off, instead of letting the pipe balloon. The question every design must answer is: what happens when input exceeds sustainable capacity for longer than the buffer can absorb? The honest options are to slow the producers, queue within a hard bound, reject work, drop low-value work, or add capacity. Each has different consequences for correctness and latency, and most real systems combine several.

A buffer absorbs a temporary mismatch. If a consumer stalls for 100 ms, a bounded queue lets the producer carry on without anyone noticing. If production stays above consumption, though, every buffer fills eventually, whatever its size. Backpressure begins at the moment fullness changes what the upstream side does.

Try the figure below. With the defaults (2,000 arriving, 1,200 served, 30,000 slots) the queue is full after 30,000 ÷ 800 = 37.5 seconds. Drag the capacity to its maximum and the failure moves out to 125 seconds. Drag the service rate to match arrivals and the line goes flat. Capacity changes when; only the rate difference changes whether.

Unbounded queues turn overload into a different outage

An unbounded queue converts a rate problem into memory exhaustion, long tail latency, and stale work. The memory failure is the loud one. The quiet one arrives first: a FIFO queue that holds N items in front of a consumer serving μ items per second makes the newest item wait about N ÷ μ seconds. At 1,200 per second, a 48,000-item backlog means every new request waits 40 seconds, long after most callers have timed out and retried. The consumer then spends its capacity on work nobody is waiting for.

A bounded queue makes overload visible early and makes the policy explicit: when the queue is full, something specific happens. Choosing the bound by available RAM is the wrong starting point. Start from the wait you can tolerate.

Sizing the bound with Little's Law

Little's Law relates the average number of items in a stable system to the average arrival rate (which equals throughput in steady state) and the average time each item spends in the system:

L = λ × W

L = average items in the system     λ = average throughput     W = average time in system

example:   λ = 500 requests/s, W = 0.2 s      ->  L = 100 requests in flight on average

queue sizing (FIFO queue drained at service rate μ):
   max wait  W_max  ≈  N / μ          ->   N  ≈  μ × W_max
   μ = 1,200/s, W_max = 5 s           ->   N  ≈  6,000 items
   memory at 4 KiB/item               ->   6,000 × 4 KiB ≈ 23.4 MiB

The law holds for stable systems and says nothing useful about a queue that is growing without bound, so do not use it to predict where an overloaded queue "settles". It does give the practical rule: the queue bound is a latency budget in disguise. A 6,000-slot queue in front of a 1,200-per-second consumer promises at most about five seconds of queueing. Check the memory figure afterwards, as a constraint, not as the target. The bound should also be smaller than the callers' own timeout, otherwise admitted work is already worthless by the time it is served.

Every mechanism answers one question: how does the consumer's limit reach whoever decides how much work exists? They differ in who holds the state, how fast the signal travels, and what happens to the work that does not fit.

Pull-based demand

The consumer asks for only as much as it can handle. In Reactive Streams-style protocols the subscriber calls request(n), and a compliant publisher must not signal more onNext items than the outstanding demand. The in-flight work is bounded by what the subscriber has asked for, so memory is bounded by design. Kafka consumers and many job queues are pull-based in the same spirit: nothing is delivered until the consumer fetches it.

Two cautions. First, demand only controls delivery through that protocol. It does not stop an unrelated external source unless the signal is propagated all the way upstream, hop by hop. A reactive pipeline fed by an HTTP endpoint that accepts requests as fast as they arrive has bounded the inside and left the edge open. Second, request(Long.MAX_VALUE) means unbounded demand and quietly turns the mechanism off; several operators also prefetch a small bounded batch (a default such as 128 or 256 is common), which is fine, but it is a buffer you should know about.

Blocking or pausing the producer

A producer waits when a bounded channel is full. Inside a process this is the simplest correct answer: a bounded Go channel, a Java ArrayBlockingQueue.put, or a bounded asyncio.Queue. Streams expose the same idea as a return value: in Node.js, writable.write() returns false once the internal buffer passes highWaterMark, and the producer is expected to wait for the drain event. The trap is the blocked thread. Thread-per-producer designs can be exhausted by the very overload they are meant to survive; suspending asynchronously (as in the example below) holds no thread while it waits.

import asyncio

queue: asyncio.Queue = asyncio.Queue(maxsize=6000)   # sized from a latency budget

async def producer(source):
    async for job in source:
        await queue.put(job)       # suspends while full; no thread is held

async def worker():
    while True:
        job = await queue.get()
        try:
            await handle(job)
        finally:
            queue.task_done()

Credit-based flow control

The downstream side advertises how many bytes or messages it can accept. The sender spends credits as it sends and waits for replenishment. TCP's receive window and HTTP/2 stream windows are credit systems measured in bytes; many messaging protocols grant credit in messages. Credits turn the buffer from something that happens to the system into something the receiver sizes on purpose. The unit and replenishment rules differ per protocol, so the details matter more than the name.

Explicit rejection

An API returns 429 Too Many Requests or 503 Service Unavailable, or a broker refuses a publish, because it cannot safely queue more. The status codes mean different things: 429 says this caller is exceeding what it is allowed to send, while 503 says the server is temporarily unable to handle the request, which is the right signal for saturation that is not the caller's fault. Both may carry Retry-After, either as seconds or as an HTTP date; it is advisory, so well-behaved clients honor it and add jitter so they do not all return at once. gRPC services typically use RESOURCE_EXHAUSTED or UNAVAILABLE for the same purpose. The RateLimit-* response headers are still an IETF draft rather than a finished standard, so treat them as a convention you control on both ends.

Rejection is admission control, and it creates backpressure only if callers respond by sending less or later. A client that retries immediately converts each rejection into more load: three immediate retries per failure can multiply offered traffic by up to four. See Retry & Backoff for budgets and jitter, and Load Shedding for choosing what to refuse.

Dropping, sampling, and coalescing

A telemetry pipeline can drop low-priority debug events when saturated; a trading ledger cannot drop a single entry. The right loss policy depends on the value of the data, whether the source can replay it, and what you have promised. Two less blunt variants exist: sampling keeps one in N items, and coalescing (also called conflation) keeps only the latest value per key, which suits position updates and gauges where an older value is obsolete the moment a newer one exists.

Video players show the pull idea at the user's edge: they request the next segment only when their playback buffer drops below a target, and adaptive-bitrate logic lowers quality when downloads cannot keep up. The buffer is bounded, demand is driven by the consumer, and quality is the degradation knob.

The figure compares the three non-capacity outcomes for the same 2,000-versus-1,200 mismatch. Slowing the producer loses nothing but pushes latency upstream. Rejecting keeps admitted work healthy and puts failure handling on callers. Dropping keeps everything running and permanently loses data. Adding consumers is the fourth option and the only one that changes the 1,200; it costs money, takes time, and helps only if the consumers are the real bottleneck.

A bounded buffer forces a decision at the moment it fills. There are four standard answers, and each is correct for some data and dangerous for other data.

Spilling deserves skepticism. It is a legitimate way to survive a burst without loss, but it recreates the unbounded-queue problem on a bigger medium: the backlog is safe and the wait is hours. Pair it with an age limit or a clear plan for draining it.

Age matters as much as depth. A queue can drop items at dequeue time when they have waited longer than the caller is still willing to wait, instead of doing the work and discarding the result. Several brokers have equivalents: RabbitMQ queues can be given a maximum length with an overflow behavior such as dropping the head or rejecting new publishes, and a per-message or per-queue TTL expires items by age.

TCP is the original backpressure design. The receiver advertises a window in every acknowledgement; the sender may have no more unacknowledged data in flight than the smaller of that receive window and its own congestion window. When the receiving application stops reading, the socket buffer fills, the advertised window shrinks to zero, and the sender stops, probing periodically until the window reopens. The sender's own send buffer then fills, so a blocking write blocks or a non-blocking one reports that it cannot accept more. That is backpressure reaching application code, for free.

The limit is that TCP measures how fast the receiving process reads from the socket, not how fast it finishes the work. A typical read loop reads every message and puts it on an in-process queue. The socket buffer drains immediately, the window stays open, and TCP reports no pressure while the queue, and the database behind it, drown. Data can also accumulate in proxies, load balancers, and kernel buffers, all of which look like healthy transport.

HTTP/2 and gRPC have the same shape. HTTP/2 flow control uses window credits per stream and per connection, replenished by WINDOW_UPDATE frames, with an initial window of 65,535 bytes in the specification. It prevents a fast sender from overrunning a receiver's buffers; it says nothing about the business processing rate. A gRPC server stream that reads messages as fast as they arrive and enqueues them in an unbounded application queue is overloaded in exactly the TCP example above. gRPC libraries expose the hook to do better: per-call manual flow control (request(n) in the Java API) and readiness callbacks such as isReady/onReady on the sending side.

The fix is to connect the application's true limit to the read side: bound the in-process queue, and when it is full, stop reading (or stop granting stream credit). From that moment the database's limit becomes a full socket buffer, a closed window, and a sender that waits. End-to-end backpressure is that chain of bounds; one unbounded link silently breaks it.

Kafka: lag is a symptom, not a signal to producers

A Kafka consumer is pull-based: it fetches batches at its own pace, so the broker never overruns it. Consumer lag (log end offset minus committed offset) grows whenever consumption trails production. That lag does not slow producers. Producers write to the log independently of any consumer, and Kafka retains records by its time or size retention rules. If lag grows until the oldest unread records are deleted, the consumer's offset becomes invalid and its auto.offset.reset policy decides whether it jumps to the earliest or latest available record, and unread data is gone. Lag is evidence of an imbalance, not proof that backpressure is reaching anyone.

What a consumer can do is manage its own application pressure. consumer.pause(partitions) stops fetching from those partitions while poll() continues to be called, so the member stays in the group; stopping the polling would exceed max.poll.interval.ms and trigger a rebalance. max.poll.records bounds how much one poll hands to the application, and the consumer's internal fetch buffer is bounded by settings such as fetch.max.bytes. A rebalance can change partition assignment, so reapply pauses in the rebalance listener rather than assuming they persist.

Producer-side Kafka has its own, separate pressure: the producer's buffer.memory fills when the broker cannot keep up, and send() then blocks for up to max.block.ms before failing. Broker quotas can also throttle a producer's byte rate. Neither is driven by downstream consumer lag. Propagating consumer slowness back to a Kafka producer is something you build, typically by throttling at the system that feeds the producer.

Queues with leases and prefetch

In queue-based systems, prefetch limits, visibility timeouts, and maximum in-flight counts bound how much work is assigned to workers. In RabbitMQ with manual acknowledgements, basic.qos prefetch_count caps the unacknowledged messages a consumer holds; the broker withholds further deliveries until acks arrive. Too high a prefetch concentrates a backlog on a slow consumer while its peers sit idle; too low wastes capacity on round trips. Queue services with visibility timeouts hide a received message for a lease period, so an overloaded worker that cannot finish in time causes redelivery and duplicate work. Keep three things distinct: the queue's backlog, the number of messages currently leased to workers, and the application's own in-memory queue. Each needs its own bound and its own metric. See Queues & Workers for delivery semantics, and make consumers idempotent as described in Idempotency, because overloaded leases lead to duplicates.

Thread pools and executors

A bounded executor queue is a backpressure mechanism, and an unbounded one is one of the most common hidden queues. In Java, Executors.newFixedThreadPool uses an unbounded LinkedBlockingQueue, so a slow downstream grows the queue instead of pushing back. A ThreadPoolExecutor with a bounded queue and a rejection handler makes the choice explicit: the default abort policy throws, while CallerRunsPolicy makes the submitting thread run the task itself. That naturally slows the submitter, which is real backpressure, but it is dangerous when the submitter is an event-loop thread that must never block.

A controller that pauses and resumes at the same threshold will oscillate. The moment it pauses, the queue dips below the threshold, so it resumes, and the queue rises over it again. The remedy is hysteresis: pause when depth reaches a high-water mark and resume only when it falls to a lower low-water mark. The gap converts rapid flapping into occasional, longer pauses. Pausing at 80% and resuming at 50% is a common illustration; those numbers are not universal.

HIGH = 0.80 * CAPACITY        LOW = 0.50 * CAPACITY
state = FLOWING

on every enqueue or dequeue:
    depth = queue.size()
    if state == FLOWING and depth >= HIGH:
        state = PAUSED;   signal_upstream(PAUSE)       # pause / withhold credit / reject
    elif state == PAUSED and depth <= LOW:
        state = FLOWING;  signal_upstream(RESUME)      # resume / grant credit

The figure runs the same toy model twice, with 160 items per second arriving while flowing, 100 per second served, and a paused producer sending nothing. With a single threshold at 800 the controller flips state 41 times in 60 seconds. With 800/500 it flips 13 times, each pause lasting several seconds. The numbers belong to the model, but the shape is general.

Placing the marks

Derive both marks from the system rather than copying percentages. The high mark must leave room for the work that is still on its way after the signal is sent: HIGH ≤ CAPACITY − λ_max × T_react, where T_react covers signal propagation, in-flight requests, and producer reaction time. With a 10,000-slot queue, arrivals up to 600 per second, and a three-second reaction time, 1,800 more items can arrive after the signal, so the high mark should be at most 8,200. The low mark must not let consumers run dry while the producer wakes up: LOW ≥ μ × T_resume. At 350 per second and two seconds to resume, anything above about 700 items works, so 50% is generous. Lower marks mean longer pauses and less work waiting; higher marks mean shorter pauses and more queueing delay.

Queue occupancy is only one signal. CPU saturation, downstream latency, database connection exhaustion, and worker concurrency can become the binding limit well before the queue fills. A robust design watches the scarce resource itself (pool utilization, downstream p99, concurrency in use) and does not assume that queue depth captures system health.

An image-transformation service receives 600 jobs per second and processes 600. At 12:01 a storage dependency slows and throughput falls to 350 per second. The backlog now grows by 250 per second. From an empty start, a 10,000-job queue fills in 10,000 ÷ 250 = 40 seconds. The service is configured to start reducing upstream credits at 70% (7,000 jobs) and reject non-critical jobs at 90% (9,000). At constant rates those marks are reached at 28 seconds and 36 seconds.

Whether the loop works depends entirely on the callers. Suppose they honor the signal and the arrival rate falls to 300 per second. The queue now drains at 350 − 300 = 50 jobs per second, so it falls from 7,000 to 2,400 by the 120-second mark and never reaches the 90% mark. Notice how slow that recovery is: the drain rate is the difference between service and arrivals, so a producer that only eases to 340 per second drains at 10 per second and the backlog lingers for minutes. The latency cost is real too. At 7,000 items and 350 per second the newest admitted job waits about 20 seconds, and at the full mark about 28.6.

If callers ignore the signal, or retry rejections immediately, the queue reaches 10,000 at 40 seconds and stays full. From there rejection alone does not prevent upstream overload, because the rejected work returns as retries on top of fresh arrivals. The scenario separates two things that are easy to conflate: the queue's ability to absorb a short spike, and the control loop required to survive a prolonged slowdown. The first is capacity planning; the second is backpressure.

A topic receives 10,000 events per second. Each consumer instance sustains about 1,000 per second (an illustrative figure; measure yours). The group has two instances, so it sustains 2,000 and lag grows by 8,000 per second: 480,000 after a minute, 4.8 million after ten minutes, 28.8 million after an hour. Using FIFO arithmetic, the oldest unread event after ten minutes is 4.8M ÷ 2,000 = 2,400 seconds, 40 minutes, old.

A first fix is to scale out to eight instances. That gives 8,000 per second, which is still below the 10,000 arriving: lag still grows, at 2,000 per second. It is slower growth, not recovery, and a dashboard that only shows the slope flattening can hide that. The group needs more than 10,000 per second of capacity to shrink the backlog at all. With twelve instances (12,000 per second) the backlog drains at 2,000 per second, so 4.8 million events take 2,400 seconds, 40 minutes, to clear. Two constraints apply. The topic needs at least twelve partitions, because a partition is consumed by at most one member of a group at a time. And the consumers must not push the shared database over its own limit, in which case more consumers just make the real bottleneck worse.

The last row is the true backpressure option, and it only exists if the producer is something you can slow: an ingestion API you can rate-limit, a batch job you can pause. Throttling stabilizes the system but does not clear the backlog already queued; that still needs spare consumer capacity or a decision to discard. If the producers are external and you cannot throttle them, the real choices are scaling, shedding, or accepting growing lag within your retention window.

Backpressure propagates latency upstream. In a synchronous request chain, blocking consumes the caller's deadline and holds its threads and connections. Rejection makes overload explicit but requires every caller to handle failure. Dropping preserves system health at the price of lost data. Scaling workers helps when CPU is the bottleneck and can worsen a shared-database bottleneck. Pausing a consumer protects correctness while raising event age and threatening processing SLAs. Large buffers give better burst absorption and hide problems: the system looks healthy until the buffer is full, then fails abruptly. Small buffers expose trouble early and propagate signals sooner, at the cost of rejecting or pausing more often during ordinary bursts.

The most frequent failure is a hidden queue. An application advertises a bounded queue while another layer buffers without limit: an HTTP client that queues requests while its connection pool is exhausted, a thread pool with an unbounded work queue, a connection pool whose waiters are unbounded, a broker with unlimited retention. Stable behavior needs an end-to-end bound on admitted work, not a bound on one queue. A useful audit is to follow one request through the system and ask at each hop where it waits and who limits that wait.

Backpressure also interacts with other protections. Rate limiting caps what each caller may send regardless of downstream state; a circuit breaker stops calls to a dependency that is failing; load shedding chooses what to discard when saturated. Backpressure is the one that carries the downstream's capacity back to the source. Without bounded retries, any of the rejecting forms can be defeated by retry amplification.

The unbounded in-memory queue. A service puts incoming messages on a plain list while its consumer runs several times slower than the producer. Heap usage climbs for minutes, garbage collection pauses lengthen, and the process eventually dies with an out-of-memory error, losing every in-flight item. Giving the queue a capacity and defining the full-queue behavior would have turned a crash into a visible, bounded rejection.

Silent consumer lag. A Kafka consumer falls behind after a deploy that slowed its handler. No alert covers lag, so the backlog grows for hours, users see stale data, and catching up takes longer than the outage itself. Alert on the lag growth rate and on oldest-record age, not only on absolute lag, which is meaningless without knowing throughput.

Scaling only the producers. Traffic rises and the team adds producer-side instances, because that is where the load appears. Consumer throughput is unchanged, so lag grows faster. Scale the stage that is the bottleneck, and measure lag before touching the stage in front of it.

Buffer everything, process later. A large buffer hides every spike for weeks, then a longer slowdown arrives and the backlog represents hours of latency. Users experience a system-wide outage even though nothing crashed. The buffer was a latency budget nobody had set.

No queue visibility. Queue metrics were never exported, dashboards looked green, and the backlog was discovered when the process ran out of memory. Queue depth, age, and in/out rates belong on the primary health dashboard, not in an afterthought.

The slow dependency inside the consumer. Every consumer calls the same slow database or external API, so pipeline throughput collapses to that dependency's speed and the queue in front of it fills. Addressing the queue misses the cause. Use timeouts, pooled and bounded connections, a circuit breaker on the slow call, and bulkheads so one dependency cannot take all the workers.

Measure ingress and completion rates, queue depth and oldest-item age, consumer lag, active in-flight work, time spent blocked, rejected and dropped counts, retry amplification, and end-to-end latency. The ingress versus completion pair is the imbalance itself; every other metric is a consequence.

Prefer age to count. A depth of 1,000 jobs can represent a few milliseconds of work or several hours, depending on throughput; age is roughly depth ÷ service rate for a FIFO queue, and it is what users feel. Alert on its trend, because a queue that is growing and a queue that is large but stable need different responses. Track saturation of the actual scarce resource (CPU, pool usage, downstream latency) alongside the queue, and keep rejected-versus-retried counts so you can tell whether callers are backing off or piling on.

Backpressure is essential wherever a producer and consumer run at independent rates and work accumulates between them: streaming pipelines, message queues and Kafka consumers, reactive code (RxJava, Reactor, Akka Streams build it in as a first-class primitive), and asynchronous request queues in high-throughput APIs, which need size limits and a rejection path. In a plain synchronous request-response call chain there is usually no explicit queue to signal through; there the equivalent protections are timeouts, bounded connection and thread pools, concurrency limits, and circuit breakers, which bound work in a similar way.

These compose. Rate limits cap individual callers at the edge, backpressure carries the consumer's limit through the internal pipeline, load shedding is the last-resort drop when the signal is absent or ignored, and bulkheads keep one workload's backlog from consuming everyone's workers. Backpressure is also not a guarantee of anything: it cannot create capacity, it cannot help if the signal never reaches the true source, and it trades throughput and latency for stability.

• Buffering is not backpressure: an unbounded queue turns a rate mismatch into memory exhaustion and stale work. A 48,000-item backlog at 1,200 items/s means a 40-second wait for every new request.
• Size the queue bound from the wait you can tolerate (N ≈ μ × W_max), keep it below callers' own timeouts, and check memory only as a constraint.
• Decide the full-buffer policy per data type: reject or drop is fine for retryable or non-critical events and dangerous for ledger or audit data.
• Use two thresholds (high and low water marks) so the controller does not flap around a single limit.
• Hidden queues defeat the loop: Executors.newFixedThreadPool has an unbounded queue, and Kafka lag is a symptom rather than a signal to producers, so add an explicit signal or throttle the source.
• Scale the bottleneck stage, alert on lag growth rate and oldest-item age, and combine backpressure with rate limits, load shedding and bulkheads rather than expecting it to create capacity.

Rejection is only half of the story if clients retry badly, so read Retry & Backoff next and then Rate Limiting for admission at the edge. For deciding what to discard when saturated, see Load Shedding; for containing a slow dependency, see Circuit Breaker and Bulkhead. The queues and leases behind most of the examples above are in Queues & Workers and Job Processing, and Idempotency covers making redelivered work safe. For streaming APIs under load, see gRPC.