Bulkhead

The bulkhead pattern splits a service's finite resources into separate pools, so that one slow dependency or runaway workload can exhaust only its own pool and not the capacity every other workload needs. It does not make the slow dependency faster or prevent the original failure; it limits the blast radius. The compartments can be thread pools, concurrency permits, database connections, worker queues, memory budgets, or whole deployments. It is most often applied in microservice architectures, where one service calls many dependencies from a shared set of threads and connections.

The problem is that shared capacity makes failures contagious. A request handler holds a thread, a connection, and some memory for as long as its downstream call takes. If the call is fast, the pool turns over quickly and sharing is efficient. If the call becomes slow, each request holds its slot far longer, the pool drains, and every other request that needs a slot, including requests for perfectly healthy code paths, now waits behind it. The incident report says "checkout is down" while the root cause is "recommendations got slow." The same dynamic is described from the whole-system side in Cascading Failure; the bulkhead is one of the tools that stops it.

The name comes from ship design: watertight partitions inside the hull mean that flooding one compartment does not sink the vessel. The mental model for software is a budget with a hard ceiling per workload. Instead of one pool of 100 slots that anyone may take, payments is guaranteed up to 60, customer queries up to 20, and reporting up to 20. When reporting stalls, it fills its 20 and starts rejecting; the other 80 slots never see it. That ceiling is also the price: capacity reserved for a workload that is currently idle is capacity nobody else can use.

Take a service with a shared pool of 100 worker slots handling payment authorization, order lookup, and a monthly-report endpoint. A report query regresses and now takes 60 seconds instead of 2. If reports arrive at 5 per second, stalled reports occupy slots at 5 per second and, because they are not finishing, the pool is full after roughly 100 ÷ 5 = 20 seconds. From then on a payment request that takes 40 ms of actual work waits for a slot that will only free up when some report ends. The payment code is healthy and the payment dependency is healthy; the payments are failing because of what they share.

Partition the same 100 slots as 60 payments, 20 customer queries, and 20 reporting. Now the stalled reports fill their 20 slots after 20 ÷ 5 = 4 seconds, and every further report is rejected or queued in reporting's own queue. Payments and customer queries keep their 80 slots. Reporting is down, which is the correct outcome: the failure was in reporting, and the incident stays there.

The cost: statistical multiplexing

The figure also shows the price. In the isolated layout, 56 of 100 slots sit idle because nothing needs them right now. A shared pool serves any mix of demand from the same slots, so it needs less total capacity for the same average load; this is statistical multiplexing, and it is why sharing is the default. Bulkheads trade some of that utilization for predictable failure isolation. Whether the trade is worth it depends on how bad it is when the shared pool drains and how much idle reserve you can afford. For a revenue path competing with a batch job, the answer is usually yes. For ten near-identical internal endpoints with the same criticality, it usually is not.

The 60/20/20 split is an illustration, not a recommendation. Derive real sizes from each workload's peak concurrency, latency distribution, and business priority (see Sizing a Bulkhead). Equal splits rarely match real demand.

Minimum guarantees with controlled borrowing

To recover some of the lost utilization, a design can give each class a reserved minimum and let it borrow surplus capacity. The risk is that a borrower holds slots another class suddenly needs. The safe rule is to lend only capacity that is surplus after every other class's unfilled reservation is set aside. With capacity C, per-class usage used, and reservations reserved:

admit(class c):
  if used[c] < reserved[c]:
      return true                       // inside own guarantee
  free      = C - sum(used)
  protected = sum over j != c of max(0, reserved[j] - used[j])
  return free > protected               // borrow only true surplus

Because slots that other classes could still claim are never lent out, a class can always reach its minimum without having to evict a running task. That matters because running work generally cannot be revoked mid-flight. The guarantee is only as good as the sum of reservations staying at or below C, and borrowed capacity is still subject to the same timeouts and queue limits as any other.

A bulkhead is a counter with a policy for what happens when the counter is at zero. Every protected call goes through three steps: acquire a permit (or fail), run the work while holding it, and release it in a finally-style path that also covers errors, timeouts, and cancellation. The interesting design decisions are all in the acquire step, because there are three possible outcomes: admit immediately, wait in a bounded queue for a limited time, or reject at once.

class BulkheadFullError extends Error {}

class Bulkhead {
  private active = 0;
  private waiters: Array<() => void> = [];

  constructor(
    private readonly maxConcurrent: number,
    private readonly maxQueue: number,
    private readonly maxWaitMs: number,
  ) {}

  async run<T>(fn: () => Promise<T>): Promise<T> {
    await this.acquire();
    try {
      return await fn();
    } finally {
      this.release();
    }
  }

  private acquire(): Promise<void> {
    if (this.active < this.maxConcurrent) {
      this.active++;
      return Promise.resolve();
    }
    if (this.waiters.length >= this.maxQueue) {
      return Promise.reject(new BulkheadFullError('queue full'));
    }
    return new Promise<void>((resolve, reject) => {
      const waiter = () => {
        clearTimeout(timer);
        resolve();
      };
      const timer = setTimeout(() => {
        const i = this.waiters.indexOf(waiter);
        if (i >= 0) this.waiters.splice(i, 1);
        reject(new BulkheadFullError('wait timeout'));
      }, this.maxWaitMs);
      this.waiters.push(waiter);
    });
  }

  private release(): void {
    const next = this.waiters.shift();
    if (next) next();       // hand the permit over; active stays the same
    else this.active--;
  }
}

const recs = new Bulkhead(20, 0, 0);   // no queue: reject immediately when full

Two details matter. The permit is handed directly to the longest-waiting caller on release, so active never dips below the limit while someone is queued, and a waiter that times out removes itself so it does not later consume a permit nobody wants. Also note what this class does not do: when a wait times out, it stops waiting, but it cannot stop a call that is already running. The running call needs its own timeout, covered in Queues, Deadlines, and Cancellation.

What the caller sees on rejection

A rejection must be a fast, explicit, typed signal so the caller can choose a fallback instead of timing out. For an optional dependency the fallback is to omit that part of the response. For a critical path it might be an explicit "try again" state. If the rejection crosses an HTTP boundary to an external client, 503 Service Unavailable, optionally with Retry-After, describes "this instance cannot take more work right now". 429 Too Many Requests is a per-client policy response from a rate limiter, not a capacity one. Internal rejections usually never leave the service.

Libraries make the same three outcomes configurable. Resilience4j's semaphore bulkhead has maxConcurrentCalls and maxWaitDuration, and a full bulkhead throws BulkheadFullException; leaving the wait duration at zero means "do not wait at all". Its thread-pool variant adds coreThreadPoolSize, maxThreadPoolSize, and queueCapacity. Envoy expresses the same idea per upstream cluster as limits on connections, pending requests, and concurrent requests, and counts overflows.

Limits are per process

A bulkhead lives in one process. With 12 replicas each allowing 20 concurrent calls to a dependency, the dependency can see up to 240 concurrent calls, and autoscaling raises that number exactly when load is high. Fleet-wide concurrency caps need either a shared limiter, a limit at the dependency or its proxy, or per-replica limits derived from the maximum replica count.

A semaphore bulkhead limits how many calls may be inside the protected region at once. The caller runs the work on its own thread or event-loop task; no new execution resource is created. That is cheap and a natural fit for asynchronous I/O. The weakness is that it limits concurrency but does not detach the caller: with a blocking client, a call that hangs still ties up the calling thread, and the caller can only escape through the client's own timeout.

A thread-pool bulkhead sends the work to a dedicated executor with a fixed number of threads and, ideally, a bounded queue. The caller waits on a future with a timeout and can walk away while the hung call stays contained in the executor. That is stronger execution isolation for blocking code, and it is what Hystrix popularised with one pool per dependency. The costs are real: a thread hand-off per call, queueing delay, thread memory, and the need to propagate tracing, logging, and security context across the hand-off. And a timed-out future usually does not stop the work: cancellation is a request to interrupt, and a blocking socket read is stopped only by the socket timeout, so the worker thread stays occupied until the call really ends.

Where threads are cheap, as with async runtimes, Go goroutines, or Java virtual threads, the thread is no longer the scarce resource; the downstream connection or the downstream capacity is. In that world the semaphore is the right bulkhead precisely because it caps the thing that is actually scarce.

Connection-pool isolation

The most common production bulkhead is a separate database connection pool per workload, for example one for transactional requests and one for analytical queries. The semantics are those of a semaphore: the pool size is the permit count, and acquisition timeout is the maximum wait (HikariCP's connectionTimeout defaults to 30 seconds, far longer than most request deadlines, so it should be set deliberately). Pool sizes are summed across replicas: 10 replicas with a checkout pool of 20 and a reporting pool of 10 can open 300 connections, which must stay below the database's own connection limit with room for administration and other clients. For why simply enlarging a pool often makes things worse, see DB Connection Pool Exhaustion.

Suppose a reporting bulkhead allows 10 concurrent tasks and a queue of 20. When 31 reports arrive, 10 run, 20 wait, and the 31st is rejected under the overflow policy. The queue bound does its job: memory cannot grow without limit and the rejection is immediate. What it does not bound is how long the queued work waits. With a service time S, N permits and queue position k (counting from 1), a task starts after about ceil(k / N) service times when all tasks are equally long and start together, so the last task in a queue of Q waits roughly ceil(Q / N) · S.

With S = 30 s, N = 10, Q = 20, the last queued report waits about 60 s and finishes at 90 s. If the caller's deadline is 45 s, every queued task completes after its caller is gone, and the queue is full of work nobody is waiting for. A queue helps only for bursts that are short compared with the deadline. A useful queue length for deadline D is about N · floor((D − S) / S), which here is zero: a no-queue, fail-fast bulkhead is the right design.

Make waiting and running both bounded

• Wait timeout. A task may sit in the queue for at most a short, explicit time. After that, reject. Without it, the queue converts overload into latency and hides it.
• Execution timeout. The call itself needs a client-level timeout (connect and read), because that is what ultimately releases the permit. The bulkhead's guarantee is only as good as the longest time a permit can be held: with retries inside the protected region it is attempts × timeout + backoff sleeps.
• Deadline propagation. If the caller's remaining time is already smaller than the expected service time, rejecting immediately is better than starting work that cannot finish usefully.
• Cancellation. When a caller disconnects or times out, remove its task from the queue and interrupt running work where the runtime allows it. Otherwise dedicated pools keep working on obsolete requests.

Cancellation can arrive late: the downstream call may complete after the caller gave up. For operations with side effects this means a rejected or timed-out request may still have taken effect, so such operations need to be safe to repeat; see Idempotency. For the decoupled alternative, where work is queued durably and executed by workers at a controlled rate, see Queues & Workers.

The number of calls in flight in a stable system equals the arrival rate times the average time each call takes: L = λ · W (Little's law). That gives the sizing recipe. Measure peak arrival rate λ for the workload and its healthy latency W (use a high percentile if variance is large), compute L, and add headroom for bursts: N ≈ ceil(L · headroom).

The inverse is what makes the bulkhead protective. A pool of N permits can sustain at most N / W requests per second, so when latency rises, throughput through the compartment falls automatically and the rest is rejected or queued. Take a payments pool of 40 permits and a steady 120 req/s. At a healthy 250 ms, demand is 120 × 0.25 = 30 calls in flight, comfortably inside 40. If the provider degrades to 3 s, demand would be 360 calls, but the pool carries 40 / 3 ≈ 13.3 req/s and the other 106.7 req/s are refused. At most 40 threads or connections are ever held by this dependency; the rest of the service is untouched.

Three consequences follow. First, size for the healthy peak plus margin, not for the degraded case; a pool big enough to absorb a ten-fold latency increase has no protective effect. Second, the pool being too small looks exactly like the dependency being slow: requests are refused during ordinary traffic peaks, and without per-pool metrics nobody can tell which it is. Third, if the pool is large, the bulkhead is not the limit that matters; the downstream's own capacity (its connection limit, CPU, or rate allowance) is, and the sum of all callers' pools must respect it.

Allocate by criticality, not symmetry. The path that earns revenue gets headroom above its measured peak; optional enrichments get a conservative cap because refusing them is cheap. Revisit sizes after load tests and after every significant latency or traffic change, since the numbers encode both.

Partition by workload when tasks differ in criticality or latency profile: checkout versus reports. Partition by dependency when one external service is unreliable or slow, with one pool per downstream. Partition by tenant when a few heavy customers could starve the others; a pool per tenant does not scale to thousands of tenants, so tenants are usually grouped into tiers or hashed onto shards, and deeper designs run cells (independent replicas of the stack that serve a subset of customers). Partition by region or deployment when infrastructure failures must be contained. These boundaries overlap, and every extra pool adds configuration, metrics, and a way to size something wrong.

Isolation at one layer is not isolation at every layer

A poor boundary creates the illusion of safety. Separate application thread pools that all contend on one database lock do not prevent lock contention. Separate connection pools that hit the same storage I/O limit do not isolate storage saturation. In the figure below the application pools are doing their job, and payments still slows down, because a reporting query that saturates the database's disk I/O is paid for by everyone using that database.

So before choosing a compartment, trace the actual bottleneck: what resource does the noisy workload exhaust first, and who else needs it? The fix may belong one layer down: a read replica for reporting, query timeouts and resource governors in the database, separate storage for analytics, or moving heavy jobs to a different deployment. An application bulkhead is still worthwhile in that picture because it stops threads and connections from piling up, but it should not be reported as having isolated the database.

A bulkhead answers "how much of our capacity may this dependency occupy?" A timeout answers "how long may each attempt wait?" A circuit breaker answers "should we keep trying given recent outcomes?" A rate limiter answers "how much new work may enter over time?" They guard different failure modes and are complementary, not substitutes: a bulkhead alone still lets each of its permits be held for the full timeout, and a breaker alone does nothing during the slow period before it trips.

For a slow recommendation service: a 20-permit bulkhead caps the resources it can consume, a 300 ms timeout bounds each call, and a breaker opens if slow calls persist. Because recommendations are optional, the page renders without them in each failure case. The figure walks through four requests.

• Breaker outside the bulkhead is a common choice: an open breaker short-circuits before a permit is taken, so a known-bad dependency stops costing capacity at all. Other orderings are legitimate; what matters is deciding deliberately, and that rejections by the bulkhead are generally not counted as dependency failures by the breaker, since the dependency was never called.
• Do not retry a bulkhead rejection immediately. The rejection means the compartment is full; instant retries raise the arrival rate on the thing that is already saturated. Retry at most with backoff and jitter and a retry budget, or better, fall back.
• The fallback must not draw on the exhausted pool. A fallback that calls another service through the same bulkhead only moves the queue. Prefer static defaults, cached values, or a cheap local computation.
• Upstream admission control still matters. When the whole service is saturated, per-pool limits are too late and too local; load shedding decides what to refuse at the front door, and backpressure tells producers to slow down in queue-based pipelines.

Too-small critical pool. Checkout is capped at 2 connections while steady state needs 5. The bulkhead, not the downstream, throttles the revenue path, and the symptom looks like downstream slowness. Isolation is working; the sizing is wrong. Use realistic peak concurrency and latency distributions.

Shared downstream bottleneck. Isolated application pools still compete for one saturated database or one lock. Consider database workload isolation, query tuning, or separate read infrastructure.

The queue hides overload. An executor accepts thousands of tasks, tail latency grows to minutes, and there are no errors until memory runs out. Bound the queue, bound the wait, and monitor queue age.

Retry after rejection. Clients retry rejected calls immediately, raising the arrival rate. Combine rejection with backoff, a retry budget, or upstream admission control.

No cancellation or timeouts. Timed-out callers leave tasks running in dedicated pools, and permits stay held until the TCP stack gives up. Set client-level timeouts, propagate cancellation where safe, and make side-effecting operations correct under late cancellation.

Over-isolation. Ten pools of 2 connections instead of two pools of 10: most pools sit near idle while others reject, and operational complexity rises with each pool. Group by criticality and failure domain, not by feature.

Pools that add up to more than the downstream allows. Per-replica pools multiplied by replica count exceed the database or API limit. Under load the dependency fails regardless of the neat partitioning on each replica.

Leaked permits. A code path that acquires without releasing on every exit, such as an exception, a timeout branch, or a cancellation, slowly shrinks the pool until the compartment refuses everything. Always release in a finally-equivalent and alert on active permits that never return to baseline.

No pool metrics. Without per-pool utilization, queue depth, and rejections, exhaustion looks like downstream slowness and the root cause stays hidden.

Per pool, track active calls and available permits, queued tasks, rejections per second, queue waiting time, execution latency, timeout rate, and cancellations. Saturation shows up first as permits near zero and queue age rising, before error rates move. A pool that is always near 100% is mis-sized or masking a slow dependency; a pool that never rises above 10% is stranding capacity.

Alert on queue depth and rejection rate, not just error rate, and distinguish bulkhead rejections from downstream failures in the error taxonomy so that rejections do not count toward breaker thresholds or dependency SLOs. Also watch the layer below: database CPU and I/O, lock waits, and total connections across all replicas, because that is where isolation silently stops.

The most important check is the outcome. During a reporting incident, do payment latency and success rate stay within their objectives? That matters more than having separate executor names on a dashboard. Test it deliberately: inject latency into one dependency in a load test or game day and confirm that unrelated workloads do not move.

An e-commerce service has a single connection pool of 50 against a database that allows 100 connections in total. Checkout runs about 200 req/s with a 40 ms database call, so average demand is 200 × 0.04 = 8 connections in flight, 16 at a 2× peak. At 09:00 a reporting scheduler starts 50 concurrent queries that each run about 20 seconds.

Before. The reports take all 50 connections. A checkout request that needs a connection waits until one is returned, and none returns for roughly 20 seconds. Waiters stack up against the pool's acquisition timeout, typically longer than the checkout deadline, so callers time out first. Checkout latency goes from tens of milliseconds to seconds and then to errors; payments fail although the payment provider is healthy. (The exact figures depend on timeouts and queueing; the point is the shape.)

After. The same replica gets a checkout pool of 20 (16 at peak plus headroom) and a reporting pool of 10, 30 connections total. With 20 replicas this would be 600 connections, so the fleet-wide sum is checked against the database limit before the numbers are committed; the example assumes a single replica and leaves 70 of 100 connections for other clients and administration. At 09:00 reporting runs 10 queries and its own queue or rejection policy handles the other 40; scheduled reports can simply wait or be skipped. Checkout's 20 connections are never taken by reports, so its latency stays near its normal level. The remaining risk is the shared database: ten heavy scans can still consume disk I/O, so reporting is also pointed at a read replica and given a statement timeout.

The incident scope shrinks from "checkout is down" to "reports are slow." The bulkhead did not make the reports fast or remove the load from the database; it decided who pays for them.

In-process libraries. Hystrix, from Netflix, made one thread pool per dependency famous. It is in maintenance mode, and Resilience4j is the commonly used JVM successor with both semaphore and thread-pool bulkheads. Other ecosystems provide the same primitive as a semaphore or limited channel (Polly in .NET, a buffered channel or x/sync/semaphore in Go).

Proxies and meshes. Envoy's per-cluster limits on connections, pending requests, and concurrent requests are bulkheads at the network layer; an overflow results in a fast failure instead of an unbounded queue, and they work for services in any language.

Containers and orchestration. Kubernetes resource requests and limits, and namespace quotas, apply compartments to CPU and memory per pod so a noisy workload cannot take a node's capacity. They are not concurrency caps: a CPU limit throttles the container, which can itself add latency, and a memory limit ends in an OOM kill. Separate deployments or node pools for batch and online workloads are the coarser version of the same boundary.

Multi-tenant platforms. Per-tenant quotas, shards, and cells limit how many tenants a noisy neighbour can affect. Contained does not mean perfectly fair: tenants that share a shard still share its capacity.

Databases. Separate pools, users, or replicas for transactional and analytical traffic keep long analytics queries from holding every connection that checkout needs.

The pattern pays off when workloads share finite resources and differ in importance or in risk. It is largely a distraction when they do not. The decision table lists the common cases.

The same trade applies at every level: strong isolation gives excellent containment at the cost of stranded capacity; no isolation gives the best utilization and the largest blast radius; pools that are too small make the bulkhead the bottleneck, and pools too large stop protecting anything. The right point is found by load testing against measured peaks, not by intuition. And, as with the other core patterns, a bulkhead protects availability of the unaffected workloads; it does not preserve correctness of the affected ones. Rejected or timed-out work still needs a defined business outcome.

• A bulkhead trades statistical multiplexing for predictable isolation. It pays off for a revenue path competing with batch work, and rarely for near-identical endpoints of equal criticality.
• Size each compartment with Little's law: N ≈ ceil(λ × W × headroom). A pool of N permits carries at most N / W requests per second, so rising latency automatically sheds load (40 permits at 3 s sustain about 13 req/s).
• Every call needs three outcomes: admitted, queued within a bounded wait, or rejected fast. Bound both the wait and the run time, and propagate deadlines and cancellation.
• Semaphore isolation caps concurrency cheaply; thread-pool isolation adds timeout and thread isolation at a real cost, and limits are per process, not per cluster.
• Partition along the failure you actually fear and check what is shared underneath, because isolation at one layer is not isolation at every layer (shared database, connection pool, node).
• Bulkheads complement, not replace, timeouts, circuit breakers and retries. Monitor rejections, queue wait and saturation per compartment to confirm isolation held.

A bulkhead caps how much a dependency can take, but it needs partners. The Circuit Breaker lesson covers when to stop calling a dependency that is demonstrably unhealthy, and Retry & Backoff covers how not to turn a rejection into a retry storm. At the front door, Load Shedding and Rate Limiting decide what enters the system, while Backpressure governs flow in queue-based pipelines.

For the incident view, read Cascading Failure and DB Connection Pool Exhaustion. For safe handling of work that may be repeated or cancelled late, see Idempotency, and for decoupling work submission from execution, Queues & Workers.