Bulkhead

The problem

A service that calls several dependencies usually pays for all of those calls out of one shared budget: a pool of request threads, a connection pool, a cap on open sockets, or simply memory. While every dependency answers quickly nobody notices the sharing, because each call borrows a little of the budget for a very short time.

The trouble starts when one dependency becomes slow instead of failing. A call that fails hands its thread back at once. A call that hangs keeps it. The number of calls in flight is the arrival rate multiplied by the time each call takes (Little's law), so a dependency that receives 10 calls per second and answers in 0.1 s occupies about one worker, and the same traffic wants 300 workers once an answer takes 30 s. Long before that, the shared pool is full of requests waiting on that one dependency. Every other request queues behind them and times out, including the ones that never touch it. In the animation a recommendations panel, which the page could do without, stops browsing and checkout. Catalog and Payments are healthy the whole time, and nothing is left to call them with.

Google's SRE book describes the same chain as a classic route into a cascading failure: slower responses mean more requests in flight, threads run out, health checks stop being answered, and the load of the server that gave up lands on the ones that remain.

The animation uses 12 slots so that they can be counted. A real pool is larger (a Tomcat connector allows up to 200 request threads by default), which only postpones the moment it is full.

How it works

A bulkhead puts a limit on how much of a shared resource one kind of work may hold at the same time. The name comes from the watertight walls that divide a ship's hull: a breach floods one compartment and the ship stays afloat. Michael Nygard described it as a stability pattern for software in Release It!, next to timeouts and circuit breakers. A slot in the animation stands for whatever is scarce: a worker thread, a pooled connection or a semaphore permit.

In its most common form every dependency gets its own bounded compartment. In the animation the 12 workers become 5 for Catalog, 5 for Payments and 2 for Recommendations. When Recommendations hangs, its two slots fill and stay full. The third call does not wait: it is turned away immediately, the page renders without the panel, and the other ten slots never notice. The failure still happens, but it stays the size of the feature that failed.

Two ways to build a compartment.

Hystrix made the thread pool its default and published what that costs: for one command called 60 times per second, nothing measurable at the median, 3 ms at the 90th percentile and 9 ms at the 99th. It kept semaphores for calls so frequent, or so fast, that this overhead matters. Newer libraries turn the default around: in Resilience4j and in MicroProfile Fault Tolerance a bulkhead is a semaphore unless you ask for a thread pool.

One difference is easy to miss. A thread pool reserves: its threads exist for that dependency and nothing else can use them. A semaphore only caps: its permits are counted against threads that all callers share. Semaphore limits isolate only while they add up to less than the shared pool. If the sum is larger, several slow dependencies together can still fill it.

Sizing. A compartment has to hold the calls that are in flight at peak, plus some headroom:

slots ≈ peak calls per second × latency in seconds + headroom

These are the numbers of the animation, chosen so that the three compartments add up to the original 12. For the latency, take a high percentile of the healthy dependency and not the average. The Hystrix documentation states the rule the same way (the peak request rate of a healthy dependency × its 99th-percentile latency, plus some slack) and gives the reason to stay small: the limit is the thing that sheds load once latency climbs. AWS Lambda's guide estimates the concurrency a function needs with the same product, there with averages.

Then check the result against measurements, because calls arrive in bursts. The 4 in the Catalog row is a pessimistic figure: most calls are much faster than the percentile used, so the average number in flight is well below it. That margin is needed. If calls arrive at random and four are in flight on average, a compartment of five turns away roughly one call in five although nothing is wrong. The compartment's peak usage and its rejection counter under real traffic show whether the margin is enough.

What to do with the excess. A full compartment has three possible answers:

• Reject at once. The caller gets an error immediately (BulkheadFullException in Resilience4j) and decides what to do. This keeps latency bounded and is the right default for interactive traffic.
• Queue briefly. A short maximum wait or a small bounded queue absorbs a burst at the price of some latency. Keep it short: every waiter holds something as well (in a semaphore bulkhead, its own thread), so a long queue rebuilds the original problem one level up. For steady traffic, the SRE book's rule of thumb for the queue in front of a thread pool is half the pool size or less.
• Fall back. This is what the user gets after a rejection: cached or default data, or the page without that panel. Without a fallback, a bulkhead only turns a slow failure into a fast one.

What to partition by. The diagram partitions by dependency, which protects a caller from one slow downstream. The same limit can be keyed differently:

• By tenant or client, so that one customer's burst cannot occupy every worker. The SRE book suggests capping the share of a server's threads that any single client may hold, with a quarter as its example.
• By priority, so that batch or reporting work cannot crowd out checkout. Azure's description of the pattern separates critical consumers from standard ones in this way.
• By endpoint or operation, so that a slow export or search has a small limit of its own and cannot starve the cheap calls.

Pick the key along which failures actually arrive, and keep the number of compartments low enough that each one is still large enough to absorb an ordinary burst.

The same idea at other levels. Slots in a pool are the smallest bulkhead. Larger ones follow the same reasoning:

• Connection pools: one pool per dependency or per workload instead of one for everything.
• Queues: separate queues with their own consumers, so that a backlog of one kind of message doesn't delay the others.
• Processes and containers: each service in its own container with CPU and memory limits. Kubernetes enforces a CPU limit by throttling and a memory limit with out-of-memory kills.
• Node pools: dedicated nodes for one workload or tenant, set apart with taints and tolerations.
• Instances per consumer: separate deployments of the same service for different groups of callers.
• Cells: complete copies of the stack, each serving a fixed subset of customers behind a thin router, so that a bad deployment or a poisonous request stays inside one cell. The AWS Well-Architected Framework lists this as a reliability best practice under the name bulkhead architecture. Shuffle sharding, as described in the Amazon Builders' Library, refines it. With eight workers in four fixed pairs, one bad tenant takes down a quarter of the customers. Give every customer its own pair out of the 28 possible ones, and that tenant fully overlaps with only one customer in 28.
• Zones and regions: the largest compartments a cloud provider offers.

How it differs from its neighbours. A rate limit counts requests per unit of time, and a slow dependency does not change the request rate, so a rate limiter sees nothing wrong. A circuit breaker reacts to outcomes: failures or slow calls have to accumulate before it opens. A bulkhead counts calls in flight, which is exactly the number that grows when latency grows, and it acts on the first call over the limit.

When to use it

• A service calls several dependencies of different importance through shared threads or connections. Give at least the optional ones (recommendations, ratings, ads, analytics) a compartment of their own.
• Several tenants or clients share the same workers, and one of them must not be able to starve the rest.
• Interactive and batch work, or cheap and expensive endpoints, run in the same process.
• A client library blocks and its timeouts are missing or untrusted. Thread-pool isolation is the form that still lets the caller walk away.

Not when:

• There is nothing to separate. If every request needs the same single dependency, a compartment for it protects nobody. Use a timeout and shed load instead.
• The pool is too small to split. Twelve slots in three compartments is an illustration. Dividing a small pool into many compartments leaves each too small for an ordinary burst, and healthy traffic gets rejected.
• The work can wait. If the caller does not need the answer now, put the work on a queue (Queue-Based Load Leveling) and do it later instead of rejecting it.
• Idle capacity is unaffordable, or the extra configuration is not worth it for the risk. Azure's guidance names both as reasons to leave the pattern out.

Trade-offs

• Reserved capacity sits idle. Catalog's five slots cannot help while Payments has a burst. Partitioned capacity has to add up to more than a shared pool would need, and part of it is always unused. Some platforms soften this by lending: the Kubernetes API server lets a busy priority level borrow concurrency that another level is not using.
• More settings, and they age. Every limit is a guess that drifts as traffic and latency change. Too small and healthy calls are rejected, too large and the compartment protects nothing. This is why Netflix moved on from Hystrix's fixed pool sizes to concurrency limits that adapt to measured latency, and why Envoy has an adaptive concurrency filter next to its static limits.
• It limits concurrency, not latency. The two stuck calls are still stuck. Without a timeout the compartment stays full for as long as the dependency is slow, and a queue in front of it adds waiting time of its own.
• Rejections are errors that someone has to handle. Every protected call needs a fallback or an honest error, and callers must not answer a rejection with an immediate retry.
• Limits are per instance. A limit of 2 on 20 instances still allows 40 concurrent calls to the dependency, and scaling the caller out raises that number. A bulkhead protects the caller. Protecting the dependency takes a limit that is enforced where all callers meet: in the dependency itself or in a proxy in front of it.
• Thread pools cost threads and context. More pools mean more threads, more context switches and more places where a trace ID or a security context can get lost.

Implementation notes

• Timeouts come first. Every call inside a compartment needs a timeout that is shorter than its caller's patience (Timeout & Fallback). The timeout decides how quickly a slot comes back, and the bulkhead decides how many slots can be lost in the meantime. Hystrix shipped with a 1-second timeout and pools of 10 threads as its defaults.
• Mind the order of the wrappers. Resilience4j's Spring integration nests them by default as Retry ( CircuitBreaker ( RateLimiter ( TimeLimiter ( Bulkhead ( call ) ) ) ) ). The bulkhead is innermost, so every retry attempt needs a permit again, and a rejection passes through the circuit breaker on its way out. Decide deliberately whether a full compartment counts as a failure there. A Circuit Breaker that opens after repeated timeouts keeps even the two slots from being wasted and probes for recovery. A retry of a rejected call should back off or not happen at all (Retry with Backoff & Jitter). A rate limit on the way in keeps one client from filling a compartment by sheer volume.
• Resilience4j (Java), version 2.4.0 as of October 2026. The semaphore Bulkhead has maxConcurrentCalls (25 by default) and maxWaitDuration (0 by default, which means reject at once). ThreadPoolBulkhead puts a bounded queue (queueCapacity, 100 by default) in front of a fixed pool sized from the number of processors, and can carry thread-local context across with a ContextPropagator. With the annotations the semaphore is the default, and @Bulkhead(type = Bulkhead.Type.THREADPOOL) selects the pool for methods that return a CompletableFuture. Micrometer receives gauges for the available and the maximum number of concurrent calls, and every rejection is published as an event.
• Other libraries. Polly v8 (.NET) replaced its Bulkhead policy with a concurrency limiter, AddConcurrencyLimiter(permitLimit, queueLimit), built on System.Threading.RateLimiting. MicroProfile Fault Tolerance has @Bulkhead: a semaphore on its own, a thread pool with a waiting queue when combined with @Asynchronous. cockatiel offers bulkhead(limit, queue) for Node.js.
• Hystrix is where many teams first met the pattern, and its wiki is still the clearest account of thread pools against semaphores. The library itself has been in maintenance mode since its final release, 1.5.18, in November 2018, and its README says Netflix uses Resilience4j for new projects. Don't start with it.
• In a service proxy. What Envoy calls circuit breaking is a set of bulkheads: caps on connections, pending requests and active requests towards an upstream cluster, kept per cluster and per priority. The defaults are 1,024 each, far more than a small service can afford to have stuck, so set them. Excess requests fail fast with an x-envoy-overloaded header and are counted in upstream_rq_pending_overflow (the pending cap) and, since Envoy 1.38, upstream_rq_active_overflow (the active-request cap, which older versions also counted in the first). The limits are shared between worker threads only approximately. Istio exposes the same settings as connectionPool in a DestinationRule (tcp.maxConnections, http.http1MaxPendingRequests, http.http2MaxRequests), all of them effectively unlimited until you set them. See Service Mesh.
• Event loops and asynchronous runtimes. In Node.js, asyncio, Go or reactive Java a slow dependency blocks no thread, so no pool visibly fills up. The pending calls still hold memory, sockets and a share of the dependency's capacity, and the failure arrives as growing memory, exhausted sockets or timeouts everywhere. The bulkhead becomes a counter: a semaphore (asyncio.Semaphore in Python, golang.org/x/sync/semaphore or a buffered channel in Go) or a connection limit per host (Node's http.Agent allows unlimited sockets per origin until maxSockets is set). The same holds for Java's virtual threads: JEP 444 says not to pool them and to guard a scarce resource with a semaphore instead.
• On the platform. AWS Lambda's reserved concurrency is a bulkhead in both directions: it guarantees a function its share of the account's concurrency and caps it there, at no extra charge. Kubernetes resource limits, dedicated node pools and the API server's own API Priority and Fairness (stable since v1.29) apply the pattern to CPU, memory, nodes and API requests.
• Measure every compartment. Export the slots in use against the limit as gauges, and rejections and timeouts as counters, per compartment. Alert on a compartment that stays full and on any rejection in a critical one. A compartment that never gets near its limit can be made smaller.
• Keep health checks outside. A readiness probe that waits for a slot in a full compartment, or that reports the instance unhealthy because an optional dependency is slow, turns a contained failure back into an outage (Health Endpoint Monitoring).
• Test the failure, not the happy path. Inject latency into one dependency under load and check that the others keep their response times. A bulkhead that has never been full has never been tested.