Bulkhead

fairCallHandlingEnabled is not read by the Spring Boot binder in 2.4.0: the class that binds resilience4j.bulkhead.instances.<name> has no such field, and an unknown key is ignored silently. A run with fairCallHandlingEnabled: false under an instance gave a configuration with fairCallHandlingEnabled=true. Set it with a customizer, shown below. The readme.io Bulkhead page lists neither this property nor writableStackTraceEnabled for the semaphore bulkhead, and its examples use the Vavr Try, which has been optional since Resilience4j 2.0.

Source: Resilience4j: Bulkhead and the v2.4.0 SemaphoreBulkhead source. The comments show the output of the run.

With a non-zero maxWaitDuration a caller that finds the bulkhead full is parked until a permit is released or the time is up:

Source: Resilience4j: Bulkhead. The note in the maxWaitDuration Javadoc of BulkheadConfig recommends 0 for threads that run on an event loop (Netty, Reactor’s parallel scheduler, an RxJava computation scheduler), because blocking one of them costs more throughput than rejecting. The same reasoning applies to servlet request threads: a waiting caller holds one.

If the waiting thread is interrupted, acquirePermission() throws AcquirePermissionCancelledException instead of BulkheadFullException, and the interrupt flag stays set.

Bulkhead.changeConfig(BulkheadConfig) replaces the configuration of a live bulkhead: maxConcurrentCalls is raised by releasing permits to the semaphore, or lowered by acquiring the difference. In a run, raising a bulkhead from 2 to 5 permits showed 5 available and a maximum of 5, and lowering it to 1 showed 1 and 1 with no call in flight. According to the source, lowering uses acquireUninterruptibly, so when callers are inside the bulkhead the call waits for them to finish before it returns. A FixedThreadPoolBulkhead has no changeConfig: its pool is built once.

Spring Boot also reads baseConfig and eventConsumerBufferSize under both prefixes, as described below. rejectedExecutionHandler has no key in Resilience4j 2.4.0: a run with rejectedExecutionHandler: java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy under an instance still produced rejectExecutionHandle=AbortPolicy. Use a ThreadPoolBulkheadConfigCustomizer. The readme.io page lists neither contextPropagators nor rejectedExecutionHandler in its property table.

The executor is a plain ThreadPoolExecutor, so its sizing rule applies, and it surprises people: the pool grows beyond coreThreadPoolSize only when the queue is full. With the default queue of 100 and the default core size, nothing above the core threads ever runs until a hundred calls are waiting. In this run the pool has core 1, maximum 3 and a queue of 1, and every call blocks until released:

Source: the output of the example above, the Resilience4j: Bulkhead page and the ThreadPoolExecutor Javadoc (queuing rules).

The total capacity is therefore maxThreadPoolSize + queueCapacity calls: 4 here, and 6 in the Bookshelf configuration below (four threads and two queue slots). The number of calls that are running is at most maxThreadPoolSize. The queue is not free: a call that waits in it has already used part of its caller’s time budget. A deep queue turns saturation into latency instead of rejections, which is usually the opposite of what a bulkhead is for.

With queueCapacity(0) the executor uses a SynchronousQueue, a call is accepted only if a thread is free or can be created, and getQueueCapacity() and getRemainingQueueCapacity() both return 0. The extra rejected call of that run carried the message Bulkhead 'noqueue' is full and does not permit further calls.

executeSupplier, executeCallable and executeRunnable return a CompletionStage; decorateSupplier, decorateCallable and decorateRunnable return a Supplier<CompletionStage<…>>. A failure inside the task completes the stage exceptionally; get() on the CompletableFuture shows the original exception as the cause (java.lang.IllegalStateException: boom in the run), while join() wraps it in a CompletionException. When the pool rejects the task, the direct methods throw BulkheadFullException synchronously, which is not what a CompletionStage user expects. The Decorators builder of resilience4j-all (withThreadPoolBulkhead) turns the rejection into a failed stage, so that a fallback can handle both outcomes the same way:

Source: Resilience4j: Bulkhead and the v2.4.0 Decorators source. The builder and fallback model is explained in Registries, configuration and decorators.

The executor is created in the constructor and lives until close(), which shuts it down and waits up to five seconds. ThreadPoolBulkheadRegistry implements AutoCloseable and closes every bulkhead it holds, so a registry that you build by hand should be closed. In Spring Boot the registry is a bean and the container does it.

rejectedExecutionHandler accepts any java.util.concurrent.RejectedExecutionHandler. The one that is almost always wrong for a bulkhead is CallerRunsPolicy: it runs the task on the calling thread when the pool is full. In a run with core 1, maximum 1 and queue 1, the third and fourth calls ran on main and completed at once, no BulkheadFullException was thrown, and the bulkhead no longer limited anything: the caller did the dependency’s work itself, which is exactly the thread usage the bulkhead exists to prevent. Keep AbortPolicy, or write a handler that records a metric and then throws.

A BulkheadConfigCustomizer or ThreadPoolBulkheadConfigCustomizer bean receives the builder after the YAML values have been applied, for the instance of that name, so it can set what has no key, and it can pass ContextPropagator instances instead of class names:

Source: Resilience4j: Spring Boot 2 & 3 (customizers) and the v2.4.0 source of BulkheadConfigCustomizer. CountingAbortPolicy is your own RejectedExecutionHandler that increments a counter and then throws RejectedExecutionException. In a run with these customizers the isbn instance had fairCallHandlingEnabled=false, while an instance configured only in YAML kept true, and recommendations reported rejectExecutionHandle=CountingAbortPolicy. The propagator classes from YAML and the instances from the customizer are added together (the run listed MdcContextPropagator and SecurityContextPropagator), so name each propagator in one place only. Beans of type ContextPropagator are not picked up automatically: the 2.4.0 starters read only the class names from YAML and the instances passed to the builder.

A semaphore wait parks a virtual thread and releases its carrier thread. Resilience4j 2.3.0 replaced the synchronized blocks in the core modules with ReentrantLock, which removed the thread pinning that those blocks caused, and the SemaphoreBulkhead of 2.4.0 uses a ReentrantLock for configuration changes and a Semaphore for admission. In a run, 50 virtual threads called a bulkhead with maxConcurrentCalls(5) and maxWaitDuration(10s), each call blocked for 100 ms, and the observed maximum concurrency was 5. The whole batch took 1.08 s, which is the 50 calls in ten waves of five. Every task ran on a virtual thread.

Source: the output of the example above, JEP 444: Virtual Threads and the Resilience4j: Bulkhead page. Here, unlike on an event loop, a non-zero maxWaitDuration is cheap: a parked virtual thread costs almost nothing.

The pool of a FixedThreadPoolBulkhead is a ThreadPoolExecutor with platform threads, and 2.4.0 does not let you supply a thread factory or an executor: the configuration has no such property. In the same run a call made from a virtual thread reported isVirtual() == false inside the bulkhead. The hand-off therefore moves the work from a cheap thread to a scarce one, and the pool size is a hard cap on the platform threads that the dependency can occupy. That is a legitimate goal (a blocking client library that pins carrier threads is a good candidate), but for ordinary blocking I/O on virtual threads the semaphore bulkhead gives the same concurrency ceiling without the hand-off, the queue and the context propagation.

Spring Framework 7 added @ConcurrencyLimit and @EnableResilientMethods in the package org.springframework.resilience.annotation. The annotation was designed with virtual threads in mind: its Javadoc says it is particularly useful where there is no thread pool limit in place. It is a deliberately small feature, next to the @Retryable of the same package. The page Spring Framework native resilience covers the whole package; the facts that matter for choosing are these, taken from the Spring Framework 7.0.9 sources and a run:

Source: the output of the example above, the Spring Framework reference: Resilience Features and the @ConcurrencyLimit Javadoc.

Use @ConcurrencyLimit when all you need is "at most N at a time" on a bean you own, you are on Spring Framework 7, and you do not need to see the bulkhead from outside. Use the Resilience4j semaphore bulkhead when you want the numbers (saturation gauges, rejection events), a bounded wait, a fallback, configuration by environment, or the same limit expressed across Boot 3 and Boot 4. They are not meant to be stacked on one method; if you do, the inner one rejects first and the outer one never sees saturation.

Source: ContextPropagator at v2.4.0 and the Resilience4j: Bulkhead page. In a run with MDC.put("traceId", "abc123") on the caller, a task submitted to a bulkhead without the propagator read null, and with it read abc123. In the Spring Boot run, six requests whose controller set traceId to t-<isbn> showed exactly those values inside the pool threads. A class listed under contextPropagators in YAML is created with its no-argument constructor, so it cannot receive Spring beans: use the customizer for anything that needs injection.

Source: ContextPropagator at v2.4.0. The run used spring-security-core 7.1.1 and a TestingAuthenticationToken, and the pool thread saw the same principal. The propagator installs a new context that holds the Authentication, rather than sharing the caller’s mutable SecurityContext object. The task may outlive the request, so carry only what the work needs, and remember that the authentication stays valid for the duration of that task, not the request.

Micrometer’s context-propagation library can capture every ThreadLocal that has been registered in its ContextRegistry and restore it on another thread, and the Micrometer Observation current-observation accessor is registered there by micrometer-observation itself, through the service loader. A single propagator can therefore bridge the Observation (and the tracing built on it), and any accessor of your own:

Source: Micrometer Context Propagation and the Micrometer Observation reference. In the run, an Observation named bookshelf.request was open on the calling thread: inside a bulkhead configured with this propagator ObservationRegistry.getCurrentObservation() returned it, and inside a bulkhead without the propagator it returned null. A ThreadLocal registered with ContextRegistry.getInstance().registerThreadLocalAccessor("request-id", REQUEST_ID) was carried the same way. This run used the Observation API directly and did not start a tracer, so it shows that the observation is carried across the thread boundary. The Micrometer Tracing span is bound to the current observation in Spring Boot’s tracing support, but check your trace output when you add this in a real service. The tracing setup itself is covered in Observability.

Each bulkhead has an EventPublisher with three event types, defined in BulkheadEvent.Type:

The publisher has onCallPermitted, onCallRejected, onCallFinished and the generic onEvent. In the thread-pool run above (core 1, maximum 3, queue 1) the events were three PERMITTED, two REJECTED, three FINISHED, then one more PERMITTED and its FINISHED once the queued task started. Event objects print as, for example, Bulkhead 'isbn' rejected a call. with the creation time in front.

The Boot starter keeps the last eventConsumerBufferSize events (default 100) of every bulkhead, with no switch to turn it on, and exposes them through the bulkheadevents endpoint. For a thread-pool bulkhead the buffer is registered under the name ThreadPoolBulkhead-<name>, and the endpoint resolves either form.

Source: the output of the example above (most events were removed for brevity), and the Resilience4j: Spring Boot 2 & 3 endpoint section. The endpoint ids are bulkheads and bulkheadevents; both need to be listed in management.endpoints.web.exposure.include. The path accepts /{name} and /{name}/{type} with the event type in lower case. The 2.4.0 release notes also list snapshot reads for the actuator event endpoints on Boot 3 and Boot 4 (issue 2189).

The Micrometer binder registers gauges, tagged with name and any registry or instance tags. A semaphore bulkhead has two of them; a thread-pool bulkhead has seven. Both kinds share the prefix resilience4j.bulkhead:

These are gauges, not counters: Resilience4j does not count permitted or rejected calls as meters. To graph the rejection rate, count CALL_REJECTED events yourself, or count your own fallback invocations. The saturation signals to alert on are available.concurrent.calls near 0 for the semaphore, and queue.depth near queue.capacity together with active.thread.count near max.thread.pool.size for the pool.

Source: the output of the example above (taken while eight requests were in flight), and the Resilience4j: Micrometer page. On the same run /actuator/prometheus contained all seven thread-pool gauges for recommendations and both semaphore gauges for isbn, with the names in the table. The metric names are the constants of ThreadPoolBulkheadMetricNames and BulkheadMetricNames. The meters across all patterns, the Prometheus and Grafana setup and the alert rules are in Observability.