CircuitBreaker

A CircuitBreaker is a state machine with three states that do the work and three special states you switch on deliberately.

Read the diagram from the top. A new breaker is CLOSED: calls flow through and each outcome is recorded in a sliding window. When the failure rate, or the slow-call rate, reaches its threshold — and the window already holds at least minimumNumberOfCalls calls — the breaker becomes OPEN. In OPEN every call is rejected with a CallNotPermittedException until waitDurationInOpenState has elapsed. The next state is HALF_OPEN, which permits exactly permittedNumberOfCallsInHalfOpenState probe calls and evaluates them against the same thresholds: if they look healthy the breaker closes with a fresh, empty window, otherwise it opens again.

Two details are worth knowing before reading on. First, the OPEN to HALF_OPEN move is lazy by default: nothing happens when the wait elapses, and the first call that arrives afterwards moves the breaker to HALF_OPEN and becomes the first probe. Set automaticTransitionFromOpenToHalfOpenEnabled to move on a timer instead (see the tuning example for the trade-off). Second, a call that started while CLOSED and finishes after the breaker opened is still recorded in the window it belongs to; it does not reopen or extend the open period.

Besides the three states above, a breaker can be put in one of three states that are never entered by the thresholds. They are the operator’s tools.

Each state has a method on CircuitBreaker: transitionToClosedState(), transitionToOpenState(), transitionToHalfOpenState(), transitionToMetricsOnlyState(), transitionToDisabledState() and transitionToForcedOpenState(). There are also two variants that open the circuit for a given wait: transitionToOpenStateFor(Duration) and transitionToOpenStateUntil(Instant). And reset() returns the breaker to CLOSED from any state, with an empty window and cleared counters (including the not-permitted count), and publishes a RESET event, plus a STATE_TRANSITION event if the state changed.

Source: Resilience4j: CircuitBreaker (manual state transitions) and CircuitBreaker.java at v2.4.0 (State, StateTransition).

Not every transition is legal. The StateTransition enum in the source lists the allowed ones, and anything else throws IllegalStateTransitionException:

So CLOSED cannot go straight to HALF_OPEN, and METRICS_ONLY cannot go to OPEN or HALF_OPEN. Moving to the same state is legal but not a no-op: the state object is replaced, so the window starts empty, and no event is published.

While CLOSED, the breaker aggregates outcomes in a sliding window, and the rates it compares with the thresholds are computed from that aggregate. There are two window types, chosen with slidingWindowType.

Both implementations are subtract-on-evict aggregations: they never recount the window. A new outcome is added to a running total, and the outcome that falls out of the window (the oldest call, or the oldest bucket) is subtracted. That is why the cost of reading a rate does not depend on the window size, and why a very large slidingWindowSize costs only memory. The minimumNumberOfCalls cap is applied in the three-argument slidingWindow(…​) method; the metrics also apply it at runtime when you set the two properties separately:

Source: Resilience4j: CircuitBreaker (sliding window) and the SlidingTimeWindowMetrics and FixedSizeSlidingWindowMetrics sources at v2.4.0.

A time-based window moves only when something touches it: it is advanced by the next recorded call or the next read. The next snippet shows the buckets expiring on a hand-driven clock:

Source: Resilience4j: CircuitBreaker (time-based sliding window).

The breaker computes two independent rates from the window:

Either rate reaching its threshold opens the circuit (the comparison is greater than or equal), and each can be turned off in practice: the defaults are failureRateThreshold 50 and slowCallRateThreshold 100, and since a rate of 100 needs every call to be slow, the default effectively disables slow-call detection. Lower it to make the breaker react to latency before the dependency starts failing. Both thresholds must be between 1 and 100; the builder throws IllegalArgumentException otherwise, and so does Spring Boot at startup.

Both rates read -1 until the window holds minimumNumberOfCalls calls. That is the guard against opening on three calls of which two happened to fail. getMetrics() therefore returns -1 for getFailureRate() and getSlowCallRate() when there is not enough data, which is not the same as 0.

Source: Resilience4j: CircuitBreaker (slowCallRateThreshold, slowCallDurationThreshold).

Slow calls are about latency the caller would have suffered, so slowCallDurationThreshold should sit above the dependency’s normal tail latency and below the caller’s own timeout. A TimeLimiter that cancels at two seconds hides nothing from the breaker, since the timeout is a failure. Pair them as described in TimeLimiter and Fallbacks & Composition.

The table lists every property of CircuitBreakerConfig as of 2.4.0, its default (the DEFAULT_* constants and field initializers in CircuitBreakerConfig.java), the builder method, and the Spring Boot key. Spring Boot keys live under resilience4j.circuitbreaker.instances.<name>. for one instance and resilience4j.circuitbreaker.configs.<name>. for a shared configuration an instance inherits with baseConfig.

Source: Resilience4j: CircuitBreaker (the property table, which lacks several rows listed here) and, for the verified defaults, CircuitBreakerConfig.java at v2.4.0 and CommonCircuitBreakerConfigurationProperties.java at v2.4.0.

Every default in the first three tables was asserted against a running 2.4.0 configuration. The assertion block is the quickest way to keep your own copy of this table honest:

Source: CircuitBreakerConfig.java at v2.4.0.

When a call throws, the breaker classifies the exception in this order (the logic is in handleThrowable of the state machine):

A call that returns normally is a success unless recordResultPredicate matches the result, in which case it is recorded as a failure (the event carries a ResultRecordedAsFailureException). Exceptions wrapped in a CompletionException or an ExecutionException, as asynchronous code produces, are unwrapped first, so the cause is what gets classified.

Source: Resilience4j: CircuitBreaker (recordExceptions, ignoreExceptions) and the state machine at v2.4.0.

Counting a returned value, for a client that returns an object instead of throwing on a 503:

Source: Resilience4j: CircuitBreaker (recordResultPredicate).

waitDurationInOpenState is a fixed wait. A dependency that stays broken is probed every 60 seconds by default, and every probe round briefly lets traffic reach it. A growing wait makes the breaker patient with a dependency that keeps failing and quick with one that recovers: exponential backoff doubles the wait on each consecutive reopening, and randomized wait spreads probes from many instances so they do not arrive at the same moment.

Source: Resilience4j: CircuitBreaker (waitIntervalFunctionInOpenState) and the IntervalFunction source at v2.4.0.

The attempt passed to the function is the number of consecutive openings: it grows each time a HALF_OPEN breaker opens again and resets when the breaker closes. A breaker that opens, recovers and later opens again starts from the first interval again. A run with a hand-driven clock confirmed it: after the first opening the wait was 10 seconds, after the probe failed it was 20.

Configuring both waitDurationInOpenState and waitIntervalFunctionInOpenState is a bug, and build() says so: The waitIntervalFunction was configured multiple times which could result in an undesired state. In Spring Boot, the same function is chosen with keys next to waitDurationInOpenState:

Source: Resilience4j: Spring Boot 2 & 3 and the CommonCircuitBreakerConfigurationProperties source at v2.4.0.

Enabling both exponential backoff and randomized wait throws IllegalStateException at startup. The wait keys are applied together, from the properties object that sets waitDurationInOpenState: an instance that inherits waitDurationInOpenState, enableExponentialBackoff and the multiplier all from one baseConfig gets the exponential waits (10, 30 and 90 seconds for a multiplier of 3), but an instance that sets only enableExponentialBackoff and inherits the wait from its baseConfig silently keeps the fixed wait, and enableExponentialBackoff without any waitDurationInOpenState leaves the 60-second default; the run showed both. With no multiplier the library default of 1.5 applies (10, 15 and 22.5 seconds). A randomized wait with a factor of 0.25 around 10 seconds produced values between 7 500 and 12 500 ms.

A CircuitBreaker is thread-safe and meant to be shared: one instance per remote dependency, used by every thread. State changes are atomic, the metrics are updated under the window’s lock (or with compare-and-swap), and the open-to-half-open move is guarded so that exactly one thread performs it.

What the sliding window does not do is limit concurrency. The breaker counts and rates the calls; it does not queue them, cap the number in flight, or serialize them. In CLOSED, any number of threads can be inside the protected call at once; the run had 50 virtual threads inside a default breaker simultaneously. To limit concurrent calls, use a Bulkhead (see Bulkhead). The one place the breaker does cap concurrency is HALF_OPEN, where only permittedNumberOfCallsInHalfOpenState probes are granted, even under contention: 40 threads asked at the same moment and exactly 3 were permitted.

Source: Resilience4j: CircuitBreaker and Resilience4j: Bulkhead.

Two consequences. First, a breaker’s counts can overshoot slightly under contention: calls that started while CLOSED and finish after the threshold was crossed are still recorded, so the failure rate you read can be a little different from the one that triggered the transition. Second, a breaker alone does not protect a service from a slow dependency that does not fail: only the slow-call rate and a timeout do, and a Bulkhead caps how many threads a slow dependency can hold.

When the breaker rejects a call it throws CallNotPermittedException, an unchecked exception (a RuntimeException) that says which breaker rejected it. It is thrown in OPEN, FORCED_OPEN and for the excess probes in HALF_OPEN. The message names the state and the breaker, getCausingCircuitBreakerName() returns the name, and there is no cause. It is created without a stack trace when writableStackTraceEnabled is false, which is worth doing: it is thrown in bulk during an outage and a stack trace per rejection is pure overhead.

Source: Resilience4j: CircuitBreaker and the CallNotPermittedException source at v2.4.0.

CallNotPermittedException is not a failure of the dependency: it is the breaker doing its job, and it is not recorded in the window. Handle it where you can answer without the dependency, which is the fallback (see Fallbacks & Composition), and do not retry it: a Retry around the breaker should list it in ignoreExceptions, and the recommended aspect order puts Retry outside the breaker, as explained in Annotations & Aspect Order.

pricing-service is slow and flaky. Under normal load, bookshelf-service makes about 20 calls per second per instance, the 99th percentile latency is 400 ms and a call that exceeds the client timeout takes about 2 seconds to fail. The goal is to stop calling it within seconds of an outage or a slowdown, to answer from the last known price meanwhile, and to come back gradually without a thundering herd. The reasoning goes parameter by parameter.

Source: Resilience4j: Spring Boot 2 & 3 and Resilience4j: CircuitBreaker.

The YAML above was bound in a Spring Boot 4.1.1 test as written, and the resulting configuration was asserted: TIME_BASED, window 10, minimum 20, slow-call threshold 800 ms and 70 %, open waits of 20, 40, 80, 120 and 120 seconds for attempts 1 to 5, automatic transition on, five probes, a 10-second half-open limit, ResourceAccessException recorded and a 404 HttpClientErrorException ignored. The Java side is the annotated client from Getting Started, unchanged: the tuning lives only in the YAML.

Source: Resilience4j: Spring Boot 2 & 3 (@CircuitBreaker and fallbackMethod).