Production Tuning & Anti-patterns

The Resilience4j defaults are generic: a count-based window of 100 calls, a failure threshold of 50 percent, a slow-call threshold of 60 seconds, and a maxAttempts of 3. They say nothing about your dependency. Every number below comes from three measurements per backend, taken from your metrics over a representative week: the request rate, the latency percentiles (p50, p99, p99.9) and the baseline error rate while the dependency is healthy.

A Resilience4j instance is a named piece of state: a sliding window, a bucket of permits, a semaphore. The state is what makes it useful, and it is also why instances must not be shared between things that fail differently. Give every dependency its own CircuitBreaker, Bulkhead and Retry name (pricing, isbn, recommendations), not one called backend or default. If recommendations and pricing share a breaker, the outage of the first one fills the window with failures and opens a breaker that then rejects the healthy pricing calls too: the pattern that was supposed to contain the failure spreads it.

Source: Resilience4j: CircuitBreaker and the CircuitBreakerRegistry at v2.4.0. In the test the shared breaker opened after four failures and rejected the next call with CallNotPermittedException, while the separate pricing breaker answered. The granularity can be finer than the service: a dependency that has a fast endpoint and a slow one (a search and a bulk export on the same host) needs one breaker for each, or the slow one dictates the state of the fast one. The reverse also holds: one breaker per method of a client that talks to a single host multiplies state without adding isolation, which is what the OpenFeign integration does by default (see Gateway & OpenFeign).

Names are also a security and capacity matter. The registry creates an unlisted instance on first use, with the default configuration, and the Actuator write operation does the same for any name it is given. A name built from user input (a tenant id, an ISBN) therefore creates unbounded state; keep the names to a fixed list in the YAML. The configuration model is described in Registries, Configuration & Decorators.

A retry is multiplication. If the gateway retries three times, the Bookshelf service retries three times, and its pricing client retries three times, then a dependency that always fails receives 3 x 3 x 3 = 27 calls for every request that entered the system, exactly when it is least able to answer them.

The measure is not a theory. Three nested Retry decorators with three attempts each reached the failing supplier exactly 27 times in the test below, and a single user request with two attempts produced two entries in the breaker’s window.

Source: Resilience4j: Retry. Run on Resilience4j 2.4.0: backendCalls was 27.

The defences, in order of effectiveness:

The retries also inflate the circuit breaker’s statistics, because with Retry outside the breaker every attempt is recorded: the failure rate of a window is a rate of attempts, not of user requests, and the minimumNumberOfCalls of 100 means 50 failed requests when each makes two attempts. Account for it when you read the dashboards.

Every tier that waits needs a timeout, and the timeouts must nest: the call to the dependency must give up before the caller of your service does. If the gateway gives up after 3 seconds while the Bookshelf service is still waiting for its dependency for 5, the service keeps working for a client that is already gone, holds a thread, a connection and a database transaction, and a retrying gateway then adds a second copy of the same work. The work that nobody waits for is pure waste, and at load it is the work that causes the outage.

The budget arithmetic is simple and should be written down for each endpoint. The worst case of one decorated call is the number of attempts times the timeout, plus the waits between them. The calls made one after another add up, and the sum has to stay under the tier above, with room to spare.

Source: the arithmetic of Resilience4j: TimeLimiter and Retry. In the test, worstCase(2, 500 ms, 100 ms) is 1100 ms, worstCase(2, 350 ms, 200 ms) is 900 ms (2000 ms together, under the 2500 ms deadline and the 3000 ms gateway timeout), amplification(3, 3, 3) is 27 and concurrencyAtTimeout(50, 500 ms) is 25. The formula was also checked against a clock: a TimeLimiter of 200 ms inside a Retry of three attempts with a 50 ms wait, around a call that never completes, failed after 733 ms where the formula says 700 ms.

Rules that follow from the nesting:

Source: Resilience4j: TimeLimiter and the resilience4j-circuitbreaker module at v2.4.0. Run: the first breaker stayed CLOSED with no recorded calls while four calls hung; the second one opened after four timeouts.

When demand is higher than capacity, the service has two choices: queue everything and answer everyone late, or refuse early the requests it cannot serve and answer the others on time. Latency grows without bound in the first case, the timeouts above expire, retries arrive, and the service goes down for everybody. Shedding keeps the service in the second case, and three Resilience4j patterns are its tools:

The rejection must be turned into an answer the client can act on: 503 Service Unavailable with a Retry-After header, or 429 Too Many Requests when a particular client is over its quota. A @RestControllerAdvice does it in one place:

Source: RFC 9110: HTTP Semantics (503 and Retry-After) and the Spring Framework: Error Responses. Run on Spring Boot 4.1.1 with all 25 permits of the pricing bulkhead taken: GET /books/1/price answered 503 with Retry-After: 2 and the problem detail The service is shedding load, retry later; after the permits were released the same request answered 200.

Shedding only works if the clients cooperate. A client that retries a 503 immediately turns the shedding into a retry storm, which is why the Retry-After header and the backoff with jitter of the previous sections are part of the same design. Also keep the container limits in line: the maxConcurrentCalls of one bulkhead has to be smaller than the thread pool of the server (server.tomcat.threads.max), or the bulkhead never fills and the threads run out first; with virtual threads there is no such limit, and the bulkhead is the only thing that bounds the concurrency of a dependency (see Bulkhead). The Spring Framework 7 @ConcurrencyLimit is the same idea without events or metrics.

A fallback turns an exception into a value, and from that moment the caller, the retry and the circuit breaker outside the fallback see a success. A fallback that returns an empty list or a stale price without a trace makes an outage invisible: no errors in the dashboards, no alerts, and a business that discovers days later that it has been showing wrong prices. Four rules keep a fallback honest:

Source: Resilience4j: Spring Boot 2 & 3 (fallback methods) and Fallbacks & Composition. Run: with the backend failing, the call answered {"isbn":"…​","cents":0,"degraded":true}, the counter bookshelf.pricing.degraded increased and the warning was logged; with the bulkhead full, the same method did not degrade and the caller got the 503 above. The fallback is declared on the outermost annotation (@Retry) so that the retries and the breaker see the real failures; a fallback on the @CircuitBreaker makes the retry see a success and never retry.

The same patterns exist outside the JVM. A gateway (Spring Cloud Gateway, Envoy, an ingress) can time out, retry, rate-limit and open a circuit for a route, and a service mesh such as Istio does it for every call between pods with sidecar proxies and no code change. The table in Resilience Fundamentals compares them; for production the decision comes down to what each location knows:

The practical rule is one owner for each concern. Let the mesh or the gateway own the uniform ones (connection limits, outlier ejection, the outer timeout), keep the semantic ones in code (the fallback, the exception classification, the bulkhead around a specific client), and make sure that only one of them retries, because otherwise the multiplication of the retry section returns with two layers of owners that do not know about each other. The gateway side is covered in Gateway & OpenFeign. How the platform reports a service as unhealthy, and what it does about it, is the subject of Health Probes & Lifecycle, and the mesh layer is introduced in Cluster Networking, CNI and Service Mesh.

Sometimes the right move is to open the breaker yourself: the pricing database is being migrated, a third party announced maintenance, or an incident needs the traffic to stop before the failure rate says so. A forced-open breaker rejects every call immediately, whatever the dependency does, and the callers fall back or answer 503. Resilience4j exposes the transition over Actuator, and the endpoint is a write operation on production behaviour, so the procedure is: secured requests only, on the management port, from an operator account with the OPS role defined in Observability. The password comes from the environment of the operator’s shell or a secret store; it is never typed into the command, a script or a ticket.

An open breaker means "this dependency is failing", not "this process is broken". Wiring that signal to the liveness probe makes Kubernetes kill every healthy pod during a dependency outage, which restarts them into the same outage and adds a cold start to it. The Resilience4j health indicator is not a liveness signal; its own default makes that easy to get right, because the default StatusAggregator ignores the custom CIRCUIT_OPEN status (see Observability). The way to get it wrong is to include the indicator in the liveness group:

Source: Spring Boot: Kubernetes probes and Kubernetes: Liveness, Readiness and Startup Probes. Run: with allowHealthIndicatorToFail: true on the pricing breaker and the breaker open, this configuration answered 503 on /actuator/health/liveness. With the default groups (liveness is livenessState only) the same open breaker left /actuator/health/liveness and /actuator/health/readiness at 200, while /actuator/health answered 503 because of allowHealthIndicatorToFail. Leave liveness to the process itself, let the breaker degrade the behaviour and not the probe, and, if a dependency is genuinely required to serve any traffic, consider it in readiness only, with the consequences described in Health Probes & Lifecycle.