Observability

Source: the output of the sample run and the CircuitBreakerEndpoint at v2.4.0 (the rates are formatted by the endpoint as strings with a % suffix). A failure rate of -1.0% is not a bug: it means that minimumNumberOfCalls has not been reached yet, so there is no rate to report. After a failing run the same breaker reads:

Source: the output of the sample run (a four-call window; two failures opened the breaker, the other six calls were rejected). notPermittedCalls is the count of calls rejected by the open breaker; the window’s counters otherwise describe only the calls that were allowed through.

Source: the output of the sample run and the CircuitBreakerEvent.Type at v2.4.0. The type segment is case-insensitive (state_transition is converted with toUpperCase()); the types are ERROR, IGNORED_ERROR, SUCCESS, NOT_PERMITTED, STATE_TRANSITION, RESET, FORCED_OPEN, DISABLED, FAILURE_RATE_EXCEEDED and SLOW_CALL_RATE_EXCEEDED. A SUCCESS event looks like the one above with "type": "SUCCESS", a durationInMs and no stateTransition. Only STATE_TRANSITION and RESET are published in every state; the other types are published only while the breaker is CLOSED, OPEN, HALF_OPEN or METRICS_ONLY (CircuitBreakerEvent.Type.forcePublish and CircuitBreaker.State.allowPublish), so a breaker in FORCED_OPEN or DISABLED adds nothing to the buffer except the transitions themselves, as noted on CircuitBreaker. The state transition string is the toString() of the StateTransition enum on Spring Boot 4.1.1 (Jackson 3), as above, while the same request on Spring Boot 3.5.16 (Jackson 2) returned the enum name CLOSED_TO_OPEN; do not parse that field in a script that has to run on both.

Source: the output of the sample run. The types are RETRY (a failed attempt that will be repeated), SUCCESS (the call succeeded after at least one retry), ERROR (it failed and the attempts ran out) and IGNORED_ERROR (an exception in ignoreExceptions), from the RetryEvent.Type at v2.4.0. numberOfAttempts in the SUCCESS event is the number of retries that preceded the success, not the attempt number. The lookup that succeeded at the first attempt published no retry event, which is why only three events appear for two calls.

Source: the output of the sample run (trimmed to two of the four events) and the RateLimiterEvent.Type at v2.4.0. The types are SUCCESSFUL_ACQUIRE, FAILED_ACQUIRE and DRAINED. The isbn limiter in the sample had subscribeForEvents: true; the rate limiter is the only module where the event consumer is opt-in, so without that property the endpoint returns an empty list. The retry sits outside the rate limiter, so every attempt takes a permit: the lookup that needed three attempts and the one that needed one account for the four acquisitions.

Source: the output of the sample run. bulkheads lists the semaphore bulkhead isbn and the thread-pool bulkhead recommendations together. The types, from BulkheadEvent.Type at v2.4.0, are CALL_PERMITTED, CALL_REJECTED and CALL_FINISHED. A CALL_REJECTED event is the only place where the bulkhead records that it said no, because Resilience4j keeps no counter meter for rejections (see the meter tables below).

Source: the output of the sample run. The types are SUCCESS, TIMEOUT and ERROR, from the TimeLimiterEvent.Type at v2.4.0.

The timers endpoints belong to the @Timer annotation and the Timer class of resilience4j-micrometer: a decorator that records how long a call took and whether it failed, with no state machine behind it. It is configured under resilience4j.micrometer.timer.instances.<name> (keys metricNames, onFailureTagResolver and eventConsumerBufferSize), and its meter is described in the section Micrometer meters.

Source: the output of the sample run. The types are START, SUCCESS and FAILURE, and operationDuration is an ISO-8601 duration. The method that threw was annotated @Timer(name = "search"); the annotation is io.github.resilience4j.micrometer.annotation.Timer and the annotation name collides with java.util.Timer and Micrometer’s own io.micrometer.core.annotation.Timed, so import it explicitly.

streamcircuitbreakerevents (and the Hystrix-format hystrixstreamcircuitbreakerevents) are Actuator endpoints that return Flux<ServerSentEvent<String>> with the content type text/event-stream, merged with a one-second heartbeat. They are registered only when Reactor is on the classpath. In the sample application, which runs on Spring MVC, a request to /actuator/streamcircuitbreakerevents/pricing did not produce a stream: it answered 503 without an Accept header and 404 with Accept: text/event-stream. The endpoint was therefore not verified on the servlet stack and the page does not recommend it; the Javadoc in the source also still shows the old path stream-circuitbreaker-events. For a live feed on Spring MVC, use the EventPublisher listeners described in the section Logging the event streams or poll circuitbreakerevents.

Source: the CircuitBreakerServerSideEvent at v2.4.0 and the sample run.

Since Spring Boot 3.4 the access level of an endpoint is set with management.endpoint.<id>.access, which replaces the older enabled property. The values are none, read-only and unrestricted. With read-only, the GET stays and the POST is not registered.

Source: Spring Boot: Endpoints (endpoint access). In the run, with access=read-only, GET /actuator/circuitbreakers returned 200 and the same POST returned 404: the write operation does not exist at all. Keep read-only as the default, and turn it to unrestricted only in a profile that operators use, if at all.

management.server.port (shown earlier) moves every endpoint to a second port, and management.server.address binds it to one interface. In the run the application port answered 404 for /actuator/circuitbreakers and the management port answered 200. The gateway, the ingress and the public load balancer should route only to the application port; reach the management port through a Service that is not exposed, a port-forward, or a network policy that admits only the monitoring namespace.

Even on a private port the write operation should need a role, and a monitoring user (Prometheus, Grafana) should not have it. A Spring Security filter chain that matches the Actuator endpoints can express "everybody authenticated reads, only ops writes". The password comes from the environment and is never in the file.

Source: Spring Boot: Endpoints (securing HTTP endpoints). In the run, with ACTUATOR_PASSWORD supplied as a system property: /actuator/health was reachable without credentials, /actuator/circuitbreakers returned 401 without them, the grafana user got 200 for GET /actuator/circuitbreakers, /actuator/retries and /actuator/prometheus, the same user got 403 for the POST, and the ops user got 200 with "currentState":"FORCED_OPEN". Two details matter. The POST rule has to be written with HttpMethod.POST and a path matcher, because EndpointRequest.to("circuitbreakers") matches the endpoint for every method. And the example uses one shared secret for two users only to stay short; in production give each consumer its own credential, from a secret store, and prefer a token-based scheme (OAuth2 resource server, mutual TLS) over basic authentication. The Prometheus scrape configuration later on this page shows the monitoring user’s password being read from a mounted file.

On Spring Boot 3 the matcher class is org.springframework.boot.actuate.autoconfigure.security.servlet.EndpointRequest; the rest of the example is unchanged.

For the circuit breaker the mapping is in CircuitBreakersHealthIndicator.mapBackendMonitorState. CLOSED is UP. OPEN is the custom status CIRCUIT_OPEN, or DOWN when the instance has allowHealthIndicatorToFail: true. HALF_OPEN is the custom status CIRCUIT_HALF_OPEN, and allowHealthIndicatorToFail does not change it. Every other state (DISABLED, METRICS_ONLY, FORCED_OPEN) is UNKNOWN. The details of each instance are the numbers of the circuitbreakers endpoint. The rate limiter indicator reports UP while a permission is available or nobody is waiting, RATE_LIMITED (or DOWN with allowHealthIndicatorToFail) when no permission is available and the wait would exceed timeoutDuration, and UNKNOWN otherwise; its details are availablePermissions and numberOfWaitingThreads.

Source: the output of the sample run (the pricingStrict details are shortened). The aggregate in the run behaved like this, which is the part that surprises people: with only pricing open, the circuitBreakers component was UP and the service answered 200; when pricingStrict was also opened, the component was DOWN and /actuator/health answered 503; with pricing half-open and pricingStrict open it was still DOWN; and with pricing forced open (UNKNOWN) and pricingStrict open it was DOWN. Spring Boot’s default StatusAggregator orders DOWN, OUT_OF_SERVICE, UP and UNKNOWN and ignores any other status, so the custom statuses CIRCUIT_OPEN and CIRCUIT_HALF_OPEN never make the aggregate worse (a component whose instances are all custom statuses aggregates to UNKNOWN). That is the point of the custom statuses: they let an open breaker be visible in the details without turning the service DOWN, because a service with one broken dependency and a working fallback is degraded, not dead. Use allowHealthIndicatorToFail: true only for a breaker whose dependency the service cannot do its job without.

An indicator that is enabled while the health endpoint is not exposed stops the application from starting. The run with management.endpoints.web.exposure.include set to circuitbreakers only and management.health.circuitbreakers.enabled=true failed with a missing bean of type org.springframework.boot.health.actuate.endpoint.StatusAggregator, because the health endpoint’s configuration, which provides that bean, is conditional on the endpoint being available. Expose health whenever the indicator is enabled.

Spring Boot health groups select which indicators answer a given URL: /actuator/health/<group>. They are the right way to use Resilience4j’s indicators. A custom group gives an operator or a status page the dependency view without touching what the platform probes.

Source: Spring Boot: Health groups. In the run, /actuator/health/dependencies with pricingStrict open answered 503 {"status":"DOWN"} and the readiness group answered 200.

The liveness probe answers one question: "is this process stuck so badly that restarting it is the cure?" An open breaker means that a dependency is unhealthy, and restarting the caller does not fix a dependency. Worse, every replica of bookshelf-service sees the same open breaker at about the same time, so a liveness probe that includes circuitBreakers makes Kubernetes restart the whole deployment, which empties the breaker state, reconnects every pool at once and adds a thundering herd to the outage. In the run, a configuration that added circuitBreakers to the liveness group made /actuator/health/liveness answer 503 as soon as a strict breaker opened. Keep circuitBreakers and rateLimiters out of liveness, leave readiness with readinessState, and add the indicators to readiness only for a dependency that the service genuinely cannot serve without (and then with allowHealthIndicatorToFail), accepting that the instance leaves the load balancer while the breaker is open. The probe mechanics, timings and the Spring Boot group configuration are covered in Health Probes and Lifecycle.

The retry counter counts calls by outcome, not attempts: a call that failed twice and then succeeded adds one to successful_with_retry. failed_without_retry is a call that failed with an exception the retry does not handle, such as one in ignoreExceptions, and failed_with_retry is a call that used all its attempts and gave up.

Both bulkhead kinds share the prefix, and both are gauges with only the name tag; the semaphore bulkhead has two, the thread-pool bulkhead seven.

The failure tag is produced by onFailureTagResolver, a Function<Throwable, String> that is configured as a class name (resilience4j.micrometer.timer.instances.<name>.onFailureTagResolver). The default is the simple class name, so an application that throws exceptions with unbounded class names (generated proxies, for example) creates unbounded tag values.

Source: the /actuator/prometheus response of the sample run through PrometheusMeterRegistry (Micrometer 1.17.1; the series are a selection, and the pricing breaker was closed again by the last write-operation call, which is why state="closed" is 1.0 while pricingStrict is still open). 46.0 available permissions is the isbn limiter’s limitForPeriod of 50 minus the four acquisitions. The HELP line of a Prometheus family is taken from the first meter registered under that name, so resilience4j_circuitbreaker_calls_seconds carries the text "Total number of failed calls" even for the successful series, and resilience4j_retry_calls_total carries the text of its first kind; read the kind tag, not the help text.

Source: the CircuitBreakerMetricNames at v2.4.0. In the run the series was exposed as bookshelf_breaker_state{name="pricing",state="closed"}, the default resilience4j_circuitbreaker_state was gone, and resilience4j_circuitbreaker_calls_seconds_count was untouched, because only the state name had been overridden. Renaming breaks every dashboard and alert that uses the default name, including the official Grafana dashboard, so do it only when a naming convention forces it.

The meters are updated by the instance itself, not by the annotation: a breaker used from Decorators, from a Reactor operator or from a Feign interceptor is measured exactly like an annotated one, as long as the instance comes from the registry. An instance built with CircuitBreaker.ofDefaults("x") outside the registry has no meters at all.

A listener on an instance has to be attached before the first event, and breakers created by name at runtime would be missed by a loop over getAllCircuitBreakers() at start-up. The registry’s own event consumer solves both: a RegistryEventConsumer bean is called each time an instance is added, and the Boot auto-configuration collects every such bean.

Source: Resilience4j: Spring Boot 2 & 3 (event consumer registry) and the RegistryEventConsumer at v2.4.0. The run, with a forced sequence of transitions, logged these lines (message part only):

Source: the output of the sample run. Every transition made through the Actuator write operation is a STATE_TRANSITION event, so this log is also the audit trail of who forced a breaker open and when (pair it with the authenticated user of the request in your access log, because the event itself carries no user). The same pattern works for Retry, RateLimiter, Bulkhead and TimeLimiter with the matching RegistryEventConsumer<…​> type.