Bucket4j: Distributed Rate Limiting

The Bookshelf service calls an ISBN metadata provider that allows 100 calls per day for the whole account. With one instance, a local bucket of 100 enforces that. With three replicas, each replica has its own bucket, the three buckets are independent, and the provider sees up to 300 calls. The same arithmetic applies to every per-client limit: a client that is allowed 20 requests per minute gets 60 when the load balancer spreads its requests over three replicas, and the ratio changes whenever autoscaling changes the replica count.

Running both variants against a real Redis shows the difference. Three simulated replicas each try 150 times against a daily quota of 100:

Source: Bucket4j reference: Distributed features and Distributed concepts.

A shared limit has a price: every consumption becomes a network call, the store becomes a dependency of the request path, and the clocks of several machines now take part. The rest of the page is about managing that price. When the limit only needs to be approximately right, or when each replica can be given a fixed share of the quota, a local bucket is simpler and faster; the Resilience4j RateLimiter is the in-process alternative.

A ProxyManager<K> is built once per store (one Redis database, one JDBC table, one grid cache) by a backend-specific builder such as Bucket4jLettuce.casBasedBuilder(…​). Buckets inside it are told apart by a key of type K. Once built, every backend is used the same way.

A BucketProxy extends Bucket: tryConsume, tryConsumeAndReturnRemaining, addTokens, replaceConfiguration, asBlocking() and the rest work as for a local bucket. It adds toListenable(BucketListener) and getOptimizationController(). A proxy holds no bucket state, so building one per request is fine and keeps no memory on the client; the exception is a proxy created with an optimization, which is stateful and must be kept (see optimizations below).

Against Redis, isAsyncModeSupported() and isExpireAfterWriteSupported() both return true; against the PostgreSQL select-for-update manager async is false and expiry is supported. Source: Bucket4j reference: ProxyManager and BucketProxy.

proxyManager.builder() returns a RemoteBucketBuilder<K>. Everything it sets applies to the proxies it builds, not to the stored bucket.

Several replicas can consume from the same bucket at the same instant, so the read-modify-write of the bucket state must be atomic. Bucket4j does not pick one mechanism; each backend module uses what its store offers.

The settings that control how a replica talks to the store are methods of the backend builder; they live in ClientSideConfig and are shared by every proxy the manager hands out. The reference documents them through withClientSideConfig(…​), which does not exist on the 8.21.0 builders (it survives only on the deprecated SQLProxyConfigurationBuilder), so the builder methods below are the way to set them.

The key is the only thing that distinguishes one limit from another, and it decides memory, contention and fairness.

Prefix keys with the kind of limit, so that a key space shared with other uses of the same Redis (see Redis use cases and patterns) cannot collide. For Redis, withMapper lets the application work with String keys while the manager is keyed by byte[]. A hot key (one bucket hit by every request) serializes on the store; the remedies are batching, splitting the bucket per replica with a share of the quota, or moving that limit to the edge. Bucket4j keeps nothing per bucket on the client, so cardinality only costs memory in the store.

A bucket stores its configuration with its state. Passing a different supplier to getProxy for an existing bucket changes nothing, so editing a plan in a YAML file and restarting the service leaves every stored bucket on the old limits. Replacing configurations explicitly would mean enumerating the buckets, which is expensive in Redis, where there is no table to scan.

Implicit configuration replacement solves this with a version number. The proxy is built with a desired version and a TokensInheritanceStrategy; if the stored version is lower, the stored configuration is replaced when the proxy is first used, inheriting tokens by the chosen strategy, and no other code is needed.

Running it gave exactly those values (10, 10, 100); an instance that still runs the old version 1 afterwards sees 100 tokens and does not downgrade the bucket, and a plain getProxy with the old supplier is ignored. The version is a long you control; tie it to something that changes when limits change, such as a number in the plan file or a build property. Two details matter. Inheritance follows the rules of replaceConfiguration on the Bucket API page: bandwidths are matched by id, never by position (a single null-id bandwidth on each side is paired with the other), and a new bandwidth that has an id the old one lacks is treated as a different bandwidth and starts from its initial tokens, so give every bandwidth the same stable id on both sides. And the replacement happens in each store only when a proxy with the newer version is first used for that key, so rarely used keys keep the old limits until they are touched. Source: Bucket4j reference: Implicit configuration replacement.

Every consumption is a round trip. withOptimization(…​) trades some of that cost for accuracy. The built-in optimizations are documented only in the Javadoc of 8.21.0; the factories are in Optimizations.

Observed against Redis for a bucket of 1000 tokens: with delaying(new DelayParameters(100, Duration.ofSeconds(10))), after 50 local tryConsume(1) calls another proxy of the same key still saw 999 tokens (the first call is synchronized; the other 49 were not), and after syncImmediately() it saw 950. With batching(), 200 tryConsume(1) calls from 16 threads were all admitted and the store showed exactly 800 left. SkipSyncOnZeroOptimization counted 10 skipped synchronizations for 10 rejected calls on an exhausted bucket. A ManuallySyncingOptimization proxy that consumed 30 tokens was visible to others as 999 until syncImmediately() made it 970.

The reuse rule: a proxy built with an optimization holds local state (batched requests, unsynchronized tokens). Build it once and share it. Building one per request throws that state away, loses the unsynchronized tokens and gets no benefit. A proxy without optimization has no state and can be rebuilt freely. Which optimization to use is a statement about how much over-admission you can accept: batching is always safe; delaying and predicting are for limits where a few percent error is acceptable (a daily quota of millions of calls) and wrong for a limit like login attempts. Source: Optimizations.java, DelayParameters.java, PredictionParameters.java and the optimization classes at 8.21.0; the reference does not describe them.

A refill is computed from elapsed time, so the clock that produces the time matters. The reference does not say which clock is used; the 8.21.0 source does:

With replicas on machines whose clocks differ, a compare-and-swap bucket sees time jump between replicas. Two proxies on the same Redis bucket, one with a clock 30 seconds ahead, drained a bucket of 10 tokens that refills 10 per minute: the replica with the correct clock saw 0 tokens, the replica 30 seconds ahead saw 5. In other words a skewed replica refills early (or late) by the size of its skew, which for a per-minute limit is large and for a per-day limit is negligible. Keep clocks synchronized with NTP, remember that a skew comparable to the time one token takes to refill is already visible, and prefer a server-side backend (or an explicit clientClock that reads one trusted time source) when exact timing across machines matters. In tests, a controllable clock makes refill deterministic:

Source: AbstractProxyManager.getClientSideTime() and AbstractProxyManagerBuilder.clientClock at 8.21.0. The clock must be wall-clock based: clientClock throws IllegalArgumentException ("Trying to use not wall-based clock …​ in distributed environment") for a TimeMeter whose isWallClockBased() returns false, such as TimeMeter.SYSTEM_NANOTIME or the local-only test clock on the Bucket API page. Use the ManualClock from the testing page, which returns true.

The cost of a limit check is network time. A compare-and-swap check is two round trips (read, then the conditional write), plus more on a conflict; an entry-processor or server-side backend needs one; a JDBC check is one transaction of several statements. Per request that is typically a fraction of a millisecond to a few milliseconds within a data centre, and tens of milliseconds across regions, which is why the store must be in the same region as the service. Throughput per key is bounded by one round trip per consumption, because operations on the same key are serialized; throughput across keys scales with the store. So a limit shared by every request in the service (the ISBN quota) is the first one to need batching, while per-client limits rarely do. A rejected request still pays the round trip; if hostile traffic is a concern, reject cheaply earlier (at the edge, or with a small local bucket in front, see Resilience4j versus Bucket4j) so the store only sees traffic that deserves it.

The Bookshelf limits two things across all replicas: the outbound ISBN quota of 1000 calls per day, and the per-API-key plans (free 20 per minute, a five-token search endpoint). Both live in Redis, reached through one ProxyManager.

Spring Boot does not create a bare RedisClient bean (its Lettuce auto-configuration builds a LettuceConnectionFactory), so the bean above is defined here; the manual-integration page shows the full wiring and the host and port come from spring.data.redis.* as described in configuration and profiles.

Source: Bucket4j reference: Lettuce and Bucket4j reference: client-side configuration.

Three RateLimits objects, each with its own ProxyManager as three replicas would have, were driven by nine threads trying 400 ISBN calls each; exactly 1000 were admitted. For a free-plan key, a search costing 5 tokens returned 15 remaining on the first replica and 10 on the second; after the third and fourth searches the fifth was rejected by whichever replica handled it, with a positive getNanosToWaitForRefill() of about 15 seconds as the Retry-After. The same shared quota against PostgreSQL (Bucket4jPostgreSQL.selectForUpdateBasedBuilder(dataSource), three managers, 12 worker threads trying 100 times each) admitted exactly 1000. How a failing store changes the decision for an HTTP request, fail open or fail closed, is on the manual Spring Boot integration page.

Source: Bucket4j reference: implicit configuration replacement and Bucket4j reference: PostgreSQL.

The reference’s checklist, in the terms of this page:

The complete list, with the production anti-patterns, is on the production checklist. Source: Bucket4j reference: Distributed checklist.

Is it safe for two replicas to create the same bucket at once? Yes. Creation is an atomic create-if-absent (a set-if-absent script in Redis, an insert that may fail in SQL); the loser continues with the bucket the winner created, and the supplier of the loser is not used.

Does the ProxyManager keep buckets in my JVM’s memory? No. It retains nothing about buckets; a proxy is a pointer. Memory grows in the store, not in the application.

Can I keep a BucketProxy and reuse it? Yes for a few well-known buckets, as long as you understand that an optimization makes the proxy stateful. For many buckets (per user, per IP) build the proxy per request.

Why did my new limit not apply? Because the stored configuration wins over the supplier. Use implicit configuration replacement or replaceConfiguration.

Can I change the key type? Yes: withMapper on the manager or on the builder, for example String over byte[] for Redis. Source: Bucket4j reference: FAQ.