Resilience, Idempotency & Locks

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

This page is written against NestJS 12.1 (npm @nestjs/core 12.1.2), Node.js 22 and 24 LTS, TypeScript 6 (6.0.x), Express 5.2 and Fastify 5.12, and the three packages it covers, @nestjs/resilience, @nestjs/idempotency and @nestjs/locks, all at version 0.0.1 on 2026-10-02. They are pre-1.0: their APIs can still change in any release, so pin the exact version and read the release notes before upgrading. The examples build with npm run build and npx tsc --noEmit, and the HTTP behaviour shown is the same on the Express and Fastify adapters. Every example runs on in-memory stores, which keep their state in one process; the page says where a shared store (PostgreSQL or Redis) is required, and the store contracts are described so that you can write one. The three packages answer one question each for the Bookshelf application that grows across this section: what to do when a dependency is slow or down (resilience), what to do when a client repeats a request (idempotency), and what to do when several instances of the application run the same work (locks). Messages that must not be lost between a database and a broker are a related but different problem, covered in Outbox & Durable Workflows.

Three packages, one family

All three packages work the same way: a global module registers one app-wide piece of infrastructure, decorators mark the entrypoints or jobs that opt in, and a typed event stream reports what happened. The first two attach through a global interceptor (see Interceptors and Request Lifecycle), so they behave the same on controllers, GraphQL resolvers, microservice handlers and, for resilience, WebSocket gateways. The third wraps scheduled methods instead, because those never run inside a request.

Package Problem Main API Needs a shared store?

@nestjs/resilience

A dependency is slow or failing

@Timeout(), @Retry(), @CircuitBreaker(), @Bulkhead(), @Fallback(), @Resilience(preset), ResilienceService

No. State is per process, on purpose

@nestjs/idempotency

A client repeats a request that had side effects

@Idempotent(), IdempotencyStore, IdempotencyStorage

Yes, as soon as there is more than one instance

@nestjs/locks

Several instances run the same job or critical section

@OnOneInstance(), @WithoutOverlapping(), @LeaderElection(), Locks, LocksContext

Yes, as soon as there is more than one instance

The idempotency and locks packages do not ship a production store. They define a contract (IdempotencyStore, LockStore), ship an in-memory implementation and a contract test suite, and expect you to implement the store on the database or Redis client your application already has. The official tutorials show the stores on Drizzle, TypeORM and Redis; this page shows what a store must guarantee and runs the contract suite against a small one.

Resilience with @nestjs/resilience

Sooner or later every application that calls another system meets one that is slow or down. Without a plan, a slow dependency ties up your handlers, clients wait for their own timeouts, and retries from every layer add load to the service that is already struggling. @nestjs/resilience offers the usual tools as decorators: timeout, retry, circuit breaker, bulkhead and fallback, plus an outbound rate limit. They apply to entrypoints, and the same settings are available as policy objects for code inside services. Each failure becomes the transport’s native error: over HTTP a timeout answers 504 Gateway Timeout and an open breaker answers 503 Service Unavailable with Retry-After.

npm i @nestjs/resilience

Source: NestJS documentation: Resilience

The running example is a Bookshelf endpoint that looks a book up by ISBN in an external metadata provider. The client takes an AbortSignal and hands it to fetch(), so that a timed-out call is really cancelled; the base URL comes from the environment, never from the source.

A client that honours cancellation
import { Injectable } from '@nestjs/common';

export interface BookMetadata {
  isbn: string;
  title: string;
  authors: string[];
}

export class IsbnProviderError extends Error {
  constructor(readonly status: number) {
    super(`The ISBN provider answered with status ${status}`);
  }
}

@Injectable()
export class IsbnClient {
  private readonly baseUrl = process.env.ISBN_PROVIDER_URL ?? 'https://isbn.example.com';

  async lookup(isbn: string, signal: AbortSignal): Promise<BookMetadata> {
    const response = await fetch(`${this.baseUrl}/v1/books/${encodeURIComponent(isbn)}`, { signal });
    if (!response.ok) {
      throw new IsbnProviderError(response.status);
    }
    return (await response.json()) as BookMetadata;
  }
}

Source: NestJS documentation: Resilience, "Put a time budget on the quotes route"

In an application, read ISBN_PROVIDER_URL through ConfigService (see Configuration); the example reads process.env only to stay short. For an HTTP client with its own retries and interceptors, see HTTP Client, and keep retries in one layer (see the checklist below).

Module, defaults and presets

ResilienceModule.forRoot() is global, so it is imported once, in the root module. defaults fills in the fields of a stage that a decorator or preset turns on (it never turns a stage on by itself), and presets are named sets of stages, usually one per dependency. A preset’s circuit breaker, bulkhead and outbound rate limit are shared under the preset’s name by every route and service that uses it, because one dependency should have one breaker.

Defaults and one preset per dependency
import { Module } from '@nestjs/common';
import { ResilienceModule } from '@nestjs/resilience';
import { IsbnClient } from './isbn.client.js';
import { IsbnController } from './isbn.controller.js';

@Module({
  imports: [
    ResilienceModule.forRoot({
      defaults: {
        timeout: '5s',
        retry: { attempts: 3, backoff: { delay: '200ms', maxDelay: '2s' } },
      },
      presets: {
        isbn: {
          timeout: '2s',
          retry: { attempts: 2 },
          circuitBreaker: { failureRateThreshold: 50, minimumCalls: 10, openDuration: '30s' },
        },
      },
    }),
  ],
  controllers: [IsbnController],
  providers: [IsbnClient],
})
export class ResilienceDemoModule {}

Source: NestJS documentation: Resilience, "Register the module"

Things to know about the configuration:

  • Every time-valued option takes milliseconds or a string such as '200ms', '2s' or '15m'.

  • The module validates its options at bootstrap: an unknown preset name, an invalid duration or number (such as the NaN of an unset environment variable), a missing fallback method, or one breaker name configured in two different ways stops the boot instead of failing on the first request. Use ResilienceModule.forRootAsync() to build the options from ConfigService.

  • The module registers one global interceptor. For handlers without resilience decorators it only maps resilience errors to the transport’s error; set mapErrors: false to turn that off.

  • The options token is RESILIENCE_MODULE_OPTIONS, which tests can override with overrideProvider().

Timeout, retry, breaker and fallback on a route

@Resilience('isbn') applies the preset, @Timeout('1.5s') overrides the preset’s timeout for this route only (settings resolve stage by stage, with the handler’s decorator winning), and @Signal() injects the signal of the current attempt. @Fallback() names a method on the same class (or a function) that is called with the error and the ExecutionContext; handleIf narrows which errors it replaces. By default a fallback replaces every error except client errors.

A route with a time budget, a retry, a breaker and a fallback
import { Controller, Get, Param } from '@nestjs/common';
import { CircuitOpenError, Fallback, Resilience, Signal, Timeout } from '@nestjs/resilience';
import { IsbnClient, type BookMetadata } from './isbn.client.js';

@Controller('isbn')
export class IsbnController {
  constructor(private readonly isbnClient: IsbnClient) {}

  @Get(':isbn')
  @Resilience('isbn')
  @Timeout('1.5s')
  @Fallback('unknownBook', { handleIf: (error) => error instanceof CircuitOpenError })
  lookup(@Param('isbn') isbn: string, @Signal() signal: AbortSignal): Promise<BookMetadata> {
    return this.isbnClient.lookup(isbn, signal);
  }

  /** Called with the error and the ExecutionContext when the breaker is open. */
  unknownBook() {
    return { title: 'Metadata temporarily unavailable', source: 'fallback' };
  }
}

Source: NestJS documentation: Resilience, "Fail fast with a circuit breaker"

When the provider hangs, the behaviour is:

curl -i http://localhost:3000/isbn/9780441172719
{ "statusCode": 504, "error": "Gateway Timeout", "message": "The operation timed out", "code": "TIMEOUT" }

The timeout applies per attempt: the first attempt is aborted after 1.5 s (its signal aborts, so fetch() rejects and the connection closes), the preset’s retry makes a second attempt because GET is a safe method, and after the second timeout the client gets the 504. The route therefore costs about 3.2 s in the worst case (two attempts plus up to 200 ms of backoff); budget that against your client’s and load balancer’s timeouts. The status is 504 rather than 408 on purpose, because a 408 says that the client was too slow to send the request and some clients repeat it automatically.

The rules worth remembering:

  • Client errors are not failures. A 4xx HttpException, or any error with a 4xx status or statusCode property, is neither retried nor counted by the breaker (408 and 429 are the exceptions, because they mean the dependency is pushing back). Such an error reaches the provider client exactly once.

  • The breaker records every attempt. With the preset above it opens once at least 10 calls were seen and at least 50 percent failed, looking at the last 20 calls by default. While open it rejects immediately with a 503 and a Retry-After header, after openDuration it lets one probe through (halfOpenMaxCalls), and a successful probe closes it. It has no timer: the state is checked on the next call. The retry wraps the breaker, so a rejection is never retried.

  • Order of stages: fallback is the outermost, so it also sees the breaker’s rejections, and the retry wraps the breaker. Guards and middleware run once per request; only pipes, inner interceptors and the handler run again on a retry.

  • A bare @Timeout() takes the preset’s or class’s duration, then defaults.timeout.

  • Cancellation is cooperative. Code that ignores the signal keeps running in the background and its result is dropped. Pass the signal to every I/O call that accepts one, or inject ResilienceContext to read signal and attempt anywhere in the call tree (it uses AsyncLocalStorage).

Bulkhead: cap concurrent heavy work

A bulkhead caps how many calls run at once, so that one expensive endpoint cannot starve the rest of the application. A heavy CSV export that reads every book is a good candidate:

At most one export at a time, one more waiting
import { Controller, Get, Header } from '@nestjs/common';
import { Bulkhead } from '@nestjs/resilience';

@Controller('books')
export class ExportController {
  @Get('export')
  @Header('Content-Type', 'text/csv')
  @Bulkhead({ maxConcurrent: 1, maxQueue: 1, queueTimeout: '10s' })
  async exportCsv(): Promise<string> {
    // A heavy query runs here.
    return 'id,title\n1,Dune\n';
  }
}

Source: NestJS documentation: Resilience, "Cap concurrent CSV exports with a bulkhead"

Three concurrent requests to this route answered 200, 200 and 503, and the rejected one carried:

{ "statusCode": 503, "error": "Service Unavailable", "message": "Server is at capacity", "code": "BULKHEAD_FULL" }

The defaults of @Bulkhead() are maxConcurrent: 10 and maxQueue: 0, with no queueTimeout. There is no Retry-After header, because nobody knows when a slot frees up, and the status is 503 rather than 429 because a full bulkhead means the server is at capacity for everyone, not that this client is too fast (see Rate Limiting & Encryption for the latter). A bulkhead without a name belongs to its handler; give it a name to share one pool of slots between handlers.

Retrying unsafe methods

On HTTP a retry applies only to the safe methods GET, HEAD and OPTIONS, and on GraphQL only to queries. A preset’s retry is silently skipped on a POST, and a @Retry() placed directly on a POST handler is skipped too, with a warning at bootstrap. Setting @Retry({ idempotent: true }) is how you state that running the handler twice within one request is harmless. On message, event and WebSocket handlers a preset’s retry never applies by itself either: only a @Retry() on the handler turns it on, so reusing a preset cannot silently re-run a handler with side effects.

idempotent: true and @Idempotent() (next section) solve different problems. The first lets the server re-run the handler within one request; the second deduplicates the client’s repeats of the whole request. When both are used, import IdempotencyModule before ResilienceModule: the first global interceptor runs outermost, so a replay or a 409 is answered before any retry, breaker or timeout runs, and all retries of one request happen under one claimed key.

Policy objects inside services

Resilience decorators are metadata that a global interceptor reads, and interceptors run only around entrypoints. On a provider method a decorator does nothing, and bootstrap warns about it. Inside services, ask ResilienceService for a policy object: preset(name) returns a preset as one policy, create(options, name) builds one from options, and circuitBreaker(name) / bulkhead(name) expose the shared instances.

A policy object that shares the route’s breaker
import { Injectable, Logger } from '@nestjs/common';
import { CircuitOpenError, ResilienceService, type ResiliencePolicy } from '@nestjs/resilience';
import { IsbnClient } from './isbn.client.js';

@Injectable()
export class RepricingService {
  private readonly logger = new Logger(RepricingService.name);
  private readonly policy: ResiliencePolicy;

  constructor(
    resilience: ResilienceService,
    private readonly isbnClient: IsbnClient,
  ) {
    // The preset's breaker is the same instance the routes use.
    this.policy = resilience.preset('isbn');
  }

  async refresh(isbns: string[]): Promise<number> {
    let refreshed = 0;
    for (const isbn of isbns) {
      try {
        await this.policy.execute(({ signal }) => this.isbnClient.lookup(isbn, signal), {
          source: 'RepricingService.refresh',
        });
        refreshed++;
      } catch (error) {
        if (error instanceof CircuitOpenError) {
          this.logger.warn(`The ISBN provider is unavailable, stopping after ${refreshed} of ${isbns.length}`);
          break;
        }
        this.logger.error(`Could not refresh ${isbn}`);
      }
    }
    return refreshed;
  }
}

Source: NestJS documentation: Resilience, "Use the same policies inside services"

Because the job and the routes share one breaker, each learns from the other: while the breaker is open the job stops at once instead of spending a timeout per book, and failures found by the job open the breaker for the routes. A policy object has no HTTP method to inspect, so the preset’s retry always applies there; wrap only calls that are safe to repeat. Create the policy once, in the constructor. Decorators also work on microservice handlers, GraphQL resolvers and WebSocket gateways with the same semantics; errors reach the caller as an RpcException or WsException with a structured payload (status, code, statusCode, message and, for rejections that know it, retryAfter in seconds), or, on GraphQL, as a field error whose extensions hold code, httpStatus and retryAfter while the response status stays 200. In a hybrid application, connect the microservice with inheritAppConfig: true, or the global interceptor does not reach its handlers.

Events and errors

Every policy emits typed events (retry, timeout, circuit-open, circuit-half-open, circuit-closed, circuit-rejected, bulkhead-rejected, rate-limited, fallback). Inject ResilienceEvents to subscribe to its events$ stream, and alert on breaker transitions and bulkhead rejections:

Logging the events someone should see
import { Injectable, Logger, type OnModuleInit } from '@nestjs/common';
import { ResilienceEvents } from '@nestjs/resilience';

@Injectable()
export class ResilienceLogger implements OnModuleInit {
  private readonly logger = new Logger('Resilience');

  constructor(private readonly resilienceEvents: ResilienceEvents) {}

  onModuleInit() {
    this.resilienceEvents.events$.subscribe((event) => {
      switch (event.type) {
        case 'circuit-open':
        case 'circuit-half-open':
        case 'circuit-closed':
          this.logger.warn(`Circuit "${event.policy}" went from ${event.from} to ${event.to}`);
          break;
        case 'bulkhead-rejected':
          this.logger.warn(`${event.source} is at capacity (${event.active} running, ${event.queued} queued)`);
          break;
        case 'fallback':
          this.logger.log(`${event.source} answered with its fallback`);
          break;
      }
    });
  }
}

Source: NestJS documentation: Resilience, "Observe breaker transitions"

Each event is also published on a node:diagnostics_channel channel named nestjs:resilience:<type>, so metrics and tracing tools can subscribe without depending on the package. All errors extend ResilienceError:

Error Code HTTP Notes

ResilienceTimeoutError

TIMEOUT

504

has timeoutMs

CircuitOpenError

CIRCUIT_OPEN

503 with Retry-After

has retryAfterMs

BulkheadFullError

BULKHEAD_FULL

503

reason is full or queue-timeout

OutboundRateLimitError

RATE_LIMITED

503 with Retry-After

from a preset’s outboundRateLimit

Testing resilience

Do not wait 30 seconds for a breaker. Replace the client with a fake, use Vitest’s fake timers for setTimeout, clearTimeout and Date only (so the HTTP server keeps real I/O), and drive the breaker with ResilienceService:

Fake timers, a hanging dependency and a forced-open breaker
import { Test } from '@nestjs/testing';
import { ResilienceService } from '@nestjs/resilience';
import request from 'supertest';
import { afterEach, beforeEach, expect, it, vi } from 'vitest';
import { IsbnClient } from '../src/isbn.client.js';
import { ResilienceDemoModule } from '../src/resilience.module.js';

const hang = (_isbn: string, signal: AbortSignal) =>
  new Promise((_, reject) => signal.addEventListener('abort', () => reject(signal.reason)));

// ... beforeEach creates the app with overrideProvider(IsbnClient).useValue(isbn) and app.listen(0)

it('answers 504 and aborts the call when the provider hangs', async () => {
  isbn.lookup.mockImplementation(hang);
  vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout', 'Date'] });
  const response = request(app.getHttpServer()).get('/isbn/9780441172719').then((res) => res);
  await vi.waitFor(() => expect(isbn.lookup).toHaveBeenCalled());
  await vi.advanceTimersByTimeAsync(5_000); // two attempts of 1.5 s, plus backoff

  expect((await response).status).toBe(504);
  expect(isbn.lookup).toHaveBeenCalledTimes(2);
  expect(isbn.lookup.mock.calls[0][1].aborted).toBe(true);
});

it('serves the fallback while the breaker is open, then recovers', async () => {
  isbn.lookup.mockResolvedValue(metadata);
  vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout', 'Date'] });
  const breaker = app.get(ResilienceService).circuitBreaker('isbn');
  breaker.trip(); // opens the breaker directly

  const fallback = await request(app.getHttpServer()).get('/isbn/9780441172719');
  expect(fallback.body.source).toBe('fallback');
  expect(isbn.lookup).not.toHaveBeenCalled();

  vi.advanceTimersByTime(30_000); // openDuration: the next call is a probe
  const live = await request(app.getHttpServer()).get('/isbn/9780441172719');
  expect(live.body).toEqual(metadata);
  expect(breaker.state).toBe('closed');
});

Source: NestJS documentation: Resilience, "Testing"

Breakers, bulkheads and outbound rate limits belong to the application that created them, so every test application starts with fresh state. See Testing for the testing basics.

Idempotency with @nestjs/idempotency

A customer buys a book in the mobile app, on a phone on a train. The app sends POST /orders, the server charges the card, and the response never reaches the phone. The app cannot tell a lost request from a lost response, so it retries. Without protection the retry either arrives after the first request finished (an error for an order that went through) or while it is still running (two charges). An idempotency key fixes both: the client generates one unique key per attempt and sends it in the Idempotency-Key header with every retry; the server runs the handler once per key, stores the outcome, and answers every retry with it. @nestjs/idempotency implements this for HTTP routes, GraphQL mutations and microservice handlers with one decorator, and its HTTP behaviour follows the IETF Idempotency-Key header draft.

npm i @nestjs/idempotency

Source: NestJS documentation: Idempotency keys

Making an endpoint idempotent

Import IdempotencyModule once, in the root module (it is global and registers one app-wide interceptor that does nothing for handlers without @Idempotent()). scope gives every user’s keys their own namespace: without it all clients share one namespace, and a key that one user picks could replay another user’s response. Because guards run before interceptors, req.user is already set when scope runs and an unauthenticated request is rejected before it can use up a key. The example reads a header as a stand-in for what an authentication guard would put on the request (see Authentication and Guards).

Module with a per-user scope and explicit TTLs
import { Module } from '@nestjs/common';
import { IdempotencyModule } from '@nestjs/idempotency';
import { OrdersController } from './orders.controller.js';
import { OrdersService } from './orders.service.js';

@Module({
  imports: [
    IdempotencyModule.forRoot({
      // Runs after guards. Keys never collide across users.
      scope: (req: { headers: Record<string, string | undefined> }) => req.headers['x-user-id'],
      ttl: '48h',
      lockTtl: '45s',
      retryAfter: '2s',
      // Memory is fine for one instance in development; production refuses to start with it.
      allowInMemoryStorage: true,
    }),
  ],
  controllers: [OrdersController],
  providers: [OrdersService],
})
export class IdempotencyDemoModule {}

Source: NestJS documentation: Idempotency keys, "Make the payment endpoint idempotent"

Then mark the handler. required: true rejects a request without a key with a 400 before the handler runs, which is better than charging a card unprotected.

A POST that is safe to repeat
import { Body, Controller, Headers, Post } from '@nestjs/common';
import { Idempotent } from '@nestjs/idempotency';
import { OrdersService } from './orders.service.js';

@Controller('orders')
export class OrdersController {
  constructor(private readonly ordersService: OrdersService) {}

  @Post()
  @Idempotent({ required: true })
  place(
    @Headers('x-user-id') userId: string,
    @Body() body: { bookId: string; quantity: number },
  ) {
    return this.ordersService.place(userId, body.bookId, body.quantity);
  }
}

Source: NestJS documentation: Idempotency keys, "Make the payment endpoint idempotent"

What the interceptor does for a request that carries a key:

  • It computes a fingerprint, a SHA-256 hash of the scope, the method, the URL and the body with object keys sorted.

  • It atomically takes a lock on the record key <scope>:<key>. If a record exists, the handler does not run. While the handler runs, the interceptor renews the lock every lockTtl / 3, so a slow handler never loses it to a retry.

  • When the handler finishes, the lock becomes a completed record holding the status, the body and a short allowlist of headers (Location, Content-Type, Content-Language, Content-Location, ETag, Last-Modified); cookies and other per-response headers are never stored. Add more with the replayHeaders option.

Every later request with the same key gets one of these answers with the example above:

Situation Response

The first request is still running

409 with code IDEMPOTENCY_KEY_IN_USE and a Retry-After header (here 2, from retryAfter)

The first request completed

The stored status and body, plus Idempotent-Replayed: true; the handler does not run

Same key, different body or URL

422 with code IDEMPOTENCY_KEY_REUSED

No key, with required: true

400 with code IDEMPOTENCY_KEY_REQUIRED

 
curl -i -X POST http://localhost:3000/orders \
  -H 'X-User-Id: usr_alice' \
  -H "Idempotency-Key: $(uuidgen)" \
  -H 'Content-Type: application/json' \
  --data '{"bookId":"book_1","quantity":1}'
{ "id": "ord_1", "bookId": "book_1", "userId": "usr_alice", "quantity": 1 }

Repeating the call with the same key returns the same body with Idempotent-Replayed: true, and the service’s charge counter stays at 1; two concurrent calls with a new key give one 201 and one 409, with the charge still counted once. A key is 1 to 255 printable ASCII characters. The same key sent by another user is a different record because of the scope. The interceptor is global, so it runs outside the interceptors of your controllers: the stored body is exactly what the client received after serializers and other interceptors ran, and a replay skips all of them (see Serialization). Pipes run inside it, so a ValidationPipe 400 is stored like any other client error.

The client side matters as much as the server side. Create the key when the user taps the button, not once per request; save it with the pending action so every retry, even after an app restart, sends the same key; create a new key only for a new decision of the user. Retry the same key after a network error, a 5xx, a 429, or a 409 with IDEMPOTENCY_KEY_IN_USE (after Retry-After); never retry a 422, and treat other 4xx answers as final. Check the code, not just the status, because a 409 can also be the application’s own "already paid" conflict.

TTLs, what is stored and what is not

Option Default Meaning

ttl

'24h'

How long a completed result can be replayed. Must be longer than the longest time a client keeps retrying; afterwards the key is forgotten and a retry runs the handler again

lockTtl

'60s'

How long an in-flight lock survives after the process running the handler crashes; renewed every lockTtl / 3 while it runs

retryAfter

'1s'

The Retry-After on a 409, in whole seconds

All three can be set per handler in @Idempotent(), together with required, keyFrom, scope, fingerprint and storeIf. @Idempotent() also works on a controller or resolver class, where it covers the handlers that change state (GET routes and GraphQL queries are skipped unless they have their own decorator). Not every outcome is stored. By default, anything below 500 is:

Outcome Default

2xx

Stored and replayed

4xx HttpException

Stored and thrown again on every replay (it is deterministic)

5xx or an unknown error

Not stored: the key is released, so a retry runs again

Streams, StreamableFile, @Res() without passthrough

Not stored: the key is released and a warning is logged

A replayed error is rebuilt as a plain HttpException with the stored status and body and passes through your exception filters again (see Exception Filters & Error Handling). storeIf: (status, error) ⇒ boolean changes the rule. It matters for the one gap the default leaves: a 5xx that happens after the side effect (the card was charged and the next statement failed) releases the key, and the retry charges again. Close the gap by passing the payment provider its own idempotency key derived from the user id and the client’s key, by storing that specific failure with storeIf, or by writing the order and its event in one transaction with an outbox (see Transactional Outbox).

GraphQL mutations and microservice events

The decorator works on a GraphQL mutation: the key comes from an idempotencyKey argument when the mutation declares one, otherwise from the Idempotency-Key header. The record key gets the field’s path appended, because one operation can contain several mutations, and a rejection is an error on that field (the response is still 200, with code, httpStatus and retryAfter in extensions). GraphQL replays do not set the Idempotent-Replayed header.

For events delivered at least once by a broker, derive an id from the data (an order is paid once, so order-paid:<orderId>), and point keyFrom at it:

Deduplicating a microservice event
import { Controller } from '@nestjs/common';
import { EventPattern, Payload } from '@nestjs/microservices';
import { Idempotent } from '@nestjs/idempotency';

export interface OrderPaidEvent {
  eventId: string;
  orderId: string;
}

@Controller()
export class ShippingController {
  @EventPattern('order.paid')
  @Idempotent({ required: true, keyFrom: { payload: 'eventId' } })
  async onOrderPaid(@Payload() event: OrderPaidEvent) {
    // Book the courier here.
  }
}

Source: NestJS documentation: Idempotency keys, "Deduplicate order.paid events"

By default a microservice handler reads the key from the payload’s idempotencyKey property, then from a transport header named by the header option. A duplicate whose first delivery completed is skipped; one that arrives while the first is still running is rejected with IDEMPOTENCY_KEY_IN_USE, so with a broker that acknowledges messages do not acknowledge that delivery and let the broker redeliver it later. Use a longer ttl for events (the official tutorial uses 7 days), since a message can come back from a dead-letter queue long after the first delivery. See Events, Scheduling & Queues for the producing side.

Stores

Without a registered store, records live in InMemoryIdempotencyStore: lost on restart and invisible to other instances, so a retry that lands on another instance runs the handler again. With NODE_ENV=production and no store the application refuses to start, unless allowInMemoryStorage: true is set. A production store implements IdempotencyStore, an ordinary singleton provider that calls registerSource(this) on IdempotencyStorage in its constructor. The registry accepts one store, locks when IdempotencyModule initializes, and logs the store in use.

Method What it does How it stays atomic

acquire(key, owner, fingerprint, lockTtl)

Takes the lock on a free or expired key, or returns the existing record (in-flight or completed)

One insert-or-take-over statement (INSERT …​ ON CONFLICT, a Lua script): of many concurrent callers exactly one wins

complete(key, owner, response, ttl)

Turns the owner’s live lock into a completed record

Compare-and-set on owner and expiry in the same write

release(key, owner)

Deletes the owner’s live lock, so a retry can run

Compare-and-set, as complete()

extend(key, owner, lockTtl)

Renews the owner’s live lock (called every lockTtl / 3)

Compare-and-set, as complete()

owner is a random id per attempt, a fencing token: if an attempt stops renewing (its process crashed) and a retry takes the key over after lockTtl, the old attempt’s extend(), complete() or release() changes nothing. The owner check therefore has to happen in the same atomic operation that writes. Durations arrive in whole milliseconds, and an expired record does not exist for any method, deleted or not. The response column should be json rather than jsonb in PostgreSQL, because jsonb rejects a \u0000 inside a string and a body can contain one; with Redis use maxmemory-policy noeviction, because an evicted record is a double charge waiting to happen.

Prove a store with the suite that @nestjs/idempotency/testing exports. idempotencyStoreContract() returns test cases for any runner, and concurrent: true adds the races (16 callers racing for each key) that a store which reads and then writes gets wrong. The store below follows the contract in one process; it is a teaching store, no better than the built-in one, because a real store runs each method as one atomic statement in a shared database or one Lua script in Redis.

A contract-conforming store (single process) and its test
import { Injectable } from '@nestjs/common';
import {
  IdempotencyStorage,
  type IdempotencyAcquireResult,
  type IdempotencyStore,
  type IdempotencyStoredPayload,
} from '@nestjs/idempotency';

interface Entry {
  fingerprint: string;
  owner: string | null;
  response: IdempotencyStoredPayload | null;
  expiresAt: number;
}

@Injectable()
export class MapIdempotencyStore implements IdempotencyStore {
  private readonly entries = new Map<string, Entry>();

  constructor(storage: IdempotencyStorage) {
    storage.registerSource(this);
  }

  async acquire(key: string, owner: string, fingerprint: string, lockTtl: number): Promise<IdempotencyAcquireResult> {
    const now = Date.now();
    const found = this.live(key, now);
    if (!found) {
      this.entries.set(key, { fingerprint, owner, response: null, expiresAt: now + lockTtl });
      return { state: 'acquired' };
    }
    return found.response === null
      ? { state: 'in-flight', fingerprint: found.fingerprint }
      : { state: 'completed', fingerprint: found.fingerprint, response: structuredClone(found.response) };
  }

  async complete(key: string, owner: string, response: IdempotencyStoredPayload, ttl: number): Promise<boolean> {
    const now = Date.now();
    const found = this.live(key, now);
    if (!found || found.owner !== owner) {
      return false;
    }
    // Keep a copy: the caller may mutate what it passed, and so may whoever reads it back.
    Object.assign(found, { owner: null, response: structuredClone(response), expiresAt: now + ttl });
    return true;
  }

  async release(key: string, owner: string): Promise<boolean> {
    const found = this.live(key, Date.now());
    if (!found || found.owner !== owner) {
      return false;
    }
    return this.entries.delete(key);
  }

  async extend(key: string, owner: string, lockTtl: number): Promise<boolean> {
    const now = Date.now();
    const found = this.live(key, now);
    if (!found || found.owner !== owner) {
      return false;
    }
    found.expiresAt = now + lockTtl;
    return true;
  }

  /** An expired record does not exist, whether or not it was deleted yet. */
  private live(key: string, now: number): Entry | undefined {
    const entry = this.entries.get(key);
    return entry && entry.expiresAt > now ? entry : undefined;
  }
}
import { IdempotencyStorage } from '@nestjs/idempotency';
import { idempotencyStoreContract } from '@nestjs/idempotency/testing';
import { describe, it } from 'vitest';
import { MapIdempotencyStore } from '../src/map-idempotency.store.js';

describe('the IdempotencyStore contract', () => {
  const cases = idempotencyStoreContract(() => new MapIdempotencyStore(new IdempotencyStorage()), { concurrent: true });
  for (const c of cases) {
    it(c.name, c.run);
  }
});

Source: NestJS documentation: Idempotency keys, "The store contract"

The suite catches a typical store bug: a store that keeps the response object it is given and hands it back by reference fails the contract case "stores every payload as given, and returns an equal copy", because the caller’s later mutation changes the stored record. The structuredClone() calls are the fix; with them all 25 cases pass.

Encryption, rotation and the checklist

Stored responses can contain personal or payment data. Turn on encryption: { keys } (AES-256-GCM, newest key first, keys read from your secret manager and never from the source) before the endpoint stores its first record: the record key is authenticated with the data, so a record copied under another user’s key fails to open. Rotate by deploying old,new, then new,old, and only when ttl has passed new; removing a key too early makes replays fail closed with a 500 IDEMPOTENCY_RECORD_UNREADABLE rather than run the handler again.

The checklist from the official tutorial, condensed: register a shared store even for one instance; set required: true on every endpoint that moves money or creates something a customer would notice twice; set scope for every client-facing endpoint; keep the idempotency interceptor outside your other global interceptors (the module refuses to start next to ClassSerializerInterceptor); keep ttl above the client’s retry window; watch IdempotencyEvents (replayed, rejected, lock-lost, also published on nestjs:idempotency:* channels) and alert on lock-lost; add Idempotency-Key to the CORS allowedHeaders and Idempotent-Replayed and Retry-After to exposedHeaders for browsers (see Security Headers, CORS & CSRF); keep the state check in the service, because two taps that create two keys are two attempts; and keep idempotent responses small, since the whole body is stored and the package has no size limit yet.

Locks with @nestjs/locks

The Bookshelf runs a nightly export of invoices to the accounting system and a five-minute stock reconciliation with the warehouse, both with @nestjs/schedule (see Events, Scheduling & Queues). On one instance that works. In production the API runs on three, and @nestjs/schedule fires every @Cron() on each of them: the accounting system records every invoice three times, and a long reconciliation starts a second and a third copy alongside the first. The usual fixes have holes: a RUN_JOBS=true instance fails during a rolling deploy or when it is the one that dies, and a SET key NX PX lock at the top of the job fails when its holder pauses for longer than the key’s TTL (a stop-the-world garbage collection, a partition): another instance takes the key and, when the first wakes up, both are in the critical section.

@nestjs/locks turns a job into something one instance holds: a lease that the holder keeps renewing while it is up, checked in a store every instance shares, in the spirit of Laravel’s onOneServer() and withoutOverlapping() (Task Scheduling). Under the decorators is a distributed lock you can take yourself, with the two things a lock needs to be safe: an expiry that the store enforces and a fencing token that grows with every acquisition.

npm i @nestjs/locks @nestjs/schedule

Source: NestJS documentation: Distributed locks

Module and store

LocksModule.forRoot() works as soon as it is imported, on an in-memory store that excludes callers in one process only: three instances would each have their own and all three would take "the" lock. A shared store is therefore required for the package to do anything across instances; with NODE_ENV=production and no registered store the application does not start unless allowInMemoryStorage: true is set. The options are ttl (default '30s': how long a lock lives without a renewal, so how soon another instance takes over after a crash), clock (for tests) and allowInMemoryStorage.

The LockStore contract has three methods, each atomic, with owner (a random id per acquisition) first and durations in whole milliseconds:

Method What it does How it stays atomic

acquire(key, owner, ttl)

Takes the key if no live lock holds it, and returns a fencing token greater than every token that key had before; reports acquired: false otherwise

One insert-or-take-over statement that re-checks expiry, with the token drawn after the check

renew(key, owner, ttl)

Heartbeat: moves the expiry to ttl from now if owner still holds a live lock (called every ttl / 3)

Compare-and-set on owner and expiry

release(key, owner)

Frees the owner’s live lock, never resetting the key’s token

Compare-and-set, as renew()

The store measures ttl with its own clock (Redis PEXPIRE, the database’s clock_timestamp()), so every instance agrees on when a lock expires. Tokens are positive safe integers returned as numbers; a PostgreSQL bigint comes back as a string unless the ORM converts it, and the package refuses such a token at once. With PostgreSQL the token comes from a sequence drawn inside the statement, and with Redis from an INCR counter that must never go back, which needs persistence and maxmemory-policy noeviction. An advisory lock (pg_advisory_lock()) is not a substitute: it is bound to a session, cannot be renewed, has no expiry for a paused holder and no fencing token. Test any store with lockStoreContract() from @nestjs/locks/testing, with concurrent: true, against a server with a connection pool where the calls really overlap.

The example below registers the built-in InMemoryLockStore through LocksStorage, which is exactly how a production store registers itself (a provider whose constructor calls storage.registerSource(this)). In the tests, the same store instance is handed to two application instances, which stands in for one shared database.

The module: schedule, locks and a registered store
import { Module } from '@nestjs/common';
import { InMemoryLockStore, LocksModule, LocksStorage } from '@nestjs/locks';
import { ScheduleModule } from '@nestjs/schedule';
import { InvoiceExportJob, PendingInvoices } from './invoice-export.job.js';
import { InvoiceExportsController } from './invoice-exports.controller.js';
import { StockReconciliationJob } from './stock-reconciliation.job.js';
import { WarehouseFeed } from './warehouse-feed.js';

export const SHARED_STORE = Symbol('SHARED_STORE');

@Module({
  imports: [ScheduleModule.forRoot(), LocksModule.forRoot({ ttl: '30s' })],
  controllers: [InvoiceExportsController],
  providers: [
    // In production: your own LockStore on PostgreSQL or Redis, registering itself in its constructor.
    { provide: SHARED_STORE, useFactory: () => new InMemoryLockStore() },
    {
      provide: 'REGISTER_STORE',
      inject: [LocksStorage, SHARED_STORE],
      useFactory: (storage: LocksStorage, store: InMemoryLockStore) => storage.registerSource(store),
    },
    PendingInvoices,
    InvoiceExportJob,
    StockReconciliationJob,
    WarehouseFeed,
  ],
})
export class LocksDemoModule {}

Source: NestJS documentation: Distributed locks, "Keep locks in your database"

One instance per job: @OnOneInstance() and @WithoutOverlapping()

Add @OnOneInstance() next to @Cron(). The first instance whose scheduler fires the job takes the job’s lease, a lock named after the key with an :owner suffix, and keeps renewing it every ttl / 3; on the other instances the tick is skipped (the call resolves without running the method and a debug line says why; nothing is queued). When the owner dies it stops renewing, the lease expires after ttl, and the first instance whose tick comes after that takes the job over. @WithoutOverlapping() takes a second lock, named after the same key, for the whole run: a tick that finds it held is skipped on any instance, so a long run is never started twice. On its own it means "any instance, one run at a time".

A nightly job that runs on one instance and never overlaps itself
import { Injectable, Logger } from '@nestjs/common';
import { OnOneInstance, WithoutOverlapping } from '@nestjs/locks';
import { Cron } from '@nestjs/schedule';
import { AccountingClient } from './accounting.client.js';

/** The export's lock: held by a run, scheduled or started by hand. */
export const INVOICE_EXPORT_LOCK = 'invoices:nightly-export';

@Injectable()
export class InvoiceExportJob {
  private readonly logger = new Logger(InvoiceExportJob.name);

  constructor(
    private readonly pending: PendingInvoices,
    private readonly accounting: AccountingClient,
  ) {}

  @Cron(process.env.INVOICE_EXPORT_CRON ?? '0 2 * * *', { name: 'invoice-export', timeZone: 'Europe/Madrid' })
  @OnOneInstance({ key: INVOICE_EXPORT_LOCK })
  @WithoutOverlapping()
  async exportInvoices() {
    await this.export();
  }

  /** Call it under the INVOICE_EXPORT_LOCK lock. Does everything that is due, not "yesterday's". */
  async export(): Promise<{ exported: number }> {
    const orderIds = this.pending.orderIds.splice(0);
    if (orderIds.length > 0) {
      await this.accounting.upload(orderIds);
      this.logger.log(`Exported ${orderIds.length} invoices`);
    }
    return { exported: orderIds.length };
  }
}

Source: NestJS documentation: Distributed locks, "Run each job on one instance" and Don’t let a long run overlap itself

Firing the job’s schedule on two instances that share one store, through @nestjs/schedule’s own registry, produced one upload of the two pending orders, one log line saying the job runs on this instance with its lease name and fencing token, and one debug line on the other instance saying `Skipped InvoiceExportJob.exportInvoices: it runs on another instance. How it works and what to know:

  • The decorators wrap the method. @nestjs/schedule is not patched; its explorer finds the @Cron() metadata on the wrapper, in either decorator order. @Interval() and @Timeout() jobs work the same way, and so does calling the method yourself.

  • The key defaults to ClassName.methodName. Set it explicitly on a job that must never run twice: a renamed class or method changes the default, and during a rolling deploy the old and new instances would each run the job once. Two jobs with the same default key fail at startup, and so does a job whose key is another job’s lease (<key>:owner) or an election’s key.

  • Ownership is sticky on purpose. A race per tick would need a lock that outlives the run by the instances' clock skew and a "ran at" record; a lease held between ticks has none of that, at the cost of one instance running all of a job’s ticks.

  • Missed ticks are not caught up. While the owner is down, up to ttl of ticks are skipped, and nothing records when a job last ran. Write jobs that do what is due, not what the tick was for: the export takes every order without an invoice, so a missed night is caught up the next night.

  • Clean shutdown hands the lease back after running jobs finish, so call app.enableShutdownHooks() (see Lifecycle Events & Shutdown); the package releases leases in onModuleDestroy() and other locks in beforeApplicationShutdown(), so the connection can be closed in onApplicationShutdown().

Fencing tokens and LocksContext

A lease cannot stop a holder that is paused. Instance A reads the warehouse counts, then stalls in a garbage collection. It stops renewing, the store expires its lock, instance B takes over and writes fresher counts, and then A wakes up and writes the stale ones it read earlier. The holder’s signal aborts at its next renewal, but a write already on its way goes through. The remedy is the fencing token: every acquisition of a key gets a greater token, the holder hands it to the resource it writes, and the resource refuses a token lower than one it has already seen. The check must be in the same statement as the write.

The write is fenced: a newer run’s data is left alone
import { Injectable } from '@nestjs/common';

export interface StockRow {
  count: number;
  /** The fencing token of the run that last wrote `count`. */
  fencingToken: number;
}

/** Stands in for the books table. The check and the write must be ONE statement in a database. */
@Injectable()
export class StockRepository {
  readonly rows = new Map<string, StockRow>([['book_1', { count: 12, fencingToken: 0 }]]);

  async writeStock(counts: Record<string, number>, fencingToken: number): Promise<{ written: number; stale: number }> {
    let written = 0;
    for (const [bookId, count] of Object.entries(counts)) {
      const row = this.rows.get(bookId);
      // UPDATE books SET in_stock = $1, stock_token = $2 WHERE id = $3 AND stock_token <= $2
      if (!row || row.fencingToken <= fencingToken) {
        this.rows.set(bookId, { count, fencingToken });
        written++;
      }
    }
    return { written, stale: Object.keys(counts).length - written };
  }
}

Source: NestJS documentation: Distributed locks, "Fence the writes"

@nestjs/schedule calls the method without arguments, so the job reads its token and signal from LocksContext, which carries them through AsyncLocalStorage to every service the job calls. Outside a job, a withLock() callback or LocksContext.run(lock, fn), lock, fencingToken and signal are undefined.

A job that passes the signal on and fences its write
import { Injectable, Logger } from '@nestjs/common';
import { LocksContext, OnOneInstance, WithoutOverlapping } from '@nestjs/locks';
import { Cron, CronExpression } from '@nestjs/schedule';
import { StockRepository } from './stock.repository.js';
import { WarehouseClient } from './warehouse.client.js';

@Injectable()
export class StockReconciliationJob {
  private readonly logger = new Logger(StockReconciliationJob.name);

  constructor(
    private readonly warehouseClient: WarehouseClient,
    private readonly stockRepository: StockRepository,
    private readonly locksContext: LocksContext,
  ) {}

  @Cron(process.env.STOCK_RECONCILIATION_CRON ?? CronExpression.EVERY_5_MINUTES, { name: 'stock-reconciliation' })
  @OnOneInstance({ key: 'inventory:stock-reconciliation' })
  @WithoutOverlapping()
  async reconcileStock() {
    const { fencingToken, signal } = this.locksContext;
    // The signal aborts the call if this run loses its lock...
    const counts = await this.warehouseClient.stockCounts({ signal: signal! });
    // ...and the token makes the repository refuse a write that slips through anyway.
    const { written, stale } = await this.stockRepository.writeStock(counts, fencingToken!);
    if (stale > 0) {
      this.logger.warn(`Skipped ${stale} stale stock writes: a newer run wrote those books`);
    }
    this.logger.log(`Reconciled stock for ${written} books (fencing token ${fencingToken})`);
  }
}

Source: NestJS documentation: Distributed locks, "Fence the writes"

The signal aborts when the run’s lock or the instance’s lease is lost, with a LockLostError whose detectedBy says whether the store refused a renewal ('renewal') or ttl passed without a confirmed one ('deadline'). Pass the signal to what the job calls (fetch, the database driver); a job that ignores it keeps running, as in any process, because nothing is killed. The package logs a warning for every lost lock and emits a lock-lost event, since a lost lock means another instance may be in the same critical section right now and only the token keeps the two apart. The fenced repository leaves the fresher count (token 7) in place when a write with token 6 arrives.

Taking a lock yourself: Locks

An admin endpoint that runs the export by hand must never overlap the scheduled run, on any instance. The scheduled run holds the lock named by its key, so the endpoint takes the same lock with Locks.withLock(), waiting up to two seconds for a run that is about to finish:

A manual run that excludes the scheduled one
import { ConflictException, Controller, HttpCode, Post } from '@nestjs/common';
import { LockNotAcquiredError, Locks } from '@nestjs/locks';
import { INVOICE_EXPORT_LOCK, InvoiceExportJob } from './invoice-export.job.js';

@Controller('admin/invoice-exports')
export class InvoiceExportsController {
  constructor(
    private readonly locks: Locks,
    private readonly invoiceExportJob: InvoiceExportJob,
  ) {}

  // Protect this route with your application's authorization (see the Authorization page).
  @Post()
  @HttpCode(200)
  async exportNow() {
    try {
      return await this.locks.withLock(INVOICE_EXPORT_LOCK, () => this.invoiceExportJob.export(), { wait: '2s' });
    } catch (error) {
      if (error instanceof LockNotAcquiredError) {
        throw new ConflictException('An invoice export is already running. Try again when it has finished.');
      }
      throw error;
    }
  }
}

Source: NestJS documentation: Distributed locks, "Run the export by hand"

While another holder had the lock the request answered 409 after waiting its two seconds; once the lock was released the same call answered 200 with { "exported": 2 }. withLock(key, fn, options) releases the lock whether fn resolves or throws; wait (default 0, try once) retries with backoff before giving up with LockNotAcquiredError, and signal stops the waiting. acquire(key, options) is the same without the callback: it resolves a Lock or null, and you release it yourself or with await using. A Lock has key, owner, fencingToken, signal, held and release(). None of the package’s errors carries an HTTP status, because contention is not the caller’s mistake: the handler decides, here with a 409. Protect such a route with real authorization (see Authorization).

Leader election

Some work is not a scheduled job but a resource one instance should hold for as long as it lives, such as a feed that allows one connection per shop. That is a leader election, the same lease as a job’s ownership held on purpose. Put @LeaderElection(key) on a singleton provider that implements OnLeadershipAcquired, OnLeadershipLost or both:

One instance holds the warehouse feed
import { Injectable, Logger } from '@nestjs/common';
import { LeaderElection, type Lock, type OnLeadershipAcquired, type OnLeadershipLost } from '@nestjs/locks';

@Injectable()
@LeaderElection('inventory:warehouse-feed')
export class WarehouseFeed implements OnLeadershipAcquired, OnLeadershipLost {
  private readonly logger = new Logger(WarehouseFeed.name);

  onLeadershipAcquired(lock: Lock) {
    this.logger.log(`Connected to the warehouse feed (fencing token ${lock.fencingToken})`);
    // Open the connection here and pass `lock.signal` to it: it aborts when the lease is
    // lost or this instance steps down.
  }

  onLeadershipLost() {
    this.logger.log('Disconnected from the warehouse feed');
  }
}

Source: NestJS documentation: Distributed locks, "Hold the warehouse feed on one instance"

From onApplicationBootstrap, every instance tries to take the lease and keeps trying every ttl / 3; the one that has it renews it. With two instances on one store, exactly one logged Connected to the warehouse feed. onLeadershipAcquired() receives the lease’s signal and fencingToken, and onLeadershipLost() runs when the lease was lost or the instance stepped down (at shutdown, or after onLeadershipAcquired() threw). The ttl option sets the lease’s own ttl.

Events, errors and the checklist

LocksEvents.events$ emits lock-lost (with detectedBy), leadership-acquired and leadership-lost (with reason), each also published on nestjs:locks:<event> diagnostics channels. The errors are LocksError (the base class), LockNotAcquiredError (key, waitMs) and LockLostError (key, fencingToken, detectedBy).

For production: register a shared store and run the contract suite against it in CI on the database version you run; set key explicitly on every job; pick a ttl well above the store’s round trip and the longest event-loop stall you accept, and short enough that a crash does not skip many ticks (30 seconds fits most jobs, and a long job’s run lock is renewed at the same pace, so its ttl need not cover the run); pass LocksContext.signal to what jobs call and fence irreversible writes with LocksContext.fencingToken; alert on lock-lost; call app.enableShutdownHooks(); and keep the instances' clocks synchronized anyway, because a holder counts its own deadline and @nestjs/schedule fires on each instance’s clock. The skipped ticks are logged at debug, so keep that level on for a while after a rollout to see the jobs settle on one instance.

Testing several instances in one process

An instance is a Nest application, and instances of one application share a store, so a test can start several in one process and prove the behaviour. Override the shared store provider with one object, start two applications, and fire the schedule through `@nestjs/schedule’s own registry, the way the cron expression would:

Two instances, one store, one run
import { SchedulerRegistry } from '@nestjs/schedule';
import { Test } from '@nestjs/testing';
import { InMemoryLockStore } from '@nestjs/locks';
import { expect, it, vi } from 'vitest';
import { LocksDemoModule, SHARED_STORE } from '../src/locks.module.js';

it('runs the export once with two instances', async () => {
  const store = new InMemoryLockStore(); // the one store every instance shares
  const apps = [];
  for (let i = 0; i < 2; i++) {
    const moduleRef = await Test.createTestingModule({ imports: [LocksDemoModule] })
      .overrideProvider(SHARED_STORE).useValue(store)
      .compile();
    const app = moduleRef.createNestApplication({ logger: false });
    await app.init();
    apps.push(app);
  }
  // 02:00 on every instance: fire the job through @nestjs/schedule's own callback.
  await Promise.all(apps.map((app) => app.get(SchedulerRegistry).getCronJob('invoice-export').fireOnTick()));
  await vi.waitFor(() => expect(accounting.batches).toHaveLength(1));
  expect(accounting.batches).toEqual([['ord_1001', 'ord_1002']]);
  await Promise.all(apps.map((app) => app.close()));
});

Source: NestJS documentation: Distributed locks, "Testing"

For the handover after a crash, give the "crashed" instance a ManualLockClock (exported by @nestjs/locks) through the module’s clock option: it only moves when told to, so an instance on it never renews and the store, on its own clock, expires its locks after ttl. await clock.advance('5m') fires every renewal that falls due, in order, which makes a long run with its locks renewed testable in a moment.

Where the book is outdated

The requester’s reference book builds some of these features by hand, notably a Redlock-style lock and its own idempotency handling for retried requests. Three things have changed since:

  • Each of them now has a first-party package (@nestjs/locks, @nestjs/idempotency, @nestjs/resilience), with the hard parts specified instead of improvised. A lock made only of SET key value NX PX has no way to tell that its holder paused past the expiry, so two instances end up in the critical section; the fencing token and the signal that aborts on a lost lease are the answer, and they are what a hand-rolled lock usually lacks. The package’s own stores for PostgreSQL and Redis are shown with their atomic statements instead of a read followed by a write.

  • Retries are now policy, not a loop. A hand-written retry loop around an HTTP call tends to retry everything, including POST requests and client errors, and multiplies across layers. @nestjs/resilience retries only what is safe by default, never retries client errors, and keeps one breaker per dependency shared by routes and background jobs.

  • Idempotency belongs at the boundary. The package stores the outcome per user-scoped key, fingerprints the request, answers concurrent duplicates with 409, and encrypts what it stores when asked; a hand-written "has this id been processed" table usually misses the in-flight case.

Do not copy a Redlock implementation from a tutorial for a job-scheduling problem. Decide first whether you need mutual exclusion at all (a single instance, a queue consumer from Events, Scheduling & Queues or an idempotent job may be enough), and if you do, use a lease with a store whose contract you have tested.

What changed recently

  • The three packages are new, published at 0.0.1 and documented in the Reliability chapter that the Sep 2026 documentation restructure added (/reliability/resilience, /reliability/idempotency, /reliability/locks, next to /reliability/outbox and /reliability/workflows). All require Node.js 20.19 or later. Because they are pre-1.0, expect option names and defaults to move; this page lists what the published 0.0.1 packages export.

  • ResilienceModule and IdempotencyModule each register a global interceptor, so the import order matters: the first global interceptor runs outermost. Import IdempotencyModule first.

  • Task scheduling moved to /application/task-scheduling in the same restructure; @nestjs/locks decorates its methods without patching @nestjs/schedule (the examples use @nestjs/schedule 12.0.2).

  • On the JVM the same concepts (timeout, retry, circuit breaker, bulkhead, fallback) are provided by Resilience4j; see Resilience Fundamentals.

Pitfalls

  • State is per process in @nestjs/resilience. With several instances each has its own breakers, bulkheads and outbound rate limits, and a restart starts them fresh. Each breaker opens on its own, and a bulkhead limit applies per instance.

  • Time budget arithmetic. Time per attempt times attempts, plus backoff, must fit inside the caller’s timeout and the load balancer’s. Retry at one layer only and switch off retries in SDKs and HTTP clients underneath.

  • A breaker learns only from calls that finish. Give every breaker a timeout, or a hanging call is never counted and a hanging probe keeps a half-open breaker rejecting everything else.

  • Decorators on providers do nothing. Resilience decorators apply to controllers, resolvers, message handlers and gateways; use policy objects inside services.

  • In-memory stores lie across instances. Both the idempotency and the lock stores must be shared and atomic; with no store, production refuses to start.

  • A lock is not a transaction. A lease expires, and a paused holder can still write. Pass the signal, fence the writes with the token, and make jobs do what is due.

  • Express and Fastify. The packages are platform-neutral: the same decorators gave the same 504 on a timeout and the same replay with Idempotent-Replayed: true on the Fastify adapter. Headers beyond the replay allowlist are written by the platform for every response, so they are never stored.

References