Cluster & Scaling

Source: Node.js API: Cluster, Node.js API: os.availableParallelism(), Node.js API: Stability index

Three limits push a Node.js service past a single process. The first is CPU: JavaScript runs on one thread, so a service that spends 1 ms of CPU per request tops out near a thousand requests per second per process, and the other cores sit idle (see Don’t Block the Event Loop). The second is availability: one process is one failure domain, and an uncaught exception, an out-of-memory kill or a deploy takes the whole service down with it. The third is throughput of the machine itself, which has a ceiling, so at some point you need more machines.

Three strategies cover these, and they compose.

The Node.js Learn article on concurrency models (https://nodejs.org/learn/concurrency/comparing-nodejs-concurrency-models) compares these options with worker threads and child processes.

Cloning is the cheapest of these because it changes no application code, which is exactly why it is easy to do wrong: the clones do not share memory, so any code that assumed one process (an in-memory cache, a session map, a rate-limit counter, a WebSocket room) is now wrong in a way that only shows up under load.

A cluster is one primary process and N worker processes. The primary does not run your request handlers; it forks the workers, listens for their messages and deaths, and, depending on the scheduling policy, hands connections to them. Each worker is a complete Node.js process with its own V8 heap, its own event loop and its own libuv thread pool (see The Event Loop), so it also pays its own startup time and baseline memory.

Under the hood the workers are started with child_process.fork(), so every worker has an IPC channel to the primary, and the two sides can exchange messages and even network handles over it (see Child Processes). A process is a worker when the environment variable NODE_UNIQUE_ID is set, which is how cluster.isPrimary and cluster.isWorker are computed. The same script runs in both roles, which is why most examples start with an if (cluster.isPrimary) branch.

The trick that lets many processes share one port is in server.listen(). When a worker calls it, the call is forwarded to the primary, which opens the socket once (or reuses it) and then either accepts connections itself and passes each one to a worker, or shares the listening socket with the workers. Because the primary does the work on a worker’s behalf, three things differ from a normal process, all documented by Node.js:

Source: Node.js API: Cluster, how it works

On Windows it is not yet possible to set up a named pipe server in a worker, and Node.js does not manage the worker count for you: if no workers are alive, existing connections are dropped and new ones refused. Everything that follows is the code that closes those gaps.

The smallest useful cluster forks one worker per core and lets each worker run an ordinary HTTP server. The response tells you which worker answered, and Connection: close makes every curl request open a new connection so the balancing is visible.

Source: Node.js API: cluster.fork(), Node.js API: cluster, Event: 'listening', Node.js API: cluster, Event: 'exit'

Started with three workers and queried six times:

Source: Node.js API: Cluster

The 'listening' event on cluster fires in the primary when a worker’s server is really accepting, with the address in its second argument; 'online' fires earlier, when the worker process has merely started, and 'fork' earlier still. Use 'listening' as your readiness signal, as the rolling restart below does. Output order differs between runs and between Node.js lines, which is observed behaviour and no cause for concern.

The cluster module supports two ways of distributing incoming connections, selected through cluster.schedulingPolicy:

The setting is global and effectively frozen once the first worker is forked or setupPrimary() is called, so set it first. It can also come from the environment variable NODE_CLUSTER_SCHED_POLICY with the values rr and none, which is handy for experiments because it needs no code change. The next program counts requests per worker by sending each one to the primary over IPC, and prints the tally when it receives SIGTERM.

Source: Node.js API: worker, Event: 'message', Node.js API: worker.send(), Node.js API: process.send(), Node.js API: cluster.schedulingPolicy

It was driven with 8 persistent connections for 3 seconds, once per policy, on a 10-core macOS machine with the load generator on the same host:

Those numbers are observations on one machine, not a guarantee: round-robin was even every time, while SCHED_NONE was skewed on 24 and 22 and even on 26 in this setup. That matches the documented warning, and it is the reason to keep the default unless a measurement on your own hardware says otherwise. Round-robin is balanced by connection, not by work: a worker holding a few long keep-alive connections that each issue expensive requests is as loaded as it can get while the count looks even. If your load is made of a few long-lived connections (WebSockets, HTTP/2 or gRPC channels), put a load balancer that balances per request in front, or use more connections than workers.

Source: Node.js API: cluster.schedulingPolicy

The standard answer is one worker per core, and the right way to ask for the core count is os.availableParallelism(). The documentation describes it as an estimate of the default amount of parallelism a program should use, always greater than zero, and states that os.cpus().length should not be used for this. The difference shows up in containers, where the CPU limit is a quota the host enforces and not a smaller machine. These values were measured on a 10-core host with Docker, using the official node images:

In every case os.cpus().length stayed at 10. The quota is rounded down, never below one on 26 and 24, while the 22.23.3 image reported the whole host for a quota under one CPU, so do not take the number as a given on an older line and set the worker count explicitly through an environment variable in containers (the examples read WORKERS). One worker per core is a starting point, not a law: workers that mostly wait on a database leave a core partly idle, workers that use native add-ons or the thread pool compete for it, and every worker costs memory (an idle Node.js worker with a small app was around 80 MB resident in the PM2 run below).

Source: Node.js API: os.availableParallelism(), libuv: miscellaneous utilities, uv_available_parallelism()

The IPC channel between primary and worker is the same one child_process.fork() creates. worker.send(message[, sendHandle[, options]][, callback]) sends from the primary to one worker; inside a worker process.send() sends to the primary. Both sides receive with the 'message' event: on the worker as process.on('message'), on the primary as worker.on('message') for one worker or cluster.on('message') for all of them. The ipc.js program above uses all four directions: each worker reports a hit, and the primary broadcasts a message by looping over cluster.workers, which is the documented way to address every worker.

Messages are serialized and parsed, so the receiver gets a copy and never a reference. The default serialization is JSON, which drops undefined, functions and cycles, and turns a Map or a Date into something else; serialization: 'advanced' in setupPrimary() switches to the node:v8 structured clone format. worker.send() returns false when the channel is closed or the backlog of unsent messages is too large, and its optional callback is the flow-control hook: pass it, or an unsent message on a dead worker raises an 'error' event. A cmd or type field is only a convention of your own, with one exception: Node.js reserves messages whose cmd property starts with NODE_ for itself (they surface as 'internalMessage' events) and says applications should avoid them.

IPC is the right tool for control traffic: "drain now", "reload configuration", a periodic metrics snapshot. It is the wrong tool for the hot path. In ipc.js every request sends one message to the primary, which means the single-threaded primary does work proportional to the whole cluster’s traffic; the demonstration only works at all because the message is tiny. Aggregate in the worker and report on a timer. Do not use it as a database either: a message from one worker to another has to travel through the primary, and nothing queues it if the target dies. For shared data use an external store (see State outside the process), and for sharing memory between threads of one process use SharedArrayBuffer in Worker Threads.

Source: Node.js API: worker.send(), Node.js API: subprocess.send()

Workers die: an uncaught exception, an out-of-memory kill, a native crash, or a deliberate process.exit(). The cluster module will not replace them, and when the last one is gone the port stops answering. The primary therefore has to react to the 'exit' event, and the first decision in that handler is whether the death was asked for. worker.exitedAfterDisconnect is true when the worker left because someone called worker.disconnect() (or cluster.disconnect()), false for any other exit, and undefined while the worker is still running, so the handler can respawn accidents and leave deliberate stops alone.

Source: Node.js API: worker.exitedAfterDisconnect

Respawning immediately is a trap. A worker that crashes at startup (a bad configuration, a database that is down) would be forked, die and be forked again in a tight loop that burns a core and floods the logs. The primary below counts consecutive crashes and doubles the delay up to five seconds, resetting once a replacement is really listening. A worker should also not try to survive an uncaught exception: its state is unknown, so log it, stop accepting work and exit, and let the supervisor start a clean one (see Errors & Error Handling).

The pair of files below is the supervisor used for the rest of the page. The worker, server.js, is a plain server that also knows how to drain; the primary, primary.js, never loads application code, because setupPrimary({ exec }) points the workers at a different file. That separation matters for deployments: when new code is on disk, every worker forked afterwards runs it, while the long-lived primary keeps its old code in memory and does not need to change.

Source: Node.js API: cluster.worker, Node.js API: process, Event: 'message', Node.js API: http.createServer()

Run with three workers, a request to /crash takes one worker down and the primary replaces it after 100 ms; the other two keep answering meanwhile:

Source: Node.js API: Cluster events, Node.js API: cluster.setupPrimary(), Node.js API: url.fileURLToPath()

A rolling restart replaces the workers one at a time so that some workers are always accepting connections: after a deploy, a configuration change or a memory-leak cleanup, the pool turns over without a gap. The figure shows the sequence the rollingRestart() function in primary.js follows.

The design choices in primary.js, each of which has a reason:

The trigger is the signal SIGUSR2. SIGUSR1 is reserved by Node.js to start the inspector, and Windows does not support signals, so there the trigger has to be something else: a message from a deploy script over a loopback-only admin socket, a watched file, or a stdin command. Run it like this:

Source: Node.js API: Signal events, Node.js API: worker.disconnect()

To check the claim, not just state it, this client fires a 300 ms request on a fresh connection every 50 ms for seven seconds while the restart runs, and counts what came back:

Keep-alive clients are a different and honest caveat. The same restart under autocannon with 20 persistent connections (about one million requests in six seconds) returned no non-2xx response but between 31 and 39 ECONNRESET errors in repeated runs, and 16 out of about 12,000 at a fixed 2,000 requests per second. The cause is the unavoidable race of HTTP/1.1 keep-alive: a connection that is idle at the instant the old worker closes it can receive a request from the client in the same moment, and that request gets a reset. HTTP lets a client retry an idempotent request on a reused connection that fails this way, and a reverse proxy normally does so; no purely in-process scheme removes the race, which is why an orchestrator or a proxy that retries is the complete answer (see Cluster, PM2 or container replicas?).

Two operational details complete the picture. First, a rolling restart replaces the workers, not the primary: a change to the primary’s own code needs a restart of the primary, which is downtime unless something in front routes around it, so keep the primary small and stable (the reason for exec). Second, the same drain path serves SIGTERM and SIGINT for a normal shutdown, and note that a terminal’s Ctrl+C is delivered to the whole process group, so workers also see SIGINT unless you handle or block it; the signal rules are in Process, Signals & Exit Codes and the shutdown sequence in Production Deployment.

Source: Node.js API: worker.disconnect(), Node.js API: Signal events, Node.js API: server.close()

Each worker has its own memory, and Node.js provides no routing logic of its own, so the documentation advises designing the application so that it does not rely heavily on in-memory data for things like sessions and login. Whatever one worker stores in a variable, the next request may land on a worker that has never seen it. These are the usual victims:

The next program shows the problem and one fix side by side. Each request increments a counter held in a variable and a counter held in a SQLite database file with node:sqlite, which every worker opens (see SQLite Built In). node:sqlite is Stability 1.2 (Release candidate) on Node.js 26 and needs no flag on 24 and 22; on 22.23.3 it printed an ExperimentalWarning once per process in the runs for this page, and 26 and 24 printed none. A file database only works while the workers share a filesystem, so it is a demonstration for one machine; across machines the same idea needs a networked store.

After 100 requests spread over four workers, two more requests returned:

The in-memory counter reflects only the traffic one worker saw; the shared counter is the truth. Write contention is the price: busy_timeout makes a writer wait for the lock instead of failing at once, and WAL mode lets readers proceed during a write. For anything beyond a demo, use a server database or Redis and the patterns in Databases from Node.js. Keeping workers stateless is good practice even for a single-core service, because it is also what makes an orchestrator able to start, stop and move instances freely.

Source: Node.js API: SQLite, Node.js API: Cluster, how it works

Sometimes a client must keep reaching the same worker: a protocol that spreads one logical session over several HTTP requests with in-memory state (long-polling transports of some real-time libraries), a legacy application that cannot yet externalize its sessions, or per-user data held in a worker on purpose. The cluster module has no built-in affinity, and the round-robin policy deliberately ignores who the client is. The right pattern takes the connection away from the cluster’s own balancing and routes it yourself:

Source: Node.js API: net.createServer(), pauseOnConnect, Node.js API: worker.send(), Node.js API: child_process, sending a socket object

The two clients, an IPv4 and an IPv6 address, each stayed on one worker over four separate connections. When a worker was killed with kill -9, its slot was refilled by a new worker (id 5 in that run) and the client was served by it, with the old worker’s in-memory state gone. It behaved the same on Node.js 26, 24 and 22.

Source: Node.js API: net.createServer(), Node.js API: subprocess.send(), sending a socket object

What makes this correct, and what usually goes wrong:

The book Efficient Node.js (Samer Buna, O’Reilly 2025) presents IP hashing as a simple option, and it is, but see the outdated-material section for two defects in how such an example is easy to get wrong.

PM2 is a process manager that daemonizes your application, restarts it when it dies, collects logs and, in its cluster mode, runs several instances of one script through node:cluster so you do not write a primary yourself. The npm metadata for the version used here (7.0.4) lists the license as AGPL-3.0, so check that against your distribution model before you ship it inside a product. Because it is the cluster module underneath, everything above applies: the instances do not share memory, round-robin is the default and the application has to be stateless.

An ecosystem file declares the application once. wait_ready makes PM2 wait for the instance to call process.send('ready') before treating it as up, which is what turns pm2 reload into a rolling restart.

Source: PM2: Graceful start and shutdown

Source: PM2: Ecosystem file, PM2: Cluster mode

With PM2_HOME pointed at a scratch directory, pm2 start brought up three instances (about 80 MB each), and running the same slowload.mjs client during pm2 reload bookshelf-api produced 138 responses with status 200 and no errors; pm2 ls afterwards showed three instances with a restart count of 1 and new process ids. PM2’s strengths are a quick start on a single server and its operational tooling; its costs are another daemon to run and secure, behaviour that follows PM2’s own defaults (it sends SIGINT first and SIGKILL after kill_timeout), and an overlap with what systemd or an orchestrator already provides. In a container, one process per container is normally the better fit, as the next section explains.

Source: PM2: Cluster mode, PM2: Graceful start and shutdown, PM2: Ecosystem file

The same goal, more capacity and no downtime, can be reached at different layers. Containers and orchestrators turned the cluster module from a default into a choice.

On an orchestrator, run one Node.js process per container, give the container a CPU limit, and let the platform provide what this page built by hand. The orchestrator replaces primary.js: a Deployment starts a new replica, waits for its readiness probe, removes an old replica from the Service, sends SIGTERM and waits for the termination grace period before SIGKILL (see Deployment Strategies, Health Probes & Lifecycle and ReplicaSets & Deployments). The Node.js side of that contract is the SIGTERM handler from primary.js applied to one process: stop accepting, finish in-flight requests, close pools, exit (see Production Deployment and Docker & Containers). Replicas also give the proxy the chance to retry the keep-alive resets described earlier, and scale across machines with Autoscaling. Resource limits per replica are covered in Resource Management and, for plain Docker, in Resources, Logging & Monitoring; the NestJS equivalents are in NestJS Deployment and Lifecycle Events & Shutdown.

Cluster inside a container still has a place: a container with a CPU limit above one core that you want to saturate, or a platform that bills per container and has a high per-instance minimum. In that case size the worker count from availableParallelism() or an explicit variable, as measured above, and keep the memory limit in mind, because every worker has its own heap and a single container-wide limit is shared by all of them.

Source: Node.js Learn: Comparing Node.js concurrency models, Kubernetes: Deployments, Docker: Resource constraints

A scaling change you have not measured is a guess. Load testing answers three different questions, and conflating them produces misleading results: how much throughput one configuration handles (capacity), how latency behaves as load rises (the shape under stress), and whether a restart, deploy or failure disturbs real traffic (the resilience tests above).

Good practice follows from how the tools work:

autocannon is an HTTP benchmarking tool written in Node.js that can drive HTTP/1.1 with pipelining and many connections from a single process, from the command line or as a library. The commands used for the measurements above are the whole interface you need to start:

Source: autocannon on GitHub

Artillery describes a test as a scenario in YAML: phases that set the arrival rate over time, flows that describe what each virtual user does, and checks that fail the run. The plugin ensure turns the thresholds into an exit code, so the same file works as a CI gate.

Source: Artillery: Test script reference, Artillery: ensure plugin

With the threshold deliberately changed to 0.1 ms the same run printed fail: http.response_time.p99 < 0.1 and exited with status 1, which is the behaviour a pipeline relies on. Run the restart test from the earlier section while the load test is running to measure what a deploy costs you: errors during the roll are the number to watch. Use autocannon for quick checks of one endpoint and Artillery when you need scenarios, ramps and thresholds in version control; both are devDependencies and never belong in the production image.

Source: autocannon on GitHub, Artillery: Test script reference, Artillery: ensure plugin

Source: Node.js API: Cluster, Node.js API: os.availableParallelism(), Node.js API: server.close()

The scaling chapter of Efficient Node.js (Samer Buna, O’Reilly 2025) uses current names (isPrimary, availableParallelism()), which is a good sign; the gaps are in design rather than in names. Older articles and tutorials still teach cluster.isMaster, setupMaster() and os.cpus().length, so check any snippet against the stability table above. Points to treat with care when following that chapter:

Wexler’s Node.js Projects (Jonathan Wexler, O’Reilly 2025) mentions clustering only in concept as part of the runtime overview, so nothing in it needs correcting for this topic; the practical cluster material is the chapter above.

Decide between scaling mechanisms with Concurrency Models; keep a single process healthy with Don’t Block the Event Loop and Profiling & Performance; handle signals and shutdown with Process, Signals & Exit Codes and Production Deployment; and put the whole thing into an image with Docker & Containers.