RabbitMQ Clustering, Federation and Shovel

A RabbitMQ cluster is a set of nodes — Erlang VM instances, each named rabbit@hostname — that share users, virtual hosts, permissions, exchanges, queue and binding topology, and runtime parameters and policies. Nodes discover and join each other via peer discovery (a static list of node names, or a plugin backed by DNS, Kubernetes, AWS, or Consul) and communicate over Erlang distribution, which by default requires unobstructed TCP connectivity between every pair of nodes on the epmd port (4369) and a distribution port range (25672 and up) — clustering is not designed to tolerate a high-latency link between nodes the way federation and shovel are (below).

Queue contents are not automatically cluster-wide just because the nodes are clustered: a classic queue still lives on the single node that declared it (any other node in the cluster can still accept operations on it and proxy them there), while quorum queues and streams place their replicas on a configurable subset of nodes, each with its own Raft group — clustering is the substrate those replication types run on top of, not a replication mechanism by itself.

Cluster-wide metadata — virtual hosts, users, permissions, queue/exchange/binding topology, runtime parameters and policies — has to be kept consistent across every node, and RabbitMQ’s mechanism for that changed in 4.0. The long-standing store, Mnesia, used its own eventually-consistent-under-partition replication with ambiguous, difficult-to-reason-about behaviour when a network partition actually occurred. Khepri, RabbitMQ’s current metadata store, replaces it with the same Raft consensus algorithm quorum queues already use: cluster metadata is itself a replicated Raft log, so every node’s view of topology, users, and policies agrees by construction, and how a partition is handled is a fixed, well-defined consequence of Raft rather than a configurable, ambiguous choice. Migrating an existing Mnesia-based cluster to Khepri is a one-way, feature-flag gated operation (khepri_db); new clusters on current RabbitMQ releases are expected to run on Khepri going forward.

RabbitMQ Queue Types already covered the Raft leader/follower mechanics of a quorum queue and the append-only replicated log of a stream; the cluster-level question is what a node loss does to them. Because each quorum queue (and each stream) is its own independent Raft group with members on a subset of the cluster’s nodes, losing a minority of a given queue’s members is transparent to clients — the remaining majority still holds quorum, a new leader is elected if the lost node held leadership, and publishing/consuming continues uninterrupted, modulo the brief unavailability of an in-flight election. Losing a majority of a queue’s members, on the other hand, leaves that queue without quorum: it stops accepting new writes (and, depending on how the loss happened, may need manual intervention — forced Raft-membership change — to recover) even though the cluster as a whole may still have plenty of healthy nodes. Classic queues have no replication at all to lose gracefully: a node holding one going down simply makes that queue’s messages unavailable (or gone, if the node’s storage is not recovered) until the node returns, exactly as RabbitMQ Queue Types already described.

A client should not need to know, and generally should not hard-code, which specific node it happens to connect to — a load balancer (or a DNS record round-robining across nodes) placed in front of a cluster gives clients one stable address while spreading connections across every available node, and, just as importantly, routing new connection attempts away from a node that has gone down. Two things need to line up for this to work well: the load balancer’s own health check should query each node’s actual readiness (RabbitMQ’s health-check endpoints, covered on RabbitMQ Operations, Monitoring and Security) rather than just a bare TCP connect, since a node can accept a TCP connection while still being unable to serve AMQP traffic correctly; and a client with automatic connection recovery (see RabbitMQ Reliability, Dead-Lettering and TTL) reconnecting through the same load balancer address will simply be handed whichever node is currently healthy, without any cluster-awareness of its own.

Federation links independent clusters — clusters that do not share Erlang distribution or Khepri/Mnesia metadata, and may be geographically distant or connected only by a higher-latency link — by copying messages from a downstream federation link’s local exchange or queue to an upstream cluster’s exchange or queue, asynchronously, over an ordinary AMQP connection rather than cluster-internal Erlang distribution. A federated exchange re-publishes messages arriving at an upstream exchange to the local (downstream) exchange of the same name, so consumers bound locally receive copies of messages published anywhere in the federation without the publisher needing to know which cluster its consumers actually live in. A federated queue does the equivalent for a single named queue, effectively presenting one logical queue’s contents across multiple clusters. Because federation moves messages over a normal client connection rather than requiring cluster membership, it tolerates a much less reliable link between sites than clustering does, at the cost of weaker guarantees — a federation link buffers and retries on reconnect, but does not give the same all-or-nothing consistency a single cluster’s own replication does.

The shovel plugin is a simpler, more explicit tool for the same broad problem: it runs a worker that consumes from a named source queue (on any broker, including a non-RabbitMQ AMQP 0-9-1 or AMQP 1.0 peer) and republishes each message to a destination exchange or queue (on any broker), with configurable acknowledgement mode and reconnect behaviour. Shovels come in two flavours: a dynamic shovel, defined and managed at runtime as a parameter (the common case, changeable without a broker restart), and a static shovel, defined once in advanced.config and requiring a restart to change. Unlike federation — which is topology-aware and mirrors an exchange or queue’s traffic as a standing relationship — a shovel is a single, named point-to-point pipe between one source and one destination, making it the more natural fit for a one-off or clearly bounded migration/integration link rather than an always-on many-to-many topology.

A single cluster (with quorum queues or streams for replication) is the right choice whenever every node can share a reliable, low-latency network — typically the same data center or availability-zone group — because clustering’s Raft-based replication assumes exactly that and is not designed to tolerate a flaky or high-latency link between members. Once nodes need to span sites that do not offer that kind of link — across regions, across data centers connected only by a WAN, or across organizational boundaries — federation or shovel take over: both move messages asynchronously over ordinary client connections instead of requiring consensus across the link, at the cost of weaker cross-site delivery guarantees than a single cluster provides internally. Between the two, federation is the better fit for an ongoing, topology-shaped relationship (mirror this exchange’s or this queue’s traffic to every linked site, indefinitely), while shovel is the better fit for an explicit, bounded pipe between one source and one destination — migrating data during a broker upgrade, or integrating with a specific external system — where a full federation link would be more topology than the job actually needs.

Warm standby replication — continuously replicating both schema definitions and enqueued messages from an upstream (primary) cluster to a downstream (standby) cluster purely for disaster recovery, so an operator can promote the standby quickly with minimal data loss if the primary is lost entirely — is a commercial feature of Tanzu RabbitMQ (VMware/Broadcom’s commercial distribution), not part of open-source RabbitMQ. It solves a narrower problem than either federation or shovel: those two move live application traffic between clusters that are both actively serving clients, while warm standby replication exists purely to keep an otherwise-idle standby cluster ready to take over.

Running a cluster on Kubernetes by hand — coordinating StatefulSet rollout order, persistent volumes, and peer-discovery configuration for every node — is the same substantial operational burden Kafka Operations and Monitoring describes for a self-managed Kafka cluster, and RabbitMQ’s own guidance is correspondingly to avoid doing it directly. The RabbitMQ Cluster Operator manages a cluster as a RabbitmqCluster custom resource — provisioning the StatefulSet, service, persistent storage, and default users a cluster needs, and reconciling it the way any Kubernetes operator does — and the companion Topology Operator extends the same declarative model to virtual hosts, users, permissions, queues, exchanges, bindings, shovels, and federation links as their own custom resources, so a cluster’s entire topology (including the federation and shovel links this page just covered) can be managed as Kubernetes manifests rather than imperative API calls. Both are the officially recommended way to run RabbitMQ on Kubernetes, ahead of either a hand-rolled deployment or most third-party Helm charts.