RabbitMQ Reliability, Dead-Lettering and TTL

A TCP connection can be lost silently — no FIN, no RST, just packets that stop arriving, which the operating system itself can take a long time to notice (on Linux, roughly eleven minutes with default settings). RabbitMQ’s AMQP 0-9-1 heartbeat feature exists specifically to detect that faster at the application layer: client and broker each negotiate a heartbeat timeout at connection time (the greater of the two configured values when either side requests 0, defaulting to 60 seconds), send an empty heartbeat frame at half that interval whenever no other traffic has gone by, and consider the connection dead once more than two consecutive intervals pass with neither a heartbeat frame nor any other data received.

Most client libraries, including the RabbitMQ Java client, layer automatic connection recovery on top of that detection: once a connection is confirmed lost, the client reconnects, and re-declares the topology it remembers having declared (exchanges, queues, bindings) along with re-registering its consumers — transparently enough that application code often does not need its own reconnect loop. What automatic recovery does not do on its own is resume in-flight work exactly where it left off: any channel open on the failed connection is gone and has to be re-opened, and any unacknowledged deliveries that were in flight at the moment of failure are redelivered fresh once the consumer reconnects, which is exactly why the next section’s acknowledgement discipline matters as much as recovery itself.

See RabbitMQ Documentation — Reliability Guide for the full recovery contract, and RabbitMQ Documentation — Detecting Dead TCP Connections with Heartbeats and TCP Keepalives for heartbeat negotiation and tuning.

At-least-once delivery is not one mechanism — it is two, one on each side of the broker, and both have to be in place for the guarantee to hold end to end. On the publishing side, Publishing to RabbitMQ's publisher confirms tell a publisher whether the broker actually accepted a message, so a publisher that never sees a confirm knows to retry rather than assume success. On the consuming side, Consuming from RabbitMQ's manual acknowledgement (basic.ack/ basic.nack/basic.reject) tells the broker whether a consumer actually finished processing, so a message stays redeliverable until it does. Drop either half — automatic acknowledgement on the consumer side, or publishing without confirms on the producer side — and the guarantee degrades to at-most-once for that half of the pipeline: a message can be lost during publish, or lost if a consumer crashes mid-processing after auto-acking, even while the other half still behaves correctly.

A dead-letter exchange (DLX) is an ordinary exchange a queue is configured to republish certain messages to instead of discarding them outright, via the x-dead-letter-exchange queue argument (and, optionally, x-dead-letter-routing-key to override the routing key used on republish — the message’s original routing key is reused by default). Exactly four reasons cause a message to be dead-lettered, each recorded as a standardized reason value in the message’s death history (the x-death header on AMQP 0-9-1, or the x-opt-deaths annotation on AMQP 1.0 — both carrying the source queue, reason, count, and timestamp for every dead-lettering event a message has been through):

Ordinary dead-lettering is, itself, only at-most-once: the broker removes the message from the source queue and republishes it to the DLX without an internally confirmed hand-off between the two, so a broker crash landing exactly between those two steps loses the message. Quorum queues can close that gap: setting a source queue’s x-dead-letter-strategy to at-least-once (together with x-overflow set to reject-publish and the stream_queue feature flag enabled) makes the source queue keep its own copy of a dead-lettered message, internally confirmed by publisher-confirms-like bookkeeping against the target queue, until that target queue has actually stored it — at the cost of the source queue briefly holding two copies of the same message. Classic queues and streams do not support this stronger mode at all; only a quorum queue’s own dead-lettering can be upgraded to at-least-once.

A message’s time-to-live can be set two ways, and they compose rather than override each other outright: a per-queue TTL (x-message-ttl, milliseconds, applied via policy or at declaration) bounds every message in that queue uniformly, and a per-message TTL (the expiration property, set by the publisher per message) bounds just that one message. When both are set on the same message, the lower of the two wins.

TTL expiry is checked lazily, only at the head of the queue — an expired message sitting behind still-unexpired ones is not discarded the instant it expires, only once it would actually be the next one delivered (or, for a classic queue, also when certain policy changes touch the queue). A queue can therefore briefly report more ready messages than are actually still live, and a long-expired message parked behind a slow-moving head can sit around far longer than its TTL would suggest.

x-max-length (message count) and x-max-length-bytes (total body size) cap how large a queue’s ready backlog is allowed to get — unacknowledged, in-flight deliveries do not count against either limit. Whichever limit is reached first is the one enforced, and what happens next depends on the queue’s overflow setting:

A priority queue is declared with x-max-priority (an integer ceiling on the priority property a message may carry — RabbitMQ’s own guidance is to keep this small, roughly 1 to 10, since each extra priority level adds broker-side bookkeeping cost). Within such a queue, higher-priority ready messages are delivered ahead of lower-priority ones, though ordering is only a best-effort preference, not a hard real-time guarantee — a message already being delivered is not preempted by one of higher priority arriving after it.

RabbitMQ has no single, long-standing broker-native way to delay a message’s first delivery, and what exists here is worth checking against the current docs rather than assuming, since it has changed more than once:

Before 4.3’s native delayed retries (above), and still the portable approach on classic queues or on RabbitMQ versions without that feature, a delayed-retry effect is built by hand from two ordinary primitives already covered on this page: a per-queue TTL, and a dead-letter exchange pointed back at the original work queue. A message that fails processing is published (or dead-lettered) to a wait queue with no consumers and a fixed TTL instead of being retried immediately; once that TTL elapses, the wait queue’s own DLX configuration dead-letters it straight back into the work queue’s exchange, where an active consumer picks it up again — the TTL wait becomes the retry delay, entirely without the consumer sleeping or polling.

Each retry attempt still needs a bounded number of hops — typically tracked from the x-death header’s own count field, or, on a quorum queue, simply left to the built-in x-delivery-limit — so a message that keeps failing eventually lands in a genuine dead-letter queue for manual inspection instead of cycling forever.