The Transactional Outbox Pattern

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production; see the section bibliography for the reference material consulted while preparing these pages.

Domain Events ends with a promise this page keeps: once Order.place(command) returns the events it raised, PlaceOrderService (Structuring the Code) has to actually get OrderPlaced out of the process reliably, and "reliably" here means something precise — the event must be published if and only if the aggregate’s own database write commits, with no window in which one happens without the other. This page is about the pattern that closes that window: the transactional outbox, shown with the same Java 21 / Spring Boot 4.1.x / Spring Data JPA / Spring Modulith 2.x / Spring for Apache Kafka 4.1.x baseline as the rest of this section, and reusing the exact Order, OrderCreation, OrderPlaced, DomainEventEnvelope, PlaceOrderService, and OrderRepository shapes Structuring the Code, Aggregates, and Domain Events already established.

The dual-write problem, stated precisely

PlaceOrderService.placeOrder has to do two things that look independent but are not: save the new Order row, and tell the rest of the system OrderPlaced happened. The naive way to do both is two separate steps — commit the database transaction, then call a Kafka producer (or an HTTP client, or any other broker call) to publish the event — and Why Not Distributed Transactions already explains in depth why wrapping both steps in one distributed transaction is not the fix: most brokers are not XA resource managers a two-phase-commit coordinator can enlist, and even where a coordinator could technically be built, the availability and lock-holding costs it introduces are worse than the problem it solves. Without a distributed transaction, "commit, then publish" as two independent steps has exactly two ways to fail, and Richardson’s Microservices Patterns 3.3.5 names both precisely:

  • Commit, then crash before publish. The database transaction commits — Order is real, durable, and visible to any other transaction that queries it — and the process dies (an out-of-memory kill, a deployment restart, a network partition to the broker) before the publish call ever goes out. Order exists; nothing else in the system ever learns it does. Inventory never reserves stock for it, Payment never authorizes a charge for it, and the order silently stalls forever in APPROVAL_PENDING, with no error raised anywhere to say why.

  • Publish, then roll back. The publish call goes out first (or the database write is attempted after the publish, on the theory that publishing "can’t hurt" if the save fails later), and then the database transaction fails or rolls back for a reason that has nothing to do with messaging — a constraint violation, a concurrent @Version conflict, the process crashing mid-transaction. Now OrderPlaced is on the broker, Inventory reserves stock and Payment authorizes a charge for an order that, from the Order context’s own database, was never actually placed.

Reordering the two steps only trades one failure mode for the other; there is no ordering of "write to the database" and "call the broker" as two independent operations that closes the gap, because nothing links their outcomes together. The fix does not add a coordinator between two systems — it removes the second system from the equation entirely, for exactly as long as it takes to make the write durable.

The outbox table

The transactional outbox pattern turns the event into an ordinary row, written to an outbox table in the same local database transaction as the aggregate’s own write. Because both writes go through the same JPA EntityManager inside the same @Transactional method, they commit or roll back together as a single unit of work — the same local ACID guarantee PlaceOrderService.placeOrder already relies on for the Order row alone, now covering the outbox row too. There is no dual write left to fail, because there is only one write: one database transaction, two tables.

The canonical schema this section uses everywhere an outbox is shown:

Column Purpose

id

Primary key of the outbox row itself (a generated UUID). The publisher below reuses it as the eventId of the envelope it serializes into payload, so the row id and the key every consumer deduplicates on are one value.

aggregate_type

The aggregate’s type name ("Order"), letting a generic relay route or log a row without deserializing payload — the same purpose DomainEventEnvelope.aggregateType serves in-process (Domain Events).

aggregate_id

The specific aggregate instance’s id (the OrderId value) — also doubles as the natural ordering and partition key for the relay, covered below.

event_type

The event’s type name ("OrderPlaced"), letting a consumer dispatch to the right handler before deserializing the full payload.

payload

The event’s data, serialized (JSON is the common choice) — the fields OrderPlaced itself carries.

headers

Serialized metadata that travels alongside the payload but is not part of the business fact — causationId, correlationId, content-type, schema version — mirroring DomainEventEnvelope’s metadata fields without forcing every consumer to parse them out of `payload itself.

created_at

When the row was written, in the same transaction as the aggregate change — the timestamp a monitoring query uses to compute relay lag (see below).

published_at

NULL until the relay has successfully handed the row to the broker, then set to the time it did — the single column that separates "not yet sent" from "sent," whether the relay marks rows in place or deletes them entirely (see below).

Writing this row is the entire scope of the transaction as far as messaging is concerned: nothing in PlaceOrderService.placeOrder calls a broker client directly, waits for a network round trip, or can fail because Kafka happens to be unreachable at that instant. The row either commits with Order, or neither does.

Getting the row to the broker: polling publisher vs. transaction-log tailing

Writing the outbox row only gets an event as far as the local database — something still has to move it onto the broker, and Richardson’s Microservices Patterns 3.3.5 names two structurally different ways to do that.

The polling publisher

A polling publisher is a separate process (or a scheduled job inside the same service) that periodically queries the outbox table for unpublished rows, publishes each one, and then marks or deletes it. Four details decide whether a polling publisher behaves well in production:

  • Batch size. Reading and publishing rows one at a time round-trips the database and the broker once per event; reading a bounded batch (SELECT …​ ORDER BY created_at LIMIT 100) amortizes both round trips across many events, at the cost of a slightly larger window between a row being written and being picked up.

  • Ordering by aggregate key. Two events about the same aggregate (OrderPlaced then OrderApproved for the same orderId) must reach the broker, and any one partition, in the order they were raised — exactly the ordering requirement Domain Events already states for consumption. A polling query ordered by created_at, publishing to a Kafka topic keyed by aggregate_id, keeps that guarantee: Kafka’s own partition-by-key delivers same-key records to the same partition in production order, so as long as the poller reads and publishes in created_at order per key, the partition preserves it.

  • At-least-once, and the duplicates it implies. A crash between "the broker acknowledged the publish" and "the row got marked or deleted" makes the next poll pass republish the same row — there is no way to make "publish, then mark" atomic across a broker and a database any more than the original dual write was atomic, which is exactly why every consumer of an outbox-relayed event must be an idempotent consumer (below), not an optional nicety.

  • Poll interval and staleness. A short interval lowers the delay between commit and publish at the cost of more idle queries when the outbox is empty; a longer interval batches better but widens the window in which other services have not yet learned about a change that already committed.

Transaction-log tailing / CDC

Transaction-log tailing — change data capture (CDC) — avoids polling the outbox table with application queries at all: a connector reads the database’s own write-ahead or binary log directly, the same log the database uses for its own crash recovery and replication, and turns each row inserted into outbox into an event on the broker as a side effect of that log already existing. This is strictly lower-latency than polling (events appear as soon as the log record is flushed, not on the next poll tick) and adds zero read load to the outbox table itself, at the cost of running the CDC infrastructure (a Debezium connector and a Kafka Connect cluster) as an operational dependency. Kafka Connect already covers Debezium’s outbox event router SMT in depth — how it derives the destination topic from aggregate_type, the record key from aggregate_id, and the record value from payload, so a raw "row inserted into the outbox table" change event becomes a clean domain event on a topic named after its aggregate — and that configuration is not repeated here; this page only needs the trade-off between the two relay strategies, not the connector’s setup.

Neither strategy changes anything about the outbox table itself or about PlaceOrderService: both read the same schema, and the choice between them is purely an operational one — whether the team already runs Kafka Connect and Debezium, how much relay latency and the resulting write load on the outbox table are acceptable, and how much additional infrastructure the team is willing to operate for the lower latency and zero-added-read-load CDC gives.

The NoSQL variant, and Spring Modulith’s outbox built into the framework

Where the aggregate itself lives in a document store rather than a relational table, there is often no separate outbox table at all: the pending events are embedded as a field directly inside the aggregate’s own document (an pendingEvents array alongside Order’s other fields), written atomically with the rest of the document by the same single-document write every document database already guarantees, and a relay process (polling the collection for documents with a non-empty `pendingEvents array, or tailing the database’s own change stream) drains and clears that array the same way a relational poller drains and marks outbox rows — the mechanism is the same pattern, just folded into the aggregate’s own storage instead of a sibling table.

Spring applications already running Spring Modulith’s application-events and event-externalization support get a comparable guarantee for free, without hand-rolling an outbox table at all: Modulith’s event publication registry persists every published application event to its own table in the same local transaction as the code that published it, tracks which listeners have completed processing it, and republishes any event whose listener never finished — for instance because the process crashed mid-delivery — on the next restart or on a configurable schedule. Combined with spring-modulith-events-kafka or spring-modulith-events-amqp, the same registry also externalizes a @Externalized-annotated event to a broker topic, which makes the registry, structurally, an outbox the framework maintains: one local transaction writes the aggregate and the pending-publication record together, and Modulith’s own infrastructure plays the relay’s role instead of a hand-written @Scheduled poller. This page does not re-teach @Externalized or EventExternalizationConfiguration — Spring Boot’s other messaging options already covers both — only that the registry itself is the outbox pattern realized as a framework feature rather than application code.

The idempotent consumer, and operational concerns

Every relay strategy above delivers at-least-once, never exactly-once, so the receiving side has to be prepared for the same event arriving more than once — the same idempotent-consumer requirement Domain Events already states for consumption in general, worth restating here specifically because it is the outbox’s unavoidable other half, not an optional hardening step. Two shapes of idempotency apply, and either is sufficient on its own:

  • A dedup table. Before applying an event’s side effect, check whether its id is already recorded as processed; apply the effect and record the id in the same local transaction, so a redelivery after a crash between "applied" and "acknowledged" is a safe no-op. The code below shows this as an INSERT …​ ON CONFLICT DO NOTHING upsert rather than a plain existence check, closing the same race between two concurrent deliveries Domain Events's processed_events example leaves as a simpler existsById guard.

  • Natural idempotency. Some operations are idempotent by construction and need no dedup table at all — an UPSERT keyed on orderId that sets `OrderHistoryView’s fields from the event’s payload converges to the same row no matter how many times it runs, because "set these columns to these values" has no notion of "already did this once."

Running an outbox relay in production also raises three concerns that only show up once real traffic flows through it:

  • Outbox growth and cleanup. A polling publisher that marks rows as published (published_at set) rather than deleting them needs a separate, unrelated cleanup job — a scheduled delete of rows older than a retention window — or the table grows without bound; deleting the row outright once it is confirmed published avoids a second job at the cost of losing the row as an audit trail.

  • Ordering per partition key. Ordering is guaranteed only within one aggregate_id’s stream of events, as stated above — a relay or broker configuration that shards work across multiple partitions or worker threads must still route every row for one `aggregate_id to the same partition, or the ordering guarantee silently disappears the moment throughput needs more than one worker.

  • Monitoring lag. The gap between created_at (when the row was written) and published_at (when it left the outbox) is the single most useful outbox metric to alert on: a growing gap means the relay is falling behind, stuck, or down entirely, and it is detectable directly from the outbox table itself (SELECT max(now() - created_at) FROM outbox WHERE published_at IS NULL) without needing broker-side tooling at all.

Code: the outbox DDL, the publisher, the relay, and a deduplicating consumer

The outbox table itself, portable SQL matching the schema above:

CREATE TABLE outbox (
    id             UUID PRIMARY KEY,
    aggregate_type VARCHAR(255) NOT NULL,
    aggregate_id   VARCHAR(255) NOT NULL,
    event_type     VARCHAR(255) NOT NULL,
    payload        TEXT NOT NULL,
    headers        TEXT,
    created_at     TIMESTAMP NOT NULL DEFAULT now(),
    published_at   TIMESTAMP
);

CREATE INDEX idx_outbox_unpublished ON outbox (created_at) WHERE published_at IS NULL;

OutboxDomainEventPublisher is the outbound adapter PlaceOrderService (Structuring the Code) already names in its hexagon figure: it implements the DomainEventPublisher port by inserting an outbox row, and nothing else — no broker client, no network call, so it can never make placeOrder’s `@Transactional method fail for a reason unrelated to the local database:

package com.example.order.infrastructure;

import com.example.order.application.DomainEventPublisher; // the public port from Structuring the Code
import com.example.shared.event.DomainEvent;
import com.example.shared.event.DomainEventEnvelope;
import org.springframework.stereotype.Component;

import java.time.Instant;
import java.util.List;
import java.util.UUID;

@Component
class OutboxDomainEventPublisher implements DomainEventPublisher {

    private final OutboxRepository outboxRepository;
    private final EventSerializer eventSerializer; // e.g. a thin Jackson ObjectMapper wrapper

    OutboxDomainEventPublisher(OutboxRepository outboxRepository, EventSerializer eventSerializer) {
        this.outboxRepository = outboxRepository;
        this.eventSerializer = eventSerializer;
    }

    @Override
    public void publish(String aggregateType, String aggregateId, long aggregateVersion, List<DomainEvent> events) {
        // Runs inside PlaceOrderService.placeOrder's own @Transactional method --
        // these inserts commit or roll back together with the Order row.
        for (DomainEvent event : events) {
            UUID eventId = UUID.randomUUID();
            // The payload is the whole envelope, never the bare event: the row id doubles as the
            // envelope's eventId, which is exactly what every consumer deduplicates on, and the
            // aggregate version rides along for read-model catch-up checks. causationId and
            // correlationId are left to the caller's tracing context (see the Messaging reference's
            // correlation-id coverage) rather than invented here.
            DomainEventEnvelope<DomainEvent> envelope = new DomainEventEnvelope<>(
                    eventId.toString(), aggregateType, aggregateId, aggregateVersion, null, null, event);
            outboxRepository.save(new OutboxRow(
                    eventId,
                    aggregateType,
                    aggregateId,
                    event.getClass().getSimpleName(),
                    eventSerializer.toJson(envelope),
                    eventSerializer.headersOf(envelope),
                    Instant.now(),
                    null));
        }
    }
}

A @Scheduled polling relay, publishing unpublished rows in created_at order, keyed by aggregate_id so Kafka preserves per-order ordering, and marking each row only after the broker has acknowledged it:

package com.example.order.infrastructure;

import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import org.springframework.transaction.annotation.Transactional;

import java.time.Instant;
import java.util.List;

@Component
class OutboxPollingRelay {

    private final OutboxRepository outboxRepository;
    private final KafkaTemplate<String, String> kafkaTemplate;

    OutboxPollingRelay(OutboxRepository outboxRepository, KafkaTemplate<String, String> kafkaTemplate) {
        this.outboxRepository = outboxRepository;
        this.kafkaTemplate = kafkaTemplate;
    }

    @Scheduled(fixedDelay = 500)
    @Transactional
    public void relayUnpublished() {
        List<OutboxRow> batch = outboxRepository.findUnpublishedOrderByCreatedAtAsc(100);
        for (OutboxRow row : batch) {
            // Keyed by aggregate_id -- same-order events land on the same partition, in order.
            // send() is asynchronous: join() blocks until the broker has acknowledged the record and
            // throws if it never does, so a row is only ever marked published after that ack. Marking
            // first and awaiting later would turn a failed send into a silently lost event.
            kafkaTemplate.send("order-events", row.aggregateId(), row.payload()).join();
            outboxRepository.markPublished(row.id(), Instant.now());
            // A crash between the ack and this mark re-delivers the row on the next pass --
            // at-least-once, which is exactly why OrderPlacedProjectionHandler below must be idempotent.
        }
    }
}

The Spring Modulith externalized-event configuration sketch, for a project that prefers the framework’s own event publication registry over a hand-written outbox table — OrderPlaced opts in once, and the registry (not OutboxPollingRelay) becomes the relay:

package com.example.order.domain.event;

import org.springframework.modulith.events.Externalized;

@Externalized("order-events::#{#this.orderId()}")
public record OrderPlaced(OrderId orderId, CustomerId customerId, List<OrderLine> lines,
                           Money totalAmount, Instant occurredOn) implements DomainEvent {
}

A consumer applying the relayed event to a read-optimized projection, deduplicating with an atomic INSERT …​ ON CONFLICT DO NOTHING rather than a plain existence check:

package com.example.order.api;

import com.example.shared.event.DomainEventEnvelope;
import com.example.order.domain.event.OrderPlaced;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
import org.springframework.transaction.annotation.Transactional;

@Component
class OrderPlacedProjectionHandler {

    private final JdbcTemplate jdbcTemplate;
    private final OrderHistoryRepository orderHistory;

    OrderPlacedProjectionHandler(JdbcTemplate jdbcTemplate, OrderHistoryRepository orderHistory) {
        this.jdbcTemplate = jdbcTemplate;
        this.orderHistory = orderHistory;
    }

    @KafkaListener(topics = "order-events")
    @Transactional
    public void on(DomainEventEnvelope<OrderPlaced> envelope) {
        int inserted = jdbcTemplate.update(
                "INSERT INTO processed_events (event_id, processed_at) VALUES (?, now()) "
                        + "ON CONFLICT (event_id) DO NOTHING",
                envelope.eventId());
        if (inserted == 0) {
            return; // redelivered -- another delivery already claimed this event id
        }
        orderHistory.upsertFromPlaced(envelope.event(), envelope.aggregateVersion());
    }
}

inserted == 0 is the same signal messaging-patterns-in-practice.adoc’s dedup query returns an empty `RETURNING clause for — the row already existed, so this delivery is a safe no-op; the effect and the dedup record commit together in the same @Transactional method, so a crash between the two can never leave one without the other.

Mermaid: the outbox flow end to end

The sequence from PlaceOrderService.placeOrder writing both rows in one transaction, through the relay, to a deduplicating consumer on the other side:

sequenceDiagram participant Service as PlaceOrderService participant DB as Order database participant Relay as OutboxPollingRelay participant Broker as Kafka participant Consumer as OrderPlacedProjectionHandler Service->>DB: begin transaction Service->>DB: insert Order row Service->>DB: insert outbox row, aggregate_id equals orderId Service->>DB: commit Note over Service,DB: one local transaction -- no dual write Relay->>DB: select unpublished rows ordered by created_at Relay->>Broker: send keyed by aggregate_id Broker-->>Relay: acknowledge Relay->>DB: mark row published_at Broker->>Consumer: deliver OrderPlaced envelope Consumer->>Consumer: INSERT INTO processed_events ON CONFLICT DO NOTHING Consumer->>Consumer: apply to OrderHistoryView if newly inserted

Figure: polling publisher vs. transaction-log tailing

Side-by-side comparison of the two ways to relay an outbox row to a broker: on the left

References