The Outbox Pattern in Practice: Solving Dual-Writes in Event-Driven Data Pipelines

TL;DR: Writing to your database and publishing to a broker in the same request is a dual-write, and it will eventually leave the two out of sync. The transactional outbox fixes this by writing the event into an outbox table inside the same local transaction as the business change, then relaying it to the broker asynchronously. You get atomicity and at-least-once delivery — which means consumers must be idempotent, ordering must be designed per aggregate, and the outbox itself must be cleaned up and monitored.

Almost every event-driven system hits the same bug. A service saves an order, then publishes OrderPlaced to Kafka. Most of the time both succeed. Occasionally the process crashes, the broker times out, or a deploy kills the pod between the two calls. Now the database says the order exists and no downstream system knows about it — or, if you publish first, downstream systems react to an order that was rolled back. No amount of retry logic in application code closes this gap, because the two writes go to two systems with no shared transaction.

This article covers the transactional outbox pattern as it is actually run in production: table design, the two relay strategies, ordering guarantees, consumer idempotency, cleanup, and the failure modes that catch teams out.

The dual-write problem, precisely

Consider the three orderings available to naive code:

1. Commit DB, then publish. If the publish fails or the process dies, the event is lost. The database is ahead of the stream.
2. Publish, then commit DB. If the commit fails, consumers have acted on something that never happened. The stream is ahead of the database.
3. Publish inside the DB transaction, before commit. Same as (2) — the broker does not participate in your database's transaction, so a rollback cannot un-publish.

Distributed transactions (two-phase commit/XA) could coordinate both resources, but most modern brokers do not support XA, and 2PC couples the availability of your service to the availability of the broker. The practical answer is to reduce two writes to one (microservices.io, Confluent).

The pattern

In the same local transaction as the business change, insert a row describing the event into an outbox table. The database guarantees both rows commit or neither does. A separate relay process reads committed outbox rows and publishes them to the broker.

┌──────────── Service ────────────┐
│  BEGIN                          │
│    INSERT INTO orders ...       │
│    INSERT INTO outbox ...       │  ← same transaction
│  COMMIT                         │
└───────────────┬─────────────────┘
                │ committed rows only
          ┌─────▼─────┐        ┌────────┐       ┌───────────┐
          │   Relay   │ ─────► │ Broker │ ────► │ Consumers │
          │ (poll/CDC)│        └────────┘       │(idempotent)│
          └───────────┘                          └───────────┘

Outbox table design

CREATE TABLE outbox (
  id             UUID PRIMARY KEY,            -- stable event ID, used for dedup downstream
  aggregate_type TEXT        NOT NULL,        -- e.g. 'order' → routes to topic
  aggregate_id   TEXT        NOT NULL,        -- used as the message key → partition → ordering
  event_type     TEXT        NOT NULL,        -- e.g. 'OrderPlaced'
  payload        JSONB       NOT NULL,        -- or bytes for Avro/Protobuf
  headers        JSONB,                       -- trace context, schema version, tenant
  created_at     TIMESTAMPTZ NOT NULL DEFAULT now()
);

This shape matches what Debezium's Outbox Event Router expects by default: it routes rows to a topic derived from aggregate_type, uses aggregate_id as the message key, and emits payload as the message value (Debezium docs).

Design choices worth making deliberately:

• Event ID generated by the producer, not the database sequence, so it is stable across retries and can be used for deduplication by consumers.
• Aggregate ID as the message key, so all events for one entity land in the same partition and stay ordered.
• A self-contained payload. Include the data consumers need, not just an ID they must call back for. Callbacks reintroduce coupling and race conditions (the consumer may read a newer state than the event describes).
• Schema version in headers, so consumers can handle payload evolution explicitly.

Application code

def place_order(conn, cmd):
    with conn.transaction():
        order = insert_order(conn, cmd)
        conn.execute(
            """INSERT INTO outbox (id, aggregate_type, aggregate_id, event_type, payload, headers)
               VALUES (%s, 'order', %s, 'OrderPlaced', %s, %s)""",
            (uuid7(), str(order.id), json.dumps(order.to_event()),
             json.dumps({"schema": "order.v3", "traceparent": current_trace()})),
        )
    return order   # no broker call here — the relay handles publishing

The request path no longer touches the broker at all. That is a latency and availability win in its own right: a broker outage no longer fails user-facing writes; events simply accumulate in the outbox until it recovers.

Relay strategy 1: polling publisher

A background worker repeatedly queries the outbox for unpublished rows, publishes them, and marks or deletes them.

-- Claim a batch without blocking other relay workers
SELECT id, aggregate_type, aggregate_id, event_type, payload, headers
FROM outbox
WHERE published_at IS NULL
ORDER BY created_at, id
LIMIT 500
FOR UPDATE SKIP LOCKED;

After the broker acknowledges each message, set published_at = now() (or delete the row) and commit.

Strengths: works on any relational database with no special configuration, easy to reason about, easy to operate. Guidance from several practitioners is to start here when first adopting the pattern (Streamkap).

Weaknesses:

• Latency is bounded by the poll interval.
• Load — frequent polling of a hot table adds query and index churn, and the published_at update doubles write volume.
• Ordering hazards — ORDER BY created_at is not commit order. A transaction that started earlier but committed later can insert a row with an earlier timestamp after the relay has already moved past that point. With SKIP LOCKED and multiple workers, ordering across workers is not guaranteed either. If you need strict per-aggregate ordering with polling, run a single relay per partition of aggregates, or accept reordering and handle it in consumers with version numbers.

Relay strategy 2: log-based CDC

Instead of querying the table, a CDC connector such as Debezium tails the database's transaction log (the Postgres WAL via logical replication, the MySQL binlog) and emits every committed insert into the outbox table as an event. The Outbox Event Router transform reshapes those change events into clean domain messages.

Strengths:

• Commit order. The log is ordered by commit, so the relay never sees a later-committed row before an earlier one.
• Low latency — events typically appear shortly after commit rather than on the next poll.
• No polling load and no published_at updates.
• Cleanup is simple: because the connector reads the log, not the table, you can delete outbox rows immediately after insert, in the same transaction. Debezium's documentation describes this pattern — the insert still appears in the log and is captured, while the table stays essentially empty.

Weaknesses:

• Operational weight. You now run Kafka Connect (or Debezium Server), manage connector offsets, and, on Postgres, a replication slot.
• Replication slot risk. If the connector stops consuming, Postgres retains WAL for the slot indefinitely, and the disk fills. This is the most common way an outbox deployment takes down its own database. Alert on slot lag and set max_slot_wal_keep_size as a safety valve.
• Failover. Logical replication slots historically did not survive primary failover on Postgres, and support varies by version and managed service. Test failover explicitly before relying on it. Running Debezium in production is a discipline of its own — connector restarts, snapshot behaviour, and offset management all deserve runbooks before the first incident, not after.

A variant: logical decoding messages

On Postgres, pg_logical_emit_message() writes an arbitrary message directly into the WAL as part of the current transaction, without any table at all. CDC tools can capture these messages, which eliminates outbox housekeeping entirely (Decodable). The trade-off is that the standard Outbox Event Router expects a table, so you need custom transformation logic, and the approach is Postgres-specific.

Choosing a relay

Delivery semantics: at-least-once, always

Neither relay gives exactly-once delivery end to end. If the relay publishes a message and crashes before recording that it did (marking the row, or committing its connector offset), it will publish again on restart (microservices.io — Idempotent Consumer). Design for duplicates:

• Every message carries the stable outbox id.
• Consumers record processed IDs in their own database in the same transaction as the side effect — an "inbox" or dedup table — and skip IDs already seen.

-- Consumer side, inside the transaction that applies the event
INSERT INTO processed_events (event_id, processed_at)
VALUES ($1, now())
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id;
-- If no row returned: duplicate, skip. Otherwise apply the side effect, then commit.

Broker features like Kafka's idempotent producer and transactions reduce duplicates between relay and broker, but they do not cover the database-to-relay boundary or the consumer's own side effects. Consumer idempotency is not optional.

Ordering, precisely

Ordering guarantees are almost always needed per aggregate, not globally. The outbox achieves this by using aggregate_id as the message key so that all events for one entity go to one partition, where the broker preserves order. Two things can still break it:

1. The relay reorders — the polling hazards described above. CDC avoids this.
2. Consumers process a partition concurrently — for example, a consumer that fans out messages to a thread pool. Preserve per-key ordering in the consumer, or include a per-aggregate version number in the payload so consumers can detect and discard stale events.

A version number is cheap insurance regardless of relay choice: increment it on the aggregate in the same transaction, include it in the event, and have consumers ignore any event whose version is not newer than what they hold.

Cleanup and table health

An outbox that is never cleaned grows without bound and slows down every relay query. Options:

• CDC: delete in the same transaction as the insert.
• Polling: batch-delete published rows older than a retention window, in small chunks to avoid long locks and replication lag.
• Partitioning: on high-volume systems, partition the outbox by time and drop old partitions instead of deleting rows. Dropping a partition is far cheaper than mass deletes and avoids table bloat from dead tuples.

Failure modes checklist

• Replication slot filling the disk when a CDC connector is down. Alert on slot lag; cap WAL retention.
• Poison messages — a payload the relay cannot serialise or the broker rejects (for example, exceeding the maximum message size). Route to a dead-letter destination with the outbox ID rather than blocking the entire relay.
• Relay backlog during broker outages. The outbox absorbs it, but monitor outbox depth and oldest unpublished age so you know when you are behind.
• Schema changes to the outbox table breaking the connector. Treat the outbox schema as a public contract and evolve it additively.
• Payload schema drift breaking consumers. Register event schemas and enforce compatibility in CI.
• Large transactions — a bulk operation inserting a million outbox rows in one transaction produces a large burst for the relay and consumers. Batch bulk operations.

Testing the outbox

The outbox exists to handle failures, so test it by injecting them. Useful tests, most of which can run in CI against containerised dependencies:

• Crash between commit and publish. Kill the relay after it reads a batch but before it records progress; verify every event is eventually delivered and that consumers handle the duplicates.
• Broker outage. Stop the broker, keep writing, restart it; verify the backlog drains in order and no user-facing write failed.
• Rollback. Force the business transaction to fail after the outbox insert; verify no event is ever published.
• Concurrent writers on one aggregate. Issue interleaved updates and assert consumers end in the state matching the final committed version.
• Connector restart and rebalance for CDC relays, including a restart that forces a re-read from the last committed offset.

These tests are cheap compared with discovering, months later, that a subtle ordering or duplication bug has been corrupting a downstream projection.

Observability

Export at minimum: outbox row count or oldest unpublished event age, relay publish rate and error rate, CDC connector lag and replication slot retained WAL size, dead-letter volume, and consumer duplicate-skip counts. Propagate trace context through the outbox headers so a single trace can follow a request from the API call through the relay into every consumer.

When not to use the outbox

The outbox adds a table, a relay, and operational surface area. Skip it when losing an occasional event is genuinely acceptable (best-effort analytics pings), when the database itself is the event log that downstream systems read via CDC directly, or when a single system can own both state and events (event sourcing, where the event store is the source of truth). For most business-critical integrations — orders, payments, inventory, anything feeding analytics platforms that finance will reconcile — it is the simplest correct option.

Teams building custom software and integration APIs on event-driven foundations tend to find that the outbox, done well, removes an entire category of "the data doesn't match" incidents.

Syslabs' engineering team builds and operates event-driven data pipelines that rely on this pattern.

Sources: microservices.io — Transactional outbox; Idempotent consumer · Confluent Developer — The transactional outbox pattern · Debezium documentation — Outbox Event Router · Decodable — The wonders of Postgres logical decoding messages · Streamkap — The outbox pattern explained · AWS Prescriptive Guidance — Transactional outbox pattern