The Transactional Outbox: Reliable Events Without Distributed Transactions

Most backends eventually need to do two things when something happens: change the database and tell the rest of the system about it. An order is placed, so the order row is written and an order.created event goes to a queue, where workers send the confirmation email, reserve stock and notify the warehouse.

The obvious code does one, then the other. It works in development and fails in production in ways that are hard to see.

#The Problem: Two Writes, No Transaction

// Looks fine. Isn't.
await db.orders.insert(order);
await broker.publish("order.created", { orderId: order.id });

There are two separate systems here, and nothing makes them succeed or fail together:

• The insert succeeds, the publish fails (broker down, network blip, process restarts between the lines). The order exists, but nothing downstream ever hears about it. No email, no stock reservation.
• You publish first, then the insert fails. Downstream services act on an order that doesn't exist.

Swapping the order of the lines just swaps which failure you get. Wrapping them in try/catch doesn't help either: you can't "un-send" a message, and a crash between the lines skips any catch entirely.

#The Idea: Make the Event Part of the Same Transaction

Instead of publishing directly, write the event into an outbox table, in the same database transaction as the business change. Either both rows commit or neither does. A separate relay then reads the outbox and publishes to the broker.

#Step 1: The Outbox Table

CREATE TABLE outbox (
  id            BIGSERIAL PRIMARY KEY,
  topic         TEXT        NOT NULL,
  payload       JSONB       NOT NULL,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at  TIMESTAMPTZ
);

-- The relay only ever looks for unpublished rows, oldest first.
CREATE INDEX outbox_unpublished ON outbox (id) WHERE published_at IS NULL;

#Step 2: Write the Change and the Event Together

await db.transaction(async (tx) => {
  const order = await tx.orders.insert({ customerId, items, total });
  await tx.outbox.insert({
    topic: "order.created",
    payload: { eventId: crypto.randomUUID(), orderId: order.id, total },
  });
});
// Nothing is published here. If this request crashes now, the event is still safe in the outbox.

Give every event its own eventId. You'll need it in Step 4.

#Step 3: The Relay

The relay is a small loop (a background worker, a cron job, or a function on a timer) that publishes pending rows and marks them done:

async function relayBatch() {
  await db.transaction(async (tx) => {
    // SKIP LOCKED lets several relay instances run without picking the same rows.
    const rows = await tx.query(
      `SELECT id, topic, payload FROM outbox
        WHERE published_at IS NULL
        ORDER BY id
        LIMIT 100
        FOR UPDATE SKIP LOCKED`,
    );
    for (const row of rows) {
      await broker.publish(row.topic, row.payload);
      await tx.query(`UPDATE outbox SET published_at = now() WHERE id = $1`, [row.id]);
    }
  });
}

If the relay crashes after publishing but before the UPDATE commits, that row is published again on the next run. That's the trade the pattern makes: at-least-once delivery. Messages are never lost, but they can arrive twice.

#Step 4: Make Consumers Idempotent

Because delivery is at-least-once, every consumer must be safe to run twice for the same event. The simplest approach is to record which events you've handled, in the same transaction as the work:

async function onOrderCreated(event: { eventId: string; orderId: string }) {
  await db.transaction(async (tx) => {
    const firstTime = await tx.query(
      `INSERT INTO processed_events (event_id) VALUES ($1) ON CONFLICT DO NOTHING RETURNING event_id`,
      [event.eventId],
    );
    if (firstTime.length === 0) return; // already handled: do nothing
    await tx.stock.reserveFor(event.orderId);
  });
}

For side effects outside your database, such as sending an email, pass the eventId as an idempotency key to the provider when it supports one.

#Operating It

A few details make the difference between a pattern that works and one that's pleasant to run:

• Clean up. Delete published rows after a retention period (say, 7 days), or the table grows forever.
• Watch the lag. Alert on the age of the oldest unpublished row. If it climbs, the relay is stuck or the broker is down, and you'll know before customers do.
• Ordering. Rows are published in id order per relay, but consumers may still see events out of order across partitions or retries. Design consumers to cope, for example by checking a version number.
• Keep payloads small. Include IDs and the fields consumers need, not entire records.

#When You Don't Need It

If an event being occasionally lost is acceptable (analytics pings, cache warm-ups), publishing directly is simpler and fine. Reach for the outbox when the event drives something the business cares about: payments, fulfilment, notifications a customer expects.

#Summary

Writing to a database and a message broker are two operations that can't share a transaction. The outbox pattern makes the event part of the database transaction, publishes it afterwards with a relay, and accepts duplicates in exchange for never losing anything. Pair it with idempotent consumers and you get events you can build a business on.