When One Database Is No Longer Enough: A Practical Guide to Distributed Transactions
From ACID transactions to Sagas, Outbox, and CQRS — a journey through the patterns that keep distributed systems consistent
You will not sit down one morning and decide to build a distributed system. You will wake up and realize you already have one — and now you need to keep it consistent.
This is the story of that realization, and the patterns that make it survivable.
Part 1: The Wall You Will Eventually Hit
Early in your system's life, everything lives in one database. One connection string. One transaction. You BEGIN, you write, you COMMIT, and the world is consistent.
BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE id = 'A';
UPDATE accounts SET balance = balance + 100 WHERE id = 'B';
COMMIT;
Either both updates land, or neither does. The database guarantees it. You sleep well.
Then the system grows.
Your monolith starts feeling slow. A team wants to own the orders service without touching the inventory database. Payment needs its own compliance boundary. You split into microservices — and suddenly those two UPDATE statements live in two different databases, owned by two different services, talking to each other over HTTP.
Now what?
The single-database transaction is gone. You cannot BEGIN across two Postgres instances. The guarantee that made your sleep peaceful has disappeared.
This is the wall.
Part 2: The Obvious Fix — and Why It Breaks
The first instinct is to coordinate. If we can't have one transaction, maybe we can have two transactions that agree with each other.
That's the idea behind Two-Phase Commit (2PC).
How 2PC Works
2PC introduces a coordinator that runs the protocol in two phases:
Phase 1 — Prepare: The coordinator asks every participant: "Can you commit?" Each participant writes the data to a durable log, locks the relevant rows, and responds YES or NO.
Phase 2 — Commit: If all participants said YES, the coordinator sends COMMIT to everyone. If anyone said NO, it sends ROLLBACK.
Coordinator
│
├──► Service A: "Prepare?" → "YES" (rows locked)
├──► Service B: "Prepare?" → "YES" (rows locked)
│
├──► Service A: "Commit"
└──► Service B: "Commit"
On paper, this is elegant. In practice, it has three problems that compound each other in distributed systems.
Problem 1: Blocking on Failure
Between Phase 1 and Phase 2, participants are holding locks and waiting. If the coordinator crashes after receiving all YES votes but before sending COMMIT, participants are stuck. They cannot commit. They cannot rollback. They are blocked until the coordinator recovers — and they are holding locks while they wait.
In a high-throughput system, this cascades. Blocked rows mean blocked requests. Blocked requests mean degraded throughput. A coordinator crash at the wrong moment brings parts of your system to a halt.
Problem 2: It Assumes Synchronous Reliability
2PC assumes all participants are reachable, responsive, and will stay that way during the protocol. But distributed systems are defined by partial failures. A network partition between the coordinator and one participant after Phase 1 means you don't know if that participant committed or not. You're in a state of permanent uncertainty.
Problem 3: It Doesn't Compose with Modern Architectures
2PC requires participants to implement a specific protocol. Most modern systems — queues, caches, third-party APIs — do not. You cannot run 2PC against Stripe. You cannot run it against SQS. The moment your transaction boundary crosses a system that doesn't speak the protocol, 2PC stops being an option.
The Real Lesson
2PC taught us something important: you cannot buy distributed consistency with coordination alone. The cost of coordination — in latency, in lock contention, in failure surface — is too high. We need a different model.
Part 3: Accepting the New Reality
The shift from 2PC to modern distributed patterns requires a philosophical adjustment.
In a single-database world, you think in terms of atomicity: everything happens, or nothing does.
In a distributed world, you think in terms of eventual consistency: the system will be consistent eventually, not necessarily right now.
This is not a concession. It's an architectural choice that unlocks scale. But it requires three things:
Operations must be idempotent — running them twice must produce the same result as running them once.
Failures must be retried, not ignored — if a step fails, the system must keep trying.
Side effects must be undoable — if something goes wrong downstream, there must be a way to compensate.
These three properties are the foundation of everything that follows.
Part 4: The Saga Pattern — Distributed Writes
A Saga is a sequence of local transactions, each in a different service, where every step has a corresponding compensating transaction that can undo its effect.
Instead of one atomic transaction, you have a chain of local ones. If the chain succeeds end-to-end, the business operation is complete. If any step fails, the saga runs compensating transactions backwards to restore consistency.
OrderService [Message Queue] PaymentService InventoryService
│ │ │ │
publish(cmd) ─────────►│ │ │
│──── deliver ─────►│ │
│ publish(PaymentCharged) ───►│ (via queue)
│ Reserve Stock
│
│ [on failure]
│◄── publish(PaymentFailed) ──────────────
│◄── publish(CompensateStock) ────────────
publish(OrderCancelled)►│
No Synchronous Calls — In Either Saga Type
Before looking at the two types, there is one rule that applies to both without exception: no service ever calls another service directly.
Not in choreography. Not in orchestration. There are no HTTP calls between services in a saga. No service opens a connection to another and waits for a response. All coordination — every command, every event, every reply — travels through the message queue.
This is not a style preference. It is what makes the saga resilient:
A service can be down when a message is sent. The queue holds it. The service processes it when it comes back up.
A sender can crash after publishing. The message is already in the queue. Nothing is lost.
A consumer can crash mid-processing. The queue redelivers to another instance because no ACK was sent.
The moment you introduce a synchronous call between two services in a saga, you reintroduce the exact failure modes you were trying to escape — timeouts, cascading failures, and operations that half-succeed with no way to recover.
Every arrow in the diagrams below passes through the message queue. That is the only way services in a saga communicate.
Two Types of Sagas
Choreography-Based Sagas
In choreography, there is no central coordinator. Each service subscribes to events from the queue, does its local work, and publishes the next event back to the queue. No service knows about any other service. They only know about the events they consume and the events they emit.
OrderService [Message Queue] PaymentService InventoryService
│ │ │ │
publish(OrderCreated) ───►│ │ │
│──► deliver ──────────►│ │
│ publish(PaymentCharged) ──────►│ (via queue)
│ publish(StockReserved)
│
│ [on failure]
│◄── publish(PaymentFailed) ────────────────────
publish(OrderCancelled) ──►│ (triggered by PaymentService compensation)
OrderService never calls PaymentService. It publishes OrderCreated to the queue and its job is done. PaymentService picks it up when it's ready. If PaymentService is restarting at that moment, the message waits. When PaymentService finishes, it publishes PaymentCharged to the queue — and InventoryService picks that up independently.
When to use it: When services are loosely coupled and you want maximum autonomy. Each team deploys independently. No one owns the flow.
The tradeoff: The business logic is distributed across services. Understanding the full flow requires reading multiple codebases. Debugging a broken saga means correlating events across multiple logs.
Orchestration-Based Sagas
In orchestration, a dedicated saga orchestrator coordinates the flow. It sends commands to services and reacts to their responses — but the same rule applies: the orchestrator never makes synchronous HTTP calls to the services it coordinates.
Everything goes through the message queue.
The orchestrator publishes a command message to a queue. The target service consumes that message, does its local work, and publishes a reply event back to another queue. The orchestrator subscribes to reply events and decides what to do next. There is no direct call. No waiting on a socket. No timeout on an HTTP request.
Orchestrator Message Queue PaymentService
│ │ │
│── publish(ChargeCard cmd) ───►│ │
│ │──── deliver(ChargeCard) ─────►│
│ │ local tx
│ │◄─── publish(PaymentCharged) ──│
│◄── deliver(PaymentCharged) ───│ │
│ │
│── publish(ReserveStock cmd) ──►│ ...and so on
This matters for a reason that goes beyond architecture taste: if the orchestrator crashes between sending a command and receiving the reply, no work is lost. The command is already in the queue. The service will process it. When the orchestrator restarts and rehydrates its state, it will see the reply event and continue the saga from where it left off.
This is only true if you treat the message queue as a durable contract, not a fire-and-forget pipe.
The Message Queue Is Not Optional Infrastructure — It's the Guarantee
When the orchestrator publishes a command and gets an acknowledgement from the queue, that acknowledgement means: "I have this message. Even if everything crashes right now, I will deliver it." That's the contract. The orchestrator can move on precisely because the queue owns delivery.
But the queue's guarantee only holds if:
The queue is durable — messages are persisted to disk, not held in memory. If the broker restarts, messages survive. RabbitMQ with durable queues, Kafka with replication, SQS — all durable by default or by config. An in-memory queue is not durable; it's a liability.
The consumer acknowledges only after processing — the queue marks a message delivered only when the consumer sends an explicit
ACK. If the consumer crashes after reading but before finishing its local transaction, the queue redelivers the message to another consumer instance. This is why at-least-once delivery and idempotent consumers are inseparable — the queue guarantees delivery, not exactly-once execution.Dead-letter queues catch what can't be processed — if a consumer fails repeatedly on the same message, the queue moves it to a dead-letter queue (DLQ) rather than dropping it or blocking. The saga orchestrator monitors the DLQ and triggers compensation when a message lands there.
// Consumer: acknowledge ONLY after the local transaction succeeds
channel.consume('charge-card-commands', async (msg) => {
if (!msg) return;
try {
const command = JSON.parse(msg.content.toString());
await db.transaction(async (tx) => {
await tx.insert('charges', { orderId: command.orderId, amount: command.amount });
});
// Publish reply BEFORE ack — if this fails, message redelivers and we retry
await channel.publish('saga-replies', Buffer.from(JSON.stringify({
type: 'PaymentCharged',
orderId: command.orderId,
})));
channel.ack(msg); // Only now do we tell the queue: "done"
} catch (err) {
// Negative ack — requeue for retry, or route to DLQ after max attempts
channel.nack(msg, false, shouldRetry(err));
}
});
The orchestrator's job is to react to reply events, not to poll services or hold open connections. It is a state machine driven by the event stream, not a synchronous coordinator waiting on responses.
When to use it: When the flow is complex, involves many steps, or requires business-level visibility into what's happening. Easier to reason about, easier to monitor, easier to debug.
The tradeoff: The orchestrator becomes a critical stateful component. It must be durable, recoverable, and observable. Its state should be persisted after every step — so on restart, it knows exactly which saga instances are in flight and where they left off.
Writing Compensating Transactions
A compensation is not a rollback. It is a new forward action that semantically undoes the effect of the original step.
// Original step
async function chargeCard(orderId: string, amount: number): Promise<void> {
const chargeId = await paymentGateway.charge(orderId, amount);
await db.insert('payments', { orderId, chargeId, amount, status: 'charged' });
}
// Compensation — a new business action, not a DB rollback
async function refundCard(orderId: string): Promise<void> {
const payment = await db.findOne('payments', { orderId });
await paymentGateway.refund(payment.chargeId);
await db.update('payments', { orderId }, { status: 'refunded' });
}
Compensations must be idempotent. The saga framework may call them more than once. They must produce the same result regardless.
Part 5: The Outbox Pattern — Reliable Event Publishing
Sagas depend on events. Services must publish events reliably. But publishing an event is itself a distributed operation — and it has a gap.
// This is broken
async function createOrder(data: OrderData) {
await db.insert('orders', { ...data, status: 'created' });
// What if the process crashes here?
await eventBus.publish('OrderCreated', data); // ← this may never happen
}
If the process crashes between the database write and the event publish, the order exists but the event is never sent. Downstream services never know the order was created. The saga never starts.
This is the dual-write problem. Two systems need to be updated atomically, but they can't be — one is a database, one is a message broker.
The Outbox Solution
Instead of publishing directly to the event bus, write the event to an outbox table in the same database transaction as your business data. A separate background process (the relay) reads from the outbox and publishes to the event bus, marking events as published once they're delivered.
// Step 1: Write business data and the event in one transaction
async function createOrder(data: OrderData) {
await db.transaction(async (tx) => {
await tx.insert('orders', { ...data, status: 'created' });
await tx.insert('outbox', {
eventType: 'OrderCreated',
payload: JSON.stringify(data),
publishedAt: null,
});
});
}
// Step 2: A relay process publishes the event and marks it done
async function relay() {
const pending = await db.query(
'SELECT * FROM outbox WHERE published_at IS NULL LIMIT 100'
);
for (const event of pending) {
await eventBus.publish(event.eventType, JSON.parse(event.payload));
await db.update('outbox', { id: event.id }, { publishedAt: new Date() });
}
}
Now the guarantee is: if the order is written, the event will eventually be published — even if the process crashes immediately after the transaction. The relay will pick it up on the next run.
The outbox pattern makes event publishing at-least-once. Consumers must be idempotent — they may receive the same event twice and must handle it without side effects.
Part 6: CQRS — Separating Reads from Writes
Sagas and the outbox solve the write side. But distributed systems also create a read problem.
When data lives in multiple services, assembling a view that spans services is painful. Joining data across service boundaries requires HTTP calls, coordination, and latency — and the result is fragile.
CQRS (Command Query Responsibility Segregation) separates the model you use for writes from the model you use for reads.
The write side owns the authoritative state. Commands mutate it. Events are emitted.
The read side owns projections — purpose-built views optimized for specific queries. They are built by subscribing to those events.
Write Side Read Side
────────── ─────────
OrderService ──── OrderCreated ────► OrderSummaryProjection
PaymentService ─── PaymentCharged ──► DashboardProjection
InventoryService ─ StockReserved ───► AvailabilityProjection
Projections Are Just Subscribers
This is the key insight: a CQRS projection is just an event subscriber that builds a read model.
// Subscriber that builds an order summary read model
eventBus.subscribe('OrderCreated', async (event) => {
await readDb.insert('order_summaries', {
orderId: event.orderId,
customerId: event.customerId,
status: 'created',
createdAt: event.timestamp,
});
});
eventBus.subscribe('PaymentCharged', async (event) => {
await readDb.update('order_summaries',
{ orderId: event.orderId },
{ paymentStatus: 'charged', totalAmount: event.amount }
);
});
eventBus.subscribe('StockReserved', async (event) => {
await readDb.update('order_summaries',
{ orderId: event.orderId },
{ fulfillmentStatus: 'reserved' }
);
});
Your API reads from order_summaries. It's fast. It's a simple query. It never crosses a service boundary.
The tradeoff: the projection is eventually consistent. There's a lag between when an event is emitted and when the read model reflects it. For most use cases — dashboards, listings, summaries — this is acceptable. For use cases where a user just performed an action and expects to see it immediately, you handle this with optimistic UI or explicit loading states.
The Read Side Is Not One Thing
This is where CQRS becomes genuinely powerful: the read side has no fixed shape. A projection subscriber can build any kind of read model that serves the query. The write side does not care. It only emits events. What you do with them on the read side is entirely up to you.
Materialized View / Denormalized Table
The most common form. A subscriber maintains a flat, query-optimized table — pre-joined, pre-aggregated, shaped exactly for the API response. No joins at query time. Fast reads, simple queries.
Events → Subscriber → orders_summary table → GET /orders/:id
Best for: dashboards, list views, order history, anything where the shape of the read is known and stable.
Event Sourcing Store
Instead of updating a mutable row, the subscriber appends every event to an immutable log for that aggregate. The current state is derived by replaying the log from the beginning (or from a snapshot).
OrderCreated t=0
PaymentCharged t=1
StockReserved t=2
OrderShipped t=3
│
▼ replay
Current state: { status: 'shipped', payment: 'charged', ... }
This is Event Sourcing used as the read model. The full history is queryable. You can ask "what was the state of this order at 3pm yesterday?" You can replay history to build a new projection you didn't anticipate when the events were first written.
The write side doesn't change. You just add a new subscriber that replays and projects differently.
Best for: audit logs, financial ledgers, anything where history and state reconstruction matter.
Search Index
A subscriber listens to events and indexes documents into Elasticsearch or similar. The read model is a search engine, not a database.
Events → Subscriber → Elasticsearch index → GET /search?q=...
Best for: full-text search, faceted filtering, anything a relational DB handles poorly at scale.
Authorized / Tenant-Scoped View
A subscriber builds a read model filtered to a specific user, role, or tenant. The event stream is global; the projection is scoped.
eventBus.subscribe('OrderCreated', async (event) => {
// Each tenant gets their own isolated read model
await readDb.insert(`orders_${event.tenantId}`, {
orderId: event.orderId,
status: 'created',
});
});
The API reads from orders_${tenantId} — a table that physically cannot contain another tenant's data. Authorization is enforced at the projection layer, not the query layer. There is no WHERE clause to miss or misconfigure.
Best for: multi-tenant systems, role-based views, any case where data isolation needs to be structural rather than conditional.
Cache / In-Memory View
A subscriber maintains a Redis hash or in-memory structure for the hottest, most-read data. The event stream is the cache invalidation mechanism — no TTL guessing, no stale data problem. An event arrives, the subscriber updates the cache.
OrderStatusChanged → Subscriber → Redis HSET order:{id}:status "shipped"
Best for: user-facing status displays, anything where read latency must be sub-millisecond.
The point is not to use all of these at once. The point is that the event stream is your asset. Once you have a reliable stream of business events flowing through your system, you can build any read model you need — now or in the future — without touching the write side. You add a subscriber, replay from the beginning of the log, and the new read model is populated.
This is what makes CQRS more than a query optimization pattern. It's a way of keeping your options open.
Part 7: How the Pieces Connect
These patterns are not independent tools. They form a coherent architecture for distributed writes and reads.
=== WRITE PATH ===
API Request
|
v
Saga Orchestrator
|
|---> Service A => local tx => Outbox entry
|---> Service B => local tx => Outbox entry
'---> Service C => local tx => Outbox entry
|
Outbox Relay
(publishes to event bus)
|
v
=== READ PATH ====
Event Bus
|
|---> Projection subscriber A --> Read Model A
|---> Projection subscriber B --> Read Model B
'---> Projection subscriber C --> Read Model C
API Query --> Read Model --> Response
(fast, local, no cross-service joins)
Sagas handle the write journey — coordinating state changes across services with compensations for failure.
The Outbox makes the event publishing step in that journey reliable — bridging the gap between your local transaction and the event bus.
CQRS projections are subscribers that turn those same events into read-optimized views — giving you fast, aggregated queries without cross-service joins.
The events flowing through your outbox are the same events powering your projections. The write path and the read path share the same event stream. You're not building two systems — you're building one system with two lenses.
Summary: Which Pattern Solves Which Problem
| Pattern | Problem It Solves | Tradeoff |
|---|---|---|
| Choreography Saga | Multi-service write coordination, loosely coupled | Distributed logic, hard to trace |
| Orchestration Saga | Multi-service write coordination, explicit flow | Orchestrator must be durable and stateful |
| Outbox Pattern | Reliable event publishing without dual-write | At-least-once delivery; consumers need idempotency |
| CQRS — Materialized View | Fast cross-service reads without joins | Eventual consistency; shape must be known upfront |
| CQRS — Event Sourcing | Full history, replayable state, retroactive projections | Storage grows with history; replay can be slow |
| CQRS — Search Index | Full-text and faceted queries at scale | Index lag; separate infrastructure to operate |
| CQRS — Authorized View | Structural data isolation per tenant or role | More projections to maintain as access patterns grow |
| CQRS — Cache Layer | Sub-millisecond reads on hot data | Cache must be kept in sync via event subscription |
Where to Go From Here
The patterns in this article are the vocabulary of distributed systems, but the sentences you write with them depend on your context.
Start simple. Add choreography before orchestration. Add CQRS projections when your queries become painful, not before. Introduce the outbox when you've been burned by a missed event in production — because you will be.
The journey from a single database to a fully event-driven system is incremental. You do not redesign everything at once. You identify the seams where consistency breaks, apply the smallest pattern that fixes the break, and move on.
The wall is real. But it has doors.
