Distributed Transactions
The Consistency Models lesson covered how copies of one record stay in sync across replicas. A distributed transaction solves a different problem. It executes a multi-step write across separate databases or services so that every change takes effect or every change is undone.
Why One Database Transaction Cannot Span Services
When all tables live on one relational database server, a single local transaction wraps multiple writes into an atomic unit:
BEGIN;
UPDATE accounts SET balance = balance - 50 WHERE id = 1;
UPDATE accounts SET balance = balance + 50 WHERE id = 2;
COMMIT;The database's write-ahead log and row locks ensure that both updates commit together or neither takes effect. The Database Transactions lesson covers this mechanism.
That mechanism stops working once data is split across separate databases. Two common architectures create this split:
- One database per microservice. An e-commerce checkout creates a record in the Order Service's database and charges 50 USD in the Payment Service's database. It then decrements stock in the Inventory Service's database.
- Sharded databases. Account 1 and Account 2 sit on different database shards, so a transfer between them touches two independent write-ahead logs.
Without a cross-database coordination protocol, a partial failure leaves data inconsistent across services. Suppose the order write and the 50 USD payment commit, and the inventory service then rejects the write because stock is 0. The buyer is charged 50 USD for an item that cannot ship.
Two families of protocols coordinate these writes. Two-Phase Commit (2PC) locks rows in every participating database and commits them together. Sagas commit each database's local transaction right away and run reverse operations, called compensating transactions, if a later step fails.
Two-Phase Commit (2PC)
Two-Phase Commit (2PC) is an atomic commitment protocol where a central coordinator ensures that all participants (the databases) either all commit or all abort. The X/Open XA specification standardizes 2PC across relational databases such as PostgreSQL and MySQL.
Phase 1 and Phase 2
Execution splits into a voting phase and a decision phase:
- Phase 1 (Prepare). The coordinator generates a transaction ID and sends a
PREPAREmessage to every participant. Each participant runs its local statements, acquires row locks, and flushes its undo and redo log records to disk. If the participant can guarantee that it is able to commit, it repliesYES. By votingYES, the participant surrenders its right to abort unilaterally and promises to keep its locks until the coordinator decides. If any constraint fails or an error occurs, the participant rolls back locally and repliesNO. - Phase 2 (Commit or Abort). If every participant voted
YES, the coordinator writes aCOMMITrecord to its own disk log. Flushing that record is the commit point, after which the transaction's outcome is irrevocable. The coordinator then sendsCOMMITto every participant. Each participant writes its local commit record, releases its row locks, and acknowledges. If any participant votedNOor timed out in Phase 1, the coordinator logsABORTand sendsABORTto all participants. The participants then undo their prepared changes and release their locks.
Why 2PC Blocks and Hurts Throughput
2PC preserves atomicity across databases, because no reader ever sees a partial checkout when one participant votes NO. The cost is lock duration, availability under coordinator failure, and boundary support:
- Blocking on coordinator crash. Once a participant votes
YES, its transaction is in doubt until the coordinator's decision arrives. If the coordinator crashes after collectingYESvotes, the participants cannot safely abort because another participant may have already receivedCOMMIT. They also cannot safely commit because another participant may have votedNO. Every participant must keep its rows locked until the coordinator recovers and replays its log. - Extended lock holding. A local database commit holds row locks for less than a millisecond while flushing the local write-ahead log. Under 2PC, each participant holds its row locks across two network round trips and two disk flushes. Every participant waits for the slowest participant in Phase 1 before Phase 2 can begin. Across regions with 50 ms round-trip latency, locks stay held for over 100 ms. Other transactions touching the same inventory item or account row must queue behind them.
- Protocol boundary limits. All participants must expose a prepare-and-commit interface under your operational control. External HTTP APIs, such as Stripe or a partner airline's reservation endpoint, do not let callers hold an open database lock on their servers while waiting for a second phase.
How Cloud Spanner runs 2PC. Google Cloud Spanner uses 2PC for transactions that span multiple splits, paired with Paxos consensus groups inside each split. Both the coordinator and every participant are Paxos groups rather than single machines. If the coordinator leader crashes during Phase 2, another replica in the coordinator's Paxos group takes over from the replicated log without waiting for one machine to reboot. Even with replicated participants, cross-split 2PC still holds locks across network round trips.
The Saga Pattern
A saga replaces lock-based atomic commitment with a sequence of local transactions (Garcia-Molina and Salem, 1987). Each service runs its local transaction, commits to its own database immediately, and releases its locks before the next service runs.
Because earlier steps have already committed to disk, the database cannot roll them back with ROLLBACK if a later step fails. Instead, each forward step $T_i$ is paired with a compensating transaction $C_i$. $C_i$ semantically undoes $T_i$'s business effect by writing a new local transaction. When step $k$ fails, the saga executes the compensating transactions in reverse order: $C_{k-1}, C_{k-2}, \dots, C_1$.
Press "Next step" to step through the coordinator crash in Two-Phase Commit and see why all three databases stay locked. Then switch Protocol to "Saga (orchestrated)" and run "Inventory out of stock" and "Coordinator crashes mid-flight" to compare how locks and reader visibility behave.
What Sagas Trade Away: Isolation and Compensation Design
A saga gives up the Isolation guarantee of ACID (sometimes described as ACD transactions):
- Visible intermediate states. Step 2 commits the 50 USD charge, and step 4 refunds it after an inventory failure. Between those moments, other queries can read the committed charge and the
PENDINGorder. Services encode workflow state in the record itself, such as settingstatus = PENDINGon the order row until the final step marks itCONFIRMED. This prevents users and downstream jobs from treating an in-flight saga as final. - Compensations are new writes, not erasures. Refunding a card creates a credit record alongside the original debit, and canceling an order sets
status = CANCELED. Side effects that touch the outside world, such as sending a confirmation email or triggering a physical shipment, cannot be silently undone. Services hold those non-reversible actions until the last step of the saga after all reversible checks have succeeded. - Compensations must be idempotent and retried until they succeed. If the network drops while the saga calls the Payment Service to refund 50 USD, the saga executor must retry the refund call until it receives an acknowledgment. Every forward and compensating endpoint therefore requires an idempotency key (such as
saga-401-refund) so that retrying a compensation never refunds the buyer twice.
| Property | Two-Phase Commit (2PC) | Saga Pattern |
|---|---|---|
| Commit mechanism | All participants prepare, lock, and commit together | Each step commits its local transaction right away |
| Rollback mechanism | Database ROLLBACK of uncommitted prepared writes | Compensating transactions run in reverse order ($C_{k-1} \dots C_1$) |
| Row lock duration | Held across network round trips until Phase 2 finishes | Held only during each local database write |
| Isolation | Strong; uncommitted changes stay hidden from other readers | None across steps; intermediate states are visible |
| Coordinator crash | Prepared participants stay blocked holding locks | No database locks held; recovery resumes or compensates from saga log |
| Third-party APIs | Not supported unless the API exposes XA prepare/commit | Supported whenever the API offers a compensating call (such as refund) |
Coordinating a Saga: Choreography vs Orchestration
A saga can coordinate its sequence of steps through choreography (peer-to-peer events) or orchestration (a central state machine).
Choreography
In choreography, no central service tracks the workflow. Each service runs its local transaction, publishes a domain event to a message broker, and subscribes to events from upstream services. If a step fails, the failing service publishes a failure event. Upstream services subscribe to that event to run their own compensating transactions.
Choreography keeps services decoupled when a workflow has only two or three steps and a linear path. As more services join, the rollback logic spreads across many event handlers, and cyclic event dependencies become possible. Furthermore, no single database table shows which step a stuck order is waiting on.
Orchestration
In orchestration, a dedicated orchestrator drives the saga. This can be the Order Service or a workflow engine like Temporal or AWS Step Functions. The orchestrator persists each state transition in a durable saga log, sends commands to participants one step at a time, and awaits their replies. If a step fails, it invokes compensating commands in reverse order.
Because the entire state machine lives in one place, engineers can inspect the saga log to see which step failed or timed out. If an orchestrator instance crashes, a replacement instance reads the durable saga log and resumes from the last recorded step. This recovery does not block any participant database. For workflows with more than two or three services or branching failure paths, orchestration is easier to operate and debug than choreography.
The Dual-Write Problem and the Transactional Outbox
Both choreography and orchestration require a service to update its own database and send a message or command to the next step. Doing those two actions as separate calls in application code creates a dual-write bug:
- Suppose the Order Service commits the new order to
Order DBand crashes before publishingOrderCreatedto the message broker. The order sits inPENDINGforever because Payment Service never hears about it. - Suppose the Order Service publishes
OrderCreatedto the broker first and then theOrder DBcommit fails. Payment Service charges the customer for an order that does not exist.
Using 2PC between the database and the message broker would reintroduce blocking. In addition, many brokers (such as Kafka or SQS) do not participate in XA transactions with relational databases.
The Transactional Outbox pattern solves the dual-write problem using a single local database transaction:
BEGIN;
INSERT INTO orders (id, user_id, amount_cents, status)
VALUES (88, 42, 5000, 'PENDING');
INSERT INTO outbox_events (id, aggregate_id, event_type, payload, published)
VALUES ('evt-901', 88, 'OrderCreated', '{"order_id": 88, "amount": 5000}', false);
COMMIT;Because orders and outbox_events live in the same database, the local transaction commits both rows together or neither row. A separate relay process — either a background worker polling WHERE published = false or a change-data-capture (CDC) connector such as Debezium tailing the database's write-ahead log — reads uncommitted-free rows from outbox_events, publishes them to the broker, and marks them published.
If the relay crashes after publishing to the broker, it might not mark the outbox row as published. After restarting, it will publish the same event a second time. The outbox pattern therefore provides at-least-once delivery. Every downstream consumer must deduplicate incoming events by event.id, as covered in the Failure Handling Patterns lesson.
Avoiding Distributed Transactions When Possible
Every cross-service coordination protocol adds latency, state tracking, or failure modes. Before introducing 2PC or a saga, check whether the boundary itself can be removed or relaxed:
- Consolidate tightly coupled tables into one database. Suppose orders and order line items are always written together and never scale independently. Keeping both tables in the same relational database lets a single local
BEGIN ... COMMITtransaction replace distributed coordination. - Use a single source-of-truth log with derived views. Multiple stores might need to reflect the same write, such as updating a relational database, an Elasticsearch index, and a cache. Treat the relational database as the single source of truth. Feed the search index and cache asynchronously from its write-ahead log or outbox stream.
Choosing a Coordination Approach
| Scenario | Recommended pattern | Why |
|---|---|---|
| Tables always change together and fit on one primary | Single database local transaction | ACID guarantees with sub-millisecond lock duration and no network coordination |
| Cross-shard transfer inside one database cluster where brief lock latency is acceptable | Two-Phase Commit (2PC) | Prevents readers from ever seeing money debited from one shard without being credited to the other |
| Multi-service workflow across independent databases or third-party APIs (checkout, travel booking) | Orchestrated Saga + Transactional Outbox | Releases database locks immediately, survives coordinator crashes without blocking rows, and works with external APIs via idempotent compensations |
| Updating a search index or cache after a database write | Single commit + CDC / Transactional Outbox | Avoids dual writes by deriving secondary stores asynchronously from the primary's log |
Key idea. Once a write spans multiple databases or external APIs, a single local transaction can no longer make it atomic. Two-Phase Commit preserves atomicity and isolation by locking rows across all participants until a central coordinator logs the commit decision. The cost is holding locks across network round trips and blocking participants if the coordinator crashes. A Saga commits each service's local transaction immediately and undoes failures in reverse order using idempotent compensating transactions. It trades isolation for availability and pairs with the Transactional Outbox so database writes and outbound events commit together.
The next lesson, Failure Handling Patterns, covers timeouts, exponential-backoff retries, circuit breakers, and idempotency keys. These mechanisms keep remote calls and saga steps reliable when networks or downstream services fail.
Further Reading
- Sagas — Hector Garcia-Molina and Kenneth Salem (1987) — the original paper introducing long-lived transactions composed of local steps and compensating transactions.
- PREPARE TRANSACTION — PostgreSQL documentation — how PostgreSQL implements Phase 1 of Two-Phase Commit and why prepared transactions hold locks until committed or rolled back.
- Cloud Spanner: TrueTime and external consistency — Google Cloud — how Spanner layers Two-Phase Commit across Paxos groups so coordinator state is replicated.
- Transactional Outbox pattern — Debezium documentation — routing outbox table inserts from the database write-ahead log to a message broker without dual writes.