Part 4 of 7 · 4 min read
When 2PC gets stuck
A crashed coordinator leaves participants in doubt, holding locks.
Think of the school trip again. All parents said yes and then the teacher fell ill before announcing. Parents cannot buy the bus tickets (maybe the trip is off) and cannot cancel (maybe it is on). They wait, and the seats stay reserved.
The DDIA book explains the weak spot. If the coordinator fails before sending prepare, participants can safely abort. But once a participant has voted yes, it cannot decide alone. It must wait to hear from the coordinator. A participant in this state is in doubt. A timeout does not help: if it aborts alone but the coordinator had told another participant to commit, atomicity breaks. Committing alone is equally unsafe.
In-doubt transaction
An in-doubt transaction is one where a participant has voted yes in the prepare phase but has not yet heard whether the coordinator decided to commit or abort.
Example: shard 1 voted yes for transaction 77 and then lost contact with the coordinator. It cannot commit or roll back until it learns the decision.
The only way out is for the coordinator to recover and read its log. That is why it writes the decision to disk before telling anyone. Transactions with no commit record in the log are aborted. For this reason 2PC is called a blocking atomic commit protocol.
Locks are the cost. A database holds row locks until commit or abort (modules 2 and 3). An in-doubt participant must keep its locks. If the coordinator takes 20 minutes to restart, those rows are locked for 20 minutes. If its log is lost, they are locked until an administrator steps in. Other transactions that touch those rows are blocked, which can make large parts of an application unavailable. The book reports orphaned in-doubt transactions do occur in practice, and even a database restart does not release their locks, since a correct implementation must keep them.
Why not three-phase commit? A three-phase version avoids blocking in theory, but assumes bounded network delay and bounded response times. With unbounded delays and process pauses (module 4), it cannot guarantee atomicity. A non-blocking protocol needs a perfect failure detector, which timeouts are not. So 2PC remains in use.
Other costs. The book cites a report that distributed transactions in MySQL are over ten times slower than single-node ones, mostly from the extra disk syncs for crash recovery and the extra network round trips. Many cloud services avoid them for operational reasons.
Two kinds of distributed transaction.
| Kind | Participants | Verdict |
|---|---|---|
| Database-internal | Nodes of one distributed database, such as VoltDB or MySQL Cluster | Can use any protocol and tune it. Often works well |
| Heterogeneous | Different technologies, such as two vendors' databases or a message broker | Needs a shared standard such as XA. Harder |
XA (1991) is a standard C API for 2PC across technologies. Many relational databases and brokers support it. The coordinator is usually a library in the application process, so if the application server dies, its log on local disk goes with it, and the participants sit in doubt until that server is restarted.
A heterogeneous transaction can give exactly-once message processing: acknowledge the message and commit the database writes together. If either fails, both abort and the message can be redelivered safely. This works only if every system takes part in the same protocol. An email server that does not support 2PC may send the email twice on a retry.
Making 2PC more robust: the UDS book notes that replicating the coordinator with a consensus algorithm such as Raft makes 2PC resilient to coordinator failures, and Google's Spanner combines 2PC with state machine replication across partitions.
A worked example of lock time. Transaction 77 locks A and B at second 0. The coordinator logs "commit" at second 1 and crashes before sending it. It restarts at second 1,200 (20 minutes). Rows A and B stay locked for about 1,200 seconds. If 50 other requests per second need row A, about 60,000 requests are blocked or fail meanwhile (made up to show the idea).
Common mistake
Assuming a participant can safely abort after a timeout once it has voted yes. It must wait, or atomicity breaks.
Check yourself:
- Why does a timeout not solve the in-doubt problem?
- What is the difference between a database-internal and a heterogeneous distributed transaction?