Diagnosing a 2PC Blocking Incident
A payments platform uses XA-style two-phase commit across an orders
database and a ledger database, coordinated by a single transaction
manager. During a deploy, the transaction manager process is killed
after collecting YES votes from both participants but before it sends
the COMMIT decision. On-call notices orders rows locked and new
transactions queuing behind them.
- Explain precisely why the participants cannot resolve this on their own.
- What are the two possible correct resolutions once the coordinator comes back, and what determines which one is correct?
- Propose two concrete changes to the deployment/architecture that would prevent this class of incident going forward.
1. Why participants are stuck
After voting YES, each participant has durably logged everything
needed to commit and has promised not to unilaterally abort — this is
the contract of phase 1. Neither participant knows what the other
voted or what the coordinator decided. If orders guesses ABORT but
the coordinator had actually decided COMMIT (because ledger also
voted YES), the system ends up inconsistent — one database committed
conceptually while the other rolled back. So each participant must
hold its locks and wait for the coordinator's decision; this is
exactly the "in-doubt" blocking window 2PC is known for.
2. Two possible resolutions
The coordinator's own durable log is the source of truth: it wrote the COMMIT (or ABORT) decision record before crashing, as part of phase 2. On restart, it reads that log. If a COMMIT record exists, it re-sends COMMIT to both participants (idempotently — they may have already applied it or may not have). If no decision was ever logged (crash happened between collecting votes and writing the decision), the safe choice is ABORT, since no participant could have received a COMMIT yet. Which one is correct is determined entirely by what the coordinator's log shows, not by anything the participants can infer locally.
3. Preventing recurrence
- Run the coordinator as a replicated, consensus-backed service (its decision log stored via Raft/etcd, not a single process) so a process kill during deploy does not create a blocking window at all — a majority of coordinator replicas can complete the decision.
- Avoid the pattern altogether where possible: redesign so orders and ledger writes are co-located (single database) or replace 2PC with a saga (charge, then confirm order; compensate with a reversal ledger entry on failure) so no cross-database locks are held during a deploy window.
Share this question