Distributed Transactions, Consensus & Coordination

On a single Postgres node, "debit account A and credit account B" is one BEGIN … COMMIT. The moment A and B live on different shards — or in different services, each with its own database — that guarantee evaporates. Now a crash between the two writes leaves money created or destroyed, and no amount of retrying fixes it without a protocol.

This subject is the toolkit for that world: how to commit atomically across nodes (2PC, and why nobody likes it), how to get the same business outcome without it (sagas, outbox), how a cluster agrees on anything at all (consensus — Paxos intuition, Raft in detail, quorums), what the coordination services built on consensus (ZooKeeper, etcd) are for, why distributed locks are much harder than they look, and how to name the consistency guarantee you are actually offering. Senior system-design interviews go here the moment you draw more than one database.

Boundaries: the trade-off between consistency and availability under partition is covered in The CAP Theorem subject; replication topologies and shard-key selection are covered in the Database Replication & Sharding subject; queue delivery semantics are covered in the Message Queues & Event Streaming subject. Here we cover the coordination protocols themselves.

Why Single-Node ACID Doesn't Survive Sharding

A single database gives atomicity via one write-ahead log: either the commit record hits disk or it doesn't. Isolation comes from one lock manager or one MVCC snapshot. Split the data across two nodes and:

• Atomicity: node 1 commits, node 2 crashes before committing. There is no single log to consult, and node 1 cannot un-commit.
• Isolation: a reader can see node 1's new state and node 2's old state at the same instant — a state that never "existed" in any serial order.
• Durability is fine per node; the problem is agreement about what was committed.

The options are: (1) coordinate commit across nodes with an atomic-commit protocol (2PC), (2) give up multi-node atomicity and design compensations (sagas), or (3) route the transaction so it touches one node (co-locate by shard key — often the best answer, and a design choice covered in the sharding subject).