Part 2 of 8 · 3 min read
Why accept writes in more than one place
The limits of a single leader and the use cases that justify the extra work.
Think of a company with offices in London and Singapore. If only London may change the company's price list, Singapore staff wait 170 milliseconds or more for every change, and when the line to London breaks, they cannot change anything. If each office can change the list and send the changes to the other, the work goes on, but two offices may change the same price on the same day.
The book "Designing Data-Intensive Applications" names the downside of leader-based replication: there is one leader and all writes go through it. If you cannot reach the leader, for example because of a network break, you cannot write.
Multi-leader replication
In multi-leader replication (also called master-master or active/active), more than one node accepts writes. Each leader also acts as a follower of the other leaders, and forwards each change it accepts to them.
Example: a database has a leader in each of three data centres. Users write to the nearest one.
The book says a multi-leader set-up rarely makes sense inside one data centre, since the benefits seldom outweigh the extra complexity. Three cases justify it.
- Several data centres. A leader in each data centre. Within a data centre, ordinary leader-follower copying. Between data centres, the leaders copy each other. - Performance: with one leader, every write crosses the internet to its data centre. With several, each write is handled locally and sent on asynchronously, so the delay between regions is hidden from users. - Outages: if the data centre with the single leader fails, a failover must promote a follower elsewhere. With several leaders, each data centre keeps working on its own, and copying catches up later. - Network faults: traffic between data centres often crosses the public internet, less reliable than the local network. A single-leader design writes synchronously across that link and suffers. Multi-leader with asynchronous copying tolerates breaks better.
- Clients that work offline. A calendar app on a phone and a laptop must work with no connection. Each device has a local database that acts as a leader. The devices sync asynchronously when online. The lag may be hours or days. The book calls this the same idea as data centres, taken to the extreme: each device is a data centre and the network is very unreliable.
- Collaborative editing. Several people edit one document at once. Changes apply at once to the local copy and are sent on. To avoid conflicts, one could lock the document, which is like a single leader. For fast collaboration, changes are made tiny (a keystroke) and unlocked, with all the problems of multi-leader.
A worked example of the gain. Two data centres are 120 ms apart (round trip). A write in data centre B with a single leader in A takes about 120 ms plus local time, say 125 ms. With a leader in each, the write in B takes 5 ms. For a page that needs 3 sequential writes, the single leader costs 375 ms and the multi-leader 15 ms.
The book calls the retrofitting of multi-leader into many databases risky. There are often subtle pitfalls with auto-incrementing keys, triggers and integrity constraints. It is often considered dangerous territory to avoid if possible.
Check yourself:
- Name the three cases in which a multi-leader set-up is reasonable.
- Why does multi-leader hide the delay between data centres?