CSE Roadmap

Core tier · about 3 hours · 8 parts

Part 5 of 8 · 2 min read

Adding a follower, and when a follower fails

Copying a database that is changing, and catching up after a gap.

Think of copying a busy library catalogue to a new branch. If the librarian photocopies the cards for an hour while new cards keep being added, the copy will be a mix of old and new. A better way: photograph the catalogue at one moment, note the number of the last card, take the photograph to the branch, and then send every card added after that number.

The book gives these steps to add a follower without downtime.

  1. Take a consistent snapshot of the leader's database at one moment, without locking the whole database if possible. Most databases can do this for backups.
  2. Copy the snapshot to the new follower.
  3. The follower connects to the leader and asks for all changes since the snapshot. This needs the snapshot to be tied to an exact position in the leader's replication log. PostgreSQL calls this position the log sequence number. MySQL calls it the binlog coordinates.
  4. When the follower has processed the backlog, it has caught up and carries on receiving changes as they happen.

A worked example. A snapshot of 500 GB is copied over 2 hours. During those 2 hours the leader took 1,000 writes per second: 7,200 seconds × 1,000 = 7,200,000 changes to replay. The follower applies 5,000 changes per second, while the leader keeps producing 1,000 per second. The backlog shrinks by 5,000 − 1,000 = 4,000 per second, so catch-up takes 7,200,000 ÷ 4,000 = 1,800 seconds, which is 30 minutes. If the follower could apply only 1,000 changes per second, the backlog would never shrink and it would never catch up.

Follower failure. This is the easy case. Each follower keeps a log of the changes it received. After a crash or a break in the network, it knows the last transaction it processed, asks the leader for everything since then, applies it and continues. This is catch-up recovery.

Check yourself:

  • Why is copying the files of a running database not enough to make a follower?
  • A snapshot takes 3 hours, the leader makes 500 changes per second, and the follower applies 2,000 per second. How long does catch-up take?