Part 6 of 8 · 2 min read
Compatibility in a rolling release
Old and new code at the same time, and the order of changes that keeps both working.
Think of changing the wheels of a moving train, one wagon at a time. For a while, some wagons run old wheels and some new. Every part must work with either.
The book says why. A large application cannot change everywhere at once.
- A server application can be upgraded in a rolling upgrade (also called a staged rollout): a few nodes at a time get the new version, the team checks that it runs smoothly, and the rest follow. This avoids downtime and encourages frequent releases.
- A client application, such as a phone app, updates when the user chooses. Old versions can live for years.
So old and new code, and old and new data formats, exist at once. The book defines the two directions of compatibility.
- Backward compatibility: newer code can read data written by older code.
- Forward compatibility: older code can read data written by newer code.
Backward compatibility is usually easy, since the author of the new code knows the old format. Forward compatibility is harder, because old code must ignore additions it knows nothing about.
A worked example of a rolling release. A service changes the format of a message from v1 to v2. It runs on 10 servers, updated 2 at a time.
| Step | Servers on v1 | Servers on v2 | What must hold |
|---|---|---|---|
| 0 | 10 | 0 | v1 writes and reads v1 |
| 1 | 8 | 2 | v2 reads v1 data (backward). v1 reads v2 data (forward) |
| 3 | 4 | 6 | Both directions still needed |
| 5 | 0 | 10 | Only v2. Old data may still be stored |
If v2 data cannot be read by v1, then during steps 1 to 4 the v1 servers fail whenever they meet a message written by a v2 server. With a rolling upgrade, that is a guaranteed fault.
The safe order for a breaking change is to split it in two releases:
- Release A teaches readers to understand both the old and new formats, but still writes the old one. Backward and forward compatibility hold.
- Wait until every instance runs Release A, and no old reader remains.
- Release B starts writing the new format.
- Later, Release C removes the code for the old format, once no old data remains.
This is sometimes called an expand and contract change. It takes three releases but never breaks anything.
Common mistake
Changing the writer and the reader in one release and relying on the idea that everything is deployed at once. Rolling upgrades, caches, queued messages and old phone apps all keep old formats alive.
Check yourself:
- Which direction is harder to maintain, backward or forward, and why?
- Why does a breaking change need two or three releases?