CSE Roadmap

Core tier · about 3 hours · 7 parts

Part 5 of 7 · 2 min read

Rollbacks

After every step, check, wait and decide.

After each step, the pipeline must judge whether the new artifact is healthy, and stop and roll back if not. UDS lists signals: end-to-end test results, health metrics such as latencies and errors, alerts, and health endpoints. It is not enough to watch only the service being released. Watch its upstream and downstream services too, since the damage may be indirect.

Bake time is the wait between one step and the next. Some issues appear only later, such as a slowdown that shows at peak time. The wait can also depend on the number of requests seen by given endpoints, to be sure the API has been exercised. As confidence grows, bake time can shrink.

When a signal worsens, the pipeline stops. It can roll back by itself, or alert the on-call engineer to decide. The engineer then chooses to retry the failed stage (perhaps something else was going out at the same time), roll back the whole release, or stop and wait for a hotfix to roll forward. Rolling forward is needed when rolling back is impossible, which happens when a backward-incompatible change went out. UDS notes pipelines can be set to run only in business hours to spare on-call staff.

Rule of thumb: rolling forward is much riskier than rolling back, so every change should be backward compatible. The most common cause of incompatibility is a change to a serialisation format, used for storage or between services.

Making an incompatible change safe. Break it into several compatible ones. UDS's example is a message schema between a producer and a consumer: 1. Prepare: change the consumer to understand both old and new formats. 2. Activate: change the producer to write the new format. 3. Cleanup: drop the old format from the consumer, only once there is confidence that activation will not need a rollback.

Each step can be rolled back alone. An automated upgrade-and-downgrade test in pre-production can check whether a change is really safe to roll back.

BM adds a point about priorities. Testing before release can never bring the chance of failure to zero, and past a point it gives diminishing returns. Faster recovery (shorter mean time to repair, MTTR) can beat more tests (a longer mean time between failures, MTBF). Fast rollbacks with good monitoring are the simplest way to get that.

Common mistake

Changing a message or database format in one step. If the new version has a bug, the old version cannot read what the new one wrote, so rollback is no longer possible.

Check yourself: 1. Name three health signals a pipeline can use. 2. What is bake time for? 3. In the three-step change, which step is done last and why?