Part 2 of 7 · 2 min read
Why automate
Rare, large, manual releases hide the change that broke things.
A kitchen that cooks one huge meal a month finds it hard to tell which dish caused the stomach ache. UDS says the same of software. When a release needs a manual process, it happens rarely, and changes pile up for days or weeks. When a batch fails, it is hard to find the breaking change, and the whole team is interrupted. The developer who started the release must also watch dashboards and decide whether to roll back.
UDS calls manual deployment a terrible use of engineering time, and the problem grows with the number of services. The answer is to automate the whole path, including the rollback, with a continuous delivery and deployment (CD) pipeline. Once a change is merged, it rolls out safely by itself, and the developer moves on to the next task.
CD takes real investment in safeguards, monitoring and automation. If a regression appears, the artifact (the deployable thing that holds the change) is either reverted to the build before it, or replaced by the next build if that one carries a hotfix. UDS stresses a balance: a safer rollout takes longer, and a good pipeline trades the two off well.
The SRE book gives advice for the moment something breaks. Check the recent changes first and consider reverting them, especially any that touched capacity or the shape of requests. A change log makes that check quick. It also suggests pushing updates off-peak, so a restart wave does not hit the system at its busiest.
Continuous delivery and deployment (CD)
A CD pipeline is an automated path that takes a merged code change, builds it, tests it and releases it to production in steps, with automatic checks and rollback.
Example: a developer merges at 10:00. By 11:30 the change is live in one region and, after the health checks pass, in all of them, with nobody clicking anything.
Check yourself: 1. Why do manual releases make failures harder to diagnose? 2. What is an artifact? 3. What does UDS say a good pipeline balances?