Part 4 of 7 · 3 min read
Production rollout
Start small, then widen as confidence grows.
UDS's rule: first release to a small number of production instances. This is canary testing. The aim is to surface undetected problems quickly, before they cause wide damage. If all health checks pass, the artifact spreads in stages to the rest of the fleet.
Two details from UDS: - During the rollout, a part of the fleet cannot serve traffic, and the others must pick up the slack. Enough capacity must remain so that users see no slowdown. - With several regions, start with a low-traffic region and split the rest into sequential stages. More stages mean a slower release, so speed up once confidence has built. For example: stage one a single region, stage two a larger region, stage three N regions at the same time.
A made-up plan for 200 instances in 4 regions. Stage 1: 4 instances in the smallest region (2 percent), bake 30 minutes. Stage 2: that whole region, bake 30 minutes. Stage 3: the second region, bake 15 minutes. Stage 4: the last two regions together. If stage 1 broke something, only 2 percent of the traffic of one region saw it.
BM describes two related techniques.
Blue-green deployment. Two complete copies of the software run. Only one gets real requests. BM's example: version 123 of the customer service is live, and version 456 is deployed beside it with no traffic. Tests run against it in place. If they pass, production traffic is switched to 456. The old version stays for a while for a fast fallback. It needs the ability to send traffic to different hosts (change DNS or load-balancer settings) and enough hosts to run both versions. Done well, the switch is invisible to customers, giving zero downtime.
Canary releasing (BM). Real traffic is sent in growing amounts to the new version to check that it behaves as expected: for example responding within 500 ms, or showing the same error rate as the old version. BM mentions Netflix, which runs a new version and a baseline cluster side by side for hours and scores both. Versions coexist longer than in blue-green, traffic is split in varying shares, and the setup is more complex. A team can also shadow production traffic by copying requests to the canary while only the real version's answers reach customers. That avoids harm to users, but replaying requests is hard if they are not idempotent (Level 6).
BM's line: both techniques separate deployment (putting software on machines) from release (sending users to it).
Canary and blue-green
A canary release sends a small and growing share of real traffic to the new version. A blue-green deployment runs the new version beside the old one with no traffic, then switches all traffic over at once.
Example: canary: 2 percent of requests go to v2, then 10, then 50. Blue-green: v2 is tested idle, then the load balancer switches 100 percent from v1 to v2.
Check yourself: 1. Why must capacity remain during an incremental rollout? 2. How does blue-green give a fast rollback? 3. Why is shadow traffic risky for requests that are not idempotent?