Part 4 of 12 · 2 min read
Progressive rollout: catching a bad release before everyone hits it
A small slice of traffic plus an automatic check catches a bad release far faster than a human watching a dashboard.
Think of a canary once carried into a mine shaft: a small, sensitive early-warning before the air became unsafe for everyone else down there.
In plain words: sending a new release to a small slice of traffic first, with an automated health check watching just that slice, catches a bad release and rolls it back far faster than waiting for a person to notice something is wrong.
A worked example. A release goes out with a 20 percent error rate against a normal 1 percent, for 30 minutes at 50 requests per second. Releasing it to everyone at once, with humans noticing and reverting after 8 minutes, produces 5,462 failed requests (6.1 percent of the total). Sending it first to a 5 percent canary slice, with an automatic health check rolling it back after just 1 minute of elevated errors, produces 936 failed requests (1.04 percent) — roughly a sixth of the damage.
The key lines.
# from code/mini-deploy/deploy.py
if strategy == "canary_auto" and m + 1 >= detect_after_min:
n_c = int(60 * rps * canary_frac); e = sum(rng.random() < bad_error_rate for _ in range(n_c)) / max(1, n_c)
if e > threshold: bad_live = False
Read it as: every minute, check the error rate of just the canary slice against a threshold. Once it is over, the bad release is pulled. The key is that only a small slice of users ever see the bad version before the check fires, instead of everyone.
Canary, blue-green, rolling
A canary release sends a small share of traffic to the new version first; blue-green runs old and new versions side by side and switches traffic between them; rolling replaces instances gradually.
Example: the 5 percent canary above, which limited exposure to a small slice while the automatic check decided whether to proceed.