CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 4 of 12 · 2 min read

Progressive rollout: catching a bad release before everyone hits it

A small slice of traffic plus an automatic check catches a bad release far faster than a human watching a dashboard.

Think of a canary once carried into a mine shaft: a small, sensitive early-warning before the air became unsafe for everyone else down there.

In plain words: sending a new release to a small slice of traffic first, with an automated health check watching just that slice, catches a bad release and rolls it back far faster than waiting for a person to notice something is wrong.

A worked example. A release goes out with a 20 percent error rate against a normal 1 percent, for 30 minutes at 50 requests per second. Releasing it to everyone at once, with humans noticing and reverting after 8 minutes, produces 5,462 failed requests (6.1 percent of the total). Sending it first to a 5 percent canary slice, with an automatic health check rolling it back after just 1 minute of elevated errors, produces 936 failed requests (1.04 percent) — roughly a sixth of the damage.

The key lines.

# from code/mini-deploy/deploy.py
if strategy == "canary_auto" and m + 1 >= detect_after_min:
    n_c = int(60 * rps * canary_frac); e = sum(rng.random() < bad_error_rate for _ in range(n_c)) / max(1, n_c)
    if e > threshold: bad_live = False

Read it as: every minute, check the error rate of just the canary slice against a threshold. Once it is over, the bad release is pulled. The key is that only a small slice of users ever see the bad version before the check fires, instead of everyone.

Canary, blue-green, rolling

A canary release sends a small share of traffic to the new version first; blue-green runs old and new versions side by side and switches traffic between them; rolling replaces instances gradually.

Example: the 5 percent canary above, which limited exposure to a small slice while the automatic check decided whether to proceed.