Part 4 of 12 · 3 min read
A canary limits the damage of a bad change while it's still being measured
Exposing a change to a small slice of traffic first means a mistake only hurts that small slice, not everyone, while you're still deciding whether it's actually a mistake.
Think of the canary once carried into coal mines. It was a small, more sensitive creature exposed first to bad air. The miners behind it could then turn back before they themselves were harmed.
In plain words: a canary release serves the candidate to a small share of real traffic first, blast radius is how many users or actions a bad change can affect, and rollback means returning all traffic to the previous version -- so a canary's whole job is detecting a real regression and triggering rollback while the blast radius is still small, with bigger stages needed to reliably catch smaller regressions in a reasonable amount of traffic.
A worked example. Across three stages (1%, 5%, 25% of traffic, with populations of 2,000, 5,000 and 20,000 requests), a large error-rate regression (2% baseline rising to 6%) gets stopped at the first stage 34% of the time, the second stage 59% of the time, and the third stage only 7% of the time -- it's almost always caught before full exposure, with about 22 extra bad outcomes served on average. A much subtler regression (2% to 3%) is caught overwhelmingly at the last, largest stage (74% of the time), with roughly 40 extra bad outcomes served -- nearly double the large regression's damage, because the subtle one survives through the small early stages where there isn't enough traffic to tell it apart from noise.
The key lines. From code/mini-rollout/rollout.py:
# from code/mini-rollout/rollout.py
def canary(stages=((0.01, 2000), (0.05, 5000), (0.25, 20000)), base_bad=0.02, cand_bad=0.06, zcrit=2.326, trials=2000, seed=3):
for i, (share, pop) in enumerate(stages):
n = int(share * pop)
bad = sum(rng.random() < cand_bad for _ in range(n))
extra += n * (cand_bad - base_bad)
se = math.sqrt(base_bad * (1 - base_bad) / n)
if (bad / n - base_bad) / se > zcrit:
stopped[i] += 1
break
Read it as: extra accumulates n * (cand_bad - base_bad) at every stage reached, whether or not that stage triggers a stop. The damage tally keeps running through every stage the rollout actually passes through. That is why a subtle regression that slips past the small early stages ends up costing more extra bad outcomes. By the time it is finally caught at a bigger stage, the tally has grown.
Canary, blast radius, rollback
A canary release serves the candidate to a small share of real traffic first. Blast radius is how many users or actions a bad change can affect. Rollback is returning all traffic to the previous version, ideally with one flag.
Example: the subtle 2-to-3-point regression causing roughly 40 extra bad outcomes versus the obvious 2-to-6-point regression's 22 is a direct, counterintuitive consequence of blast radius: the harder-to-detect change does more damage precisely because it takes longer to trigger the rollback that would have limited it.