Part 10 of 12 · 1 min read
Quick notes
- Offline evals are necessary but not sufficient. The rollout ladder is offline eval, shadow, canary, A/B test, full rollout, each rung exposing more users and answering a different question.
- Shadow mode compares behaviour, cost and latency with zero user risk, but never shows the user's reaction and needs side effects mocked. A canary limits the blast radius.
- An A/B test needs a pre-computed sample size, and peeking at the result breaks it; fix the size in advance or use a sequential method.
- Canary detection depends on stage size: small regressions need big stages or longer time. A paired shadow comparison has more power than an unpaired split on the same requests.
- Rollback must be one switch, with metrics tagged by version. A metric win is not proof; expert testers can catch what the metric misses.