Part 7 of 12 · 1 min read
The rollout ladder, and how to build it so it holds up
Five rungs from offline eval to full rollout, and the rules to apply.
| Rung | Users exposed | Finds | Main risk |
|---|---|---|---|
| Offline eval | none | regressions on known cases | cases unrepresentative |
| Shadow | none (output hidden) | behaviour, cost, latency on real inputs | side effects, no user reaction |
| Canary | 1 to 5 percent | real harm, early | too few users to see small changes |
| A/B test | split | does it improve a metric | peeking, underpowered |
| Full rollout | all | guardrails | slow rollback |
- Tag every request and metric with the version, so candidate and baseline are compared in the same window.
- Define guardrails and a rollback threshold before the rollout: error rate, p95 latency, cost per run, safety flags, user complaints.
- Make rollback one switch and test it.
- Shadow only with side effects disabled (read-only tools, mocked writes, no emails).
- Compute sample size in advance for the smallest effect you care about; do not stop early without a sequential method.
- Randomise by user, not request, if one user sees many requests, or results are correlated.
- Run long enough to cover weekly cycles and novelty effects.
- Combine with expert review, not only engagement metrics (module 10).
- Keep the offline eval gate (module 7) before any live exposure.