Part 9 of 12 · 2 min read
Mistakes that turn a rollout into an incident
Common mistakes, and what the example does not do.
Mistakes that turn a rollout into an incident
- Rolling out to 100 percent after an offline eval only.
- Shadow with live side effects.
- No version tag on metrics.
- Peeking and stopping at the first win.
- Underpowered A/B tests read as "no difference".
- No guardrail metrics, so a gain in one metric hides harm in another.
- Rollback that needs a deploy.
- Ignoring user-level correlation by randomising per request.
- Trusting engagement metrics alone.
What the example does not do
About this example What the example does not do
- No traffic router or feature flags; only the statistics.
- Independent Bernoulli outcomes; real agent outcomes are correlated by user and task type.
- Fixed z threshold for canary stops; real systems also use time windows and multiple metrics.
- The Bonferroni split is conservative; sequential methods (for example alpha-spending) are more efficient and they are not implemented.
- No novelty or seasonality effects.
- Optional real experiment: put a real prompt change behind a flag, run it in shadow on a log replay, then a small canary, and compare your detection rates with these tables.
Check yourself
Which rung of the rollout ladder shows the candidate real inputs but hides its output from users?
Show answer
Shadow mode. Any tool with side effects must be mocked or read-only, and you never see the user reaction.
For the curious Peeking breaks an A/B test
With no real difference, stopping at the first p below 0.05 gave false positives of 14 percent with 5 looks, 19 with 10 and 25 with 20. Splitting alpha across the looks held them to 4, 3 and 2 percent. Detecting a 3-point change from a 70 percent baseline at 80 percent power needs about 3,554 users per arm. These rates are simulated.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Tag every request with the version | Compare candidate and baseline in the same window. |
| Set the rollback threshold in advance | Error rate, p95 latency, cost per run, safety flags. |
| Make rollback one switch | Test it before you need it. |
| Compute the sample size first | Do not stop the test the moment it looks good. |