Peeking false positives, canary stopping stage, and paired versus unpaired power.
Sample size grows with the inverse square of the effect: halving the detectable change roughly quadruples the users needed.
A big regression is caught early and cheaply; a small one needs scale. Choose stage sizes for the smallest regression you care about.
Peeking is the most common A/B mistake; decide the stopping rule first.
Pairing removes input-to-input variance, which is why shadow comparisons are more sensitive.
Design rules
Tag every request and metric with the version, so candidate and baseline are compared in the same window.
Define guardrails and a rollback threshold before the rollout: error rate, p95 latency, cost per run, safety flags, user complaints.
Make rollback one switch and test it.
Shadow only with side effects disabled (read-only tools, mocked writes, no emails).
Compute sample size in advance for the smallest effect you care about; do not stop early without a sequential method.
Randomise by user, not request, if one user sees many requests, or results are correlated.
Run long enough to cover weekly cycles and novelty effects.
Combine with expert review, not only engagement metrics (module 10).
Keep the offline eval gate (module 7) before any live exposure.
Interview questions, with the follow-ups
How do you ship a prompt or model change safely?
"Offline eval and the CI gate first, then shadow mode on live inputs with side effects off to compare behaviour, cost and latency, then a canary at 1 to 5 percent with version-tagged metrics and an automatic rollback threshold, then an A/B test sized in advance if I need to prove a business gain, then full rollout with a one-switch rollback." Follow-up: "What if shadow looks fine and the canary fails?" Shadow does not show user reaction or side effects; that is exactly the gap the canary covers.
How many users do you need for an A/B test?
"It depends on baseline, the smallest change I care about, power and alpha. For a 70 percent baseline and a 3-point change at 80 percent power it is about 3,550 per arm; for 2 points about 8,000. Underpowered tests mostly produce noise, as at 500 per arm where power was 18 percent." Follow-up: "Why not just stop when it is significant?" Repeated looks inflate false positives to 14 to 25 percent in my simulation; I either fix the sample size or use a sequential test.
Canary or A/B?
"Different jobs. A canary limits damage and catches large regressions fast; its stages are too small to prove a small improvement. An A/B test measures an effect with a planned sample. I use the canary as a safety stage before an A/B." Follow-up: "What did your canary miss?" A 2 to 3 percent error regression was mostly caught only at the 25 percent stage, with about 40 extra bad outcomes served.
What are the limits of shadow mode?
"Users never see the output, so I get no reaction signal, and tools with side effects must be read-only or mocked, which changes behaviour. It does give a paired comparison on identical inputs, which was about 2.4 times as sensitive as an unpaired split in my simulation." Follow-up: "How do you compare outputs?" With the eval tools from this level: judge, trajectory checks and cost metrics, using the human-calibrated judge from module 4.
Common mistakes
Rolling out to 100 percent after an offline eval only.
Shadow with live side effects.
No version tag on metrics.
Peeking and stopping at the first win.
Underpowered A/B tests read as "no difference".
No guardrail metrics, so a gain in one metric hides harm in another.
Rollback that needs a deploy.
Ignoring user-level correlation by randomising per request.
Trusting engagement metrics alone.
What the example does not do
No traffic router or feature flags; only the statistics.
Independent Bernoulli outcomes; real agent outcomes are correlated by user and task type.
Fixed z threshold for canary stops; real systems also use time windows and multiple metrics.
The Bonferroni split is conservative; sequential methods (for example alpha-spending) are more efficient and I did not implement them.
No novelty or seasonality effects.
Optional real experiment (I did not run this): put a real prompt change behind a flag, run it in shadow on a log replay, then a small canary, and compare your detection rates with these tables.
Check yourself
Which rung of the rollout ladder shows the candidate real inputs but hides its output from users? (Shadow mode. Any tool with side effects must be mocked or read-only, and you never see the user reaction.)
For the curious Peeking breaks an A/B test
With no real difference, stopping at the first p below 0.05 gave false positives of 14 percent with 5 looks, 19 with 10 and 25 with 20. Splitting alpha across the looks held them to 4, 3 and 2 percent. Detecting a 3-point change from a 70 percent baseline at 80 percent power needs about 3,554 users per arm. These rates are simulated.
What you will use daily at work
Practice
What it looks like
Tag every request with the version
Compare candidate and baseline in the same window.
Set the rollback threshold in advance
Error rate, p95 latency, cost per run, safety flags.
Make rollback one switch
Test it before you need it.
Compute the sample size first
Do not stop the test the moment it looks good.
Draft: confirm how your platform splits traffic and tags versions.
What you must be able to say (one page)
Offline evals are necessary but not sufficient. They run on cases you chose; production has users and inputs you did not. A rollout ladder exposes a change gradually: offline eval, shadow, canary, A/B test, full rollout, each rung exposing more users and answering a different question.
Shadow mode runs the candidate on live inputs but hides its output (and disables its side effects), so you compare behaviour, cost and latency with zero user risk. Limit: you never see the user's reaction, and any tool with side effects must be mocked or read-only.
Canary serves the candidate to a small slice (typically 1 to 5 percent), watches version-tagged metrics and rolls back on regression. It limits the blast radius, the number of users hurt before you notice.
A/B test splits users to estimate whether the change improves a business metric. It needs a pre-computed sample size. For a 70 percent baseline, detecting a 3-point change at 80 percent power needs about 3,554 users per arm (formula); 2 points needs 8,080; 5 points needs 1,251. With 500 per arm, power for a 3-point change is only 18 percent in my simulation.
Peeking breaks the test. Stopping at the first p below 0.05 across repeated looks, with no real difference, gave false positives of 14 percent (5 looks), 19 percent (10) and 25 percent (20). Splitting alpha across looks (Bonferroni) held them to 4, 3 and 2 percent (conservative). Fix the sample size in advance or use a sequential method.
Canary detection depends on stage size. With stages at 1, 5 and 25 percent of traffic (2,000, 5,000 and 20,000 requests), a 2 to 6 percent error-rate regression was stopped at stage 1/2/3 in 34 / 59 / 7 percent of simulations, with about 22 extra bad outcomes served. A subtle 2 to 3 percent regression was caught mostly at the last stage (13 / 12 / 74 percent) with about 40 extra bad outcomes. Small regressions need big stages or longer time.
Paired shadow beats unpaired A/B on power. On the same 500 requests, a 4-point gain was detected 34 percent of the time by a paired comparison (shadow: both versions see the same inputs) versus 14 percent by an unpaired split.
Rollback must be one switch, with metrics tagged by version so you compare like with like. A metric win is not proof: A/B tests of a widely reported 2025 model update looked positive while expert testers said it "felt slightly off" (see module 10).
Honest limits: simulated rates; two-proportion z-tests; independent requests; no novelty effects, no user-level correlation, no seasonality.