Part 1 of 4 · 1 min read
Overview
Agent changes (a prompt edit, a model upgrade, a new tool) are non-deterministic and hard to cover with offline tests (module 1). Interviewers ask "how do you ship a model or prompt change safely?" The answer is the ladder, a sample-size calculation, guardrail metrics, and a rollback you have tested.
About the example, and what is real
Real: the sample-size formula, z-test, power, peeking, canary and shadow simulations and the 10 tests in code/mini-rollout. Assumed: baseline rates, independent Bernoulli outcomes, stage sizes of 2,000, 5,000 and 20,000 requests, a one-sided rollback threshold z above 2.326, and the paired noise model for shadow. Not real: no real traffic router or feature-flag system; numbers come from random simulation with fixed seeds and will differ with other parameters.