CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 8 of 12 · 2 min read

Answering rollout questions in an interview

How do you ship a prompt or model change safely?

"Offline eval and the CI gate first, then shadow mode on live inputs with side effects off to compare behaviour, cost and latency, then a canary at 1 to 5 percent with version-tagged metrics and an automatic rollback threshold, then an A/B test sized in advance if I need to prove a business gain, then full rollout with a one-switch rollback." Follow-up: "What if shadow looks fine and the canary fails?" Shadow does not show user reaction or side effects; that is exactly the gap the canary covers.

How many users do you need for an A/B test?

"It depends on baseline, the smallest change I care about, power and alpha. For a 70 percent baseline and a 3-point change at 80 percent power it is about 3,550 per arm; for 2 points about 8,000. Underpowered tests mostly produce noise, as at 500 per arm where power was 18 percent." Follow-up: "Why not just stop when it is significant?" Repeated looks inflate false positives to 14 to 25 percent in the example's simulation; I either fix the sample size or use a sequential test.

Canary or A/B?

"Different jobs. A canary limits damage and catches large regressions fast; its stages are too small to prove a small improvement. An A/B test measures an effect with a planned sample. I use the canary as a safety stage before an A/B." Follow-up: "What did your canary miss?" A 2 to 3 percent error regression was mostly caught only at the 25 percent stage, with about 40 extra bad outcomes served.

What are the limits of shadow mode?

"Users never see the output, so I get no reaction signal, and tools with side effects must be read-only or mocked, which changes behaviour. It does give a paired comparison on identical inputs, which was about 2.4 times as sensitive as an unpaired split in the example's simulation." Follow-up: "How do you compare outputs?" With the eval tools from this level: judge, trajectory checks and cost metrics, using the human-calibrated judge from module 4.