CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 10 of 12 · 1 min read

Quick notes

  • Offline evals are necessary but not sufficient. The rollout ladder is offline eval, shadow, canary, A/B test, full rollout, each rung exposing more users and answering a different question.
  • Shadow mode compares behaviour, cost and latency with zero user risk, but never shows the user's reaction and needs side effects mocked. A canary limits the blast radius.
  • An A/B test needs a pre-computed sample size, and peeking at the result breaks it; fix the size in advance or use a sequential method.
  • Canary detection depends on stage size: small regressions need big stages or longer time. A paired shadow comparison has more power than an unpaired split on the same requests.
  • Rollback must be one switch, with metrics tagged by version. A metric win is not proof; expert testers can catch what the metric misses.