CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 7 of 12 · 1 min read

The rollout ladder, and how to build it so it holds up

Five rungs from offline eval to full rollout, and the rules to apply.

Rung Users exposed Finds Main risk
Offline eval none regressions on known cases cases unrepresentative
Shadow none (output hidden) behaviour, cost, latency on real inputs side effects, no user reaction
Canary 1 to 5 percent real harm, early too few users to see small changes
A/B test split does it improve a metric peeking, underpowered
Full rollout all guardrails slow rollback
  • Tag every request and metric with the version, so candidate and baseline are compared in the same window.
  • Define guardrails and a rollback threshold before the rollout: error rate, p95 latency, cost per run, safety flags, user complaints.
  • Make rollback one switch and test it.
  • Shadow only with side effects disabled (read-only tools, mocked writes, no emails).
  • Compute sample size in advance for the smallest effect you care about; do not stop early without a sequential method.
  • Randomise by user, not request, if one user sees many requests, or results are correlated.
  • Run long enough to cover weekly cycles and novelty effects.
  • Combine with expert review, not only engagement metrics (module 10).
  • Keep the offline eval gate (module 7) before any live exposure.