CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 7 of 12 · 1 min read

How to build it so it holds up

  • Version the set and record where each case came from (date, trace id, annotator).
  • Slice and report per slice, with intervals (module 1): slices of 10 cases have wide error bars.
  • Add every production failure you fix as a regression case (module 7).
  • Refresh on a schedule, because traffic drifts; keep the old set for trend comparison.
  • Label with a written guideline, have two people label a subset and measure agreement (module 4).
  • Scrub, then store: remove personal data and secrets before traces become test data.
  • Keep a hold-out that nobody tunes against.