Part 7 of 12 · 1 min read
How to build it so it holds up
- Version the set and record where each case came from (date, trace id, annotator).
- Slice and report per slice, with intervals (module 1): slices of 10 cases have wide error bars.
- Add every production failure you fix as a regression case (module 7).
- Refresh on a schedule, because traffic drifts; keep the old set for trend comparison.
- Label with a written guideline, have two people label a subset and measure agreement (module 4).
- Scrub, then store: remove personal data and secrets before traces become test data.
- Keep a hold-out that nobody tunes against.