Part 10 of 12 · 1 min read
Quick notes
- Production traces are the best source of eval cases, because they show what users actually ask. Slice the data and keep several evaluation sets.
- The sampling method decides what the set can tell you: a random sample often misses a rare category; stratified sampling always includes it.
- A set that covers rare cases no longer estimates the overall rate unless you weight it by traffic share, and sampling on a flagged signal stays biased because selection depends on the outcome.
- Two sets, two jobs: a random, production-shaped set to estimate quality, and a stratified or adversarial set to find failures and protect rare behaviours.
- Deduplicate before you count, keep the set out of your prompts, extend with reviewed synthetic cases, and scrub personal data before logs become test data.