CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 10 of 12 · 1 min read

Quick notes

  • Production traces are the best source of eval cases, because they show what users actually ask. Slice the data and keep several evaluation sets.
  • The sampling method decides what the set can tell you: a random sample often misses a rare category; stratified sampling always includes it.
  • A set that covers rare cases no longer estimates the overall rate unless you weight it by traffic share, and sampling on a flagged signal stays biased because selection depends on the outcome.
  • Two sets, two jobs: a random, production-shaped set to estimate quality, and a stratified or adversarial set to find failures and protect rare behaviours.
  • Deduplicate before you count, keep the set out of your prompts, extend with reviewed synthetic cases, and scrub personal data before logs become test data.