CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 9 of 12 · 2 min read

Mistakes that make an eval set misleading

Common mistakes, and what the example does not do.

Mistakes that make an eval set misleading

  • Tuning prompts on the eval set and reporting its score.
  • Few-shot examples copied from the eval set.
  • Reading an oversampled set as the production rate.
  • No slices, so a regression in a small, important group is invisible.
  • Counting duplicates as independent cases.
  • Unlabelled provenance, so nobody knows what the set represents.
  • Personal data in test cases.
  • Never refreshing.

What the example does not do

About this example What the example does not do
  • No real traces or annotators; the failure label is known exactly.
  • Five categories only, assigned by construction; real slicing is messy.
  • Weighting by category only; real corrections can need several variables.
  • Simple lexical deduplication: no embeddings, and no normalisation of numbers or names.
  • No privacy scrubbing (Level 10 module 6).
  • Optional real experiment: take 500 real traces, label 100 random and 100 stratified, and compare the estimated rate and the number of distinct failure types found.

Check yourself

You sample 50 random traces and one category is 1 percent of traffic. Will the sample tell you about it?

Show answer

Usually not: that category appeared in the sample only 39 percent of the time in the simulation. Stratified sampling always had it, but then each category must be weighted by traffic share to estimate the overall rate.

For the curious Stratified sets find failures but misread the rate

True failure rate was 8.8 percent. A random sample was off by 0.2 points. A stratified sample read naively was off by 18.8 points, because rare, failure-prone cases were over-represented; weighting each category by its traffic share brought it back to 0.2. Keep two sets: a production-shaped one to estimate quality and a stratified one to find and guard failures. All traffic shares and rates are assumptions.

What you will use daily at work

Practice What it looks like
Version every eval set Record where each case came from: date, trace id, annotator.
Report per slice with intervals A slice of 10 cases has very wide error bars.
Turn each fixed production failure into a case It then runs in CI as a regression test.
Scrub personal data first Remove it before logs become test data.