CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 8 of 12 · 2 min read

Answering eval-set questions in an interview

How do you build an eval set for an agent that is already live?

"From production traces. I collect them, scrub personal data, deduplicate, and sample in two ways: a random, production-shaped set to estimate overall quality, and a stratified set with deliberate coverage of slices and rare failures. I label against a written guideline, check agreement on a subset, version the set and freeze it. In the example's simulation a random 50-case set contained a 1 percent category 39 percent of the time and a stratified one every time." Follow-up: "How do you get an overall number from the stratified set?" Weight each slice by its traffic share; naive averaging over-stated failures by 18.8 points in the example's test.

Why not just use the thumbs-down traces?

"They are biased: users flag some failures and not others, and flag some passes. In the example's simulation, a set built on flagged traces over-stated the failure rate by 21 points, and slice weights did not remove the bias because selection depended on the outcome. They are good for finding failure types, so I use them for the failure set and use a random set to estimate the rate." Follow-up: "Can you correct it?" Only with known inclusion probabilities, or by labelling a random sample.

What can go wrong with an eval set?

"Duplicates that inflate one task, leakage into the prompt or few-shot examples so the score measures memory, stale cases as traffic drifts, tiny slices with huge error bars, and labels that disagree. I deduplicate, check overlap with prompts, refresh, report intervals, and measure annotator agreement." Follow-up: "How do you know the set is big enough?" The interval must be narrower than the change I want to detect (module 1), per slice that I care about.

Can a model write your eval cases?

"Yes, to extend a real set: mutate real cases, add ambiguity, produce edge cases. Humans must review them, and I keep them labelled as synthetic and out of the production-shaped set, because they do not have the real distribution and can share the generating model's blind spots." Follow-up: "What is the risk?" Evaluating a system with cases from the same model family can overstate quality.