Part 8 of 12 · 2 min read
Answering paper-reading questions in an interview
A paper reports a 2-point improvement. How do you decide whether to believe it?
"I read the claim, setup and evidence. I check the test size against the gap: detecting 2 points needs on the order of thousands of items, and in the example's simulation two equal models looked 2 points apart 25 percent of the time on 300 items. Then seeds and intervals, baseline tuning (a 20-try method beats a 1-try baseline of the same quality by about 2.8 points), whether variants were chosen on the test set, and contamination." Follow-up: "What do you do next?" Reproduce on my own data with a fixed budget for both sides.
How many evaluation items do you need to compare two prompts?
"It scales with 1 over the gap squared. A rough rule says about 1,000 items for a 3-point gap and 10,000 for 1 point; my 80-percent-power calculation gives roughly 1,700 paired or 3,500 independent items for 3 points. I use a paired comparison on the same items and bootstrap the result." Follow-up: "What if you only have 200 items?" Accept only large gaps (about 10 points at 20 percent disagreement needed 155 items), and say so.
Why can leaderboards and benchmark tables mislead?
"Benchmarks get contaminated, saturate, and differ in selection: two leaderboards can choose different benchmarks and rank the same models differently. Selecting on the test set inflates scores, and one best-of-many number overstates the truth. Public benchmarks filter bad models; my own evaluation picks the best for my application." Follow-up: "What would you ask the authors?" Which items were held out, how many variants were tried, and results on a private set.
What is Simpson's paradox and why does it matter for evaluation?
"A model can win on each slice and lose overall because the slices are weighted differently. In my invented example model A beats B on both slices (94 against 90 and 32 against 28 percent) but loses overall (40.9 against 81.1). I report by slice and decide which mix reflects production." Follow-up: "How do you prevent it?" Evaluate on a set that mirrors the production distribution and on named slices.