CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 6 of 12 · 2 min read

What the example found

Kappa stability by calibration-set size, pairwise accuracy under position and length bias, the correction's error at different sample sizes, and the human-versus-judge agreement ceiling.

How to read the kappa-stability table: each row is a different number of labelled cases, for a judge with a true-positive rate of 0.9 and false-positive rate of 0.15, measured over 400 repeated trials.

Labelled cases Median kappa 5th percentile 95th percentile
20 0.73 0.44 1.00
50 0.75 0.58 0.88
100 0.75 0.61 0.85
300 0.75 0.68 0.81
1,000 0.75 0.71 0.79

What to notice: the median kappa barely moves across all five rows (0.73 to 0.75). The 5th-to-95th percentile spread shrinks from 0.56 points wide at 20 labels to just 0.08 points wide at 1,000. The judge's true reliability never changed. Only how precisely a small sample could pin it down changed.

How to read the pairwise-bias table: each row is a different bias configuration tested on 4,000 simulated pairs, where the judge correctly identifies the better answer 70% of the time absent any bias.

Judge Accuracy on decided pairs Undecided
No bias 85% 0%
30% position bias 75% 0%
30% position bias, both orders, keep consistent 92% 42%
30% length bias, longer answer is better 90% 0%
30% length bias, longer answer is worse 61% 0%

What to notice: the swap-and-keep-consistent fix for position bias pushes accuracy above the unbiased baseline (92% vs 85%). But it does this only by discarding 42% of all pairs as undecided. So the real trade is accuracy on what is left versus coverage of the whole set. It is not a free win.

How to read the correction-error table: each row is a different calibration-set size; the true pass rate is 30% and the judge's raw reported rate is consistently 41% before correction.

Labelled cases Raw judge rate error (points) Corrected error (points)
20 11 12
50 11 7
100 11 5
500 11 2

What to notice: at 20 labels, correcting makes the estimate worse, not better (11 to 12 points of error). A small calibration set is not just unhelpful. It can mislead you into trusting a "corrected" number that is less accurate than simply reporting the judge's raw rate.

How to read the human-ceiling comparison: three kappa values measured on the same underlying task.

Human-human kappa: 0.67. Judge-against-truth kappa: 0.73. Judge-against-a-noisy-human kappa: 0.60.

What to notice: the judge's kappa against a single noisy human (0.60) is lower than human-human agreement (0.67). That could look like the judge failing to match human performance. But the same judge scores 0.73 against ground truth, higher than the human-human ceiling. So comparing a judge to one noisy annotator understates it. The real benchmark is agreement with truth, not with any single human.

{{svg:loop|Write a rubric, label a set, run the judge, compare, then fix or correct.}} {{svg:results|Pairwise accuracy under position and length bias, and the error of a raw and a corrected judge pass rate.}}