Part 6 of 12 · 2 min read
What the example found
Kappa stability by calibration-set size, pairwise accuracy under position and length bias, the correction's error at different sample sizes, and the human-versus-judge agreement ceiling.
How to read the kappa-stability table: each row is a different number of labelled cases, for a judge with a true-positive rate of 0.9 and false-positive rate of 0.15, measured over 400 repeated trials.
| Labelled cases | Median kappa | 5th percentile | 95th percentile |
|---|---|---|---|
| 20 | 0.73 | 0.44 | 1.00 |
| 50 | 0.75 | 0.58 | 0.88 |
| 100 | 0.75 | 0.61 | 0.85 |
| 300 | 0.75 | 0.68 | 0.81 |
| 1,000 | 0.75 | 0.71 | 0.79 |
What to notice: the median kappa barely moves across all five rows (0.73 to 0.75). The 5th-to-95th percentile spread shrinks from 0.56 points wide at 20 labels to just 0.08 points wide at 1,000. The judge's true reliability never changed. Only how precisely a small sample could pin it down changed.
How to read the pairwise-bias table: each row is a different bias configuration tested on 4,000 simulated pairs, where the judge correctly identifies the better answer 70% of the time absent any bias.
| Judge | Accuracy on decided pairs | Undecided |
|---|---|---|
| No bias | 85% | 0% |
| 30% position bias | 75% | 0% |
| 30% position bias, both orders, keep consistent | 92% | 42% |
| 30% length bias, longer answer is better | 90% | 0% |
| 30% length bias, longer answer is worse | 61% | 0% |
What to notice: the swap-and-keep-consistent fix for position bias pushes accuracy above the unbiased baseline (92% vs 85%). But it does this only by discarding 42% of all pairs as undecided. So the real trade is accuracy on what is left versus coverage of the whole set. It is not a free win.
How to read the correction-error table: each row is a different calibration-set size; the true pass rate is 30% and the judge's raw reported rate is consistently 41% before correction.
| Labelled cases | Raw judge rate error (points) | Corrected error (points) |
|---|---|---|
| 20 | 11 | 12 |
| 50 | 11 | 7 |
| 100 | 11 | 5 |
| 500 | 11 | 2 |
What to notice: at 20 labels, correcting makes the estimate worse, not better (11 to 12 points of error). A small calibration set is not just unhelpful. It can mislead you into trusting a "corrected" number that is less accurate than simply reporting the judge's raw rate.
How to read the human-ceiling comparison: three kappa values measured on the same underlying task.
Human-human kappa: 0.67. Judge-against-truth kappa: 0.73. Judge-against-a-noisy-human kappa: 0.60.
What to notice: the judge's kappa against a single noisy human (0.60) is lower than human-human agreement (0.67). That could look like the judge failing to match human performance. But the same judge scores 0.73 against ground truth, higher than the human-human ceiling. So comparing a judge to one noisy annotator understates it. The real benchmark is agreement with truth, not with any single human.
{{svg:loop|Write a rubric, label a set, run the judge, compare, then fix or correct.}} {{svg:results|Pairwise accuracy under position and length bias, and the error of a raw and a corrected judge pass rate.}}