Part 3 of 12 · 2 min read
Cohen's kappa separates real agreement from agreement by luck
Two graders who agree 90% of the time on a test where 90% of answers are correct anyway haven't shown any real skill at distinguishing good from bad.
Think of two weather forecasters who both predict "sunny" every single day in a desert. They will agree with each other constantly. But that agreement says nothing about their forecasting skill, since the weather itself rarely varies.
In plain words: Cohen's kappa is observed agreement corrected for the agreement you'd expect purely by chance, with 1 meaning perfect agreement and 0 meaning no better than chance -- the true positive rate is the share of truly good items the judge passes, and the false positive rate is the share of truly bad items the judge passes, and reporting kappa alongside these two rates tells you not just how often the judge agrees with truth, but how reliably it's actually distinguishing good from bad rather than just guessing in a way that happens to look right.
A worked example. For a judge with a true kappa around 0.75 (true-positive rate 0.9, false-positive rate 0.15), measuring kappa from just 20 labelled cases gives wildly unstable results: the 5th-to-95th percentile range spans from 0.44 to a perfect 1.00. With 300 labelled cases, that range tightens to 0.68 to 0.81 -- a measurement that's actually trustworthy. The judge's kappa against ground truth (0.73) was also higher than its kappa against a single noisy human annotator (0.60), even though the judge's true accuracy never changed -- only who it was being compared against did.
The key lines. From code/mini-judge/judge.py:
# from code/mini-judge/judge.py
def kappa(a, b):
"""Cohen's kappa for two binary label lists."""
n = len(a)
po = sum(x == y for x, y in zip(a, b)) / n
pa, pb = sum(a) / n, sum(b) / n
pe = pa * pb + (1 - pa) * (1 - pb)
return 1.0 if pe == 1 else (po - pe) / (1 - pe)
Read it as: po is the raw observed agreement rate. pe is the agreement rate you would expect if both labellers were just guessing according to their own overall pass rates. Subtracting pe and rescaling turns "90% agreement" into a fairer number. It removes the part of that 90% that was guaranteed because both labellers mostly agree on how common "pass" is in the first place.
Cohen's kappa, true positive rate, false positive rate
Cohen's kappa is observed agreement corrected for the agreement expected by chance; 1 is perfect, 0 is chance. True positive rate is the share of truly good items the judge passes. False positive rate is the share of truly bad items the judge passes.
Example: a measured kappa of 0.44 at 20 labels for a judge whose real kappa is 0.75 is a warning about small calibration sets specifically -- the true-positive and false-positive rates underneath the judge never changed, only how precisely 20 cases could pin down a single summary number.