Part 5 of 12 · 2 min read
A labelled sample can correct a judge's pass rate, but only once it's big enough
Knowing a judge's exact error rates lets you work backward from its reported score to the true one -- if you've measured those error rates precisely enough to trust the correction.
Think of a bathroom scale you know reads two pounds heavy. Once you know that exact offset, you can correct every future reading. But if you only checked the offset once, against a single uncertain reference weight, your "corrected" readings might be worse than just trusting the scale's raw number.
In plain words: prevalence correction (the Rogan-Gladen estimator) turns an observed pass rate into an estimate of the true rate, calculated as (observed rate minus false positive rate) divided by (true positive rate minus false positive rate) -- but this correction is only as good as the labelled sample used to measure the judge's own true- and false-positive rates in the first place, so a small calibration set can make the "corrected" number worse than just reporting the raw one.
A worked example. Against a true pass rate of 30%, a judge with a 90% true-positive and 20% false-positive rate reports a raw pass rate of 41% -- 11 points of error. Correcting that 41% using error rates measured from a labelled calibration set of just 20 cases gives a corrected estimate with 12 points of mean absolute error -- actually worse than the uncorrected raw number. With a 500-case calibration set, the corrected estimate's error drops to just 2 points. The formula itself is exactly right; the measured inputs to it are what need enough data to be trustworthy.
The key lines. From code/mini-judge/judge.py:
# from code/mini-judge/judge.py
def corrected_rate(observed, tpr, fpr):
"""Rogan-Gladen: correct an observed positive rate for the judge's error rates."""
d = tpr - fpr
if d <= 0:
raise ValueError("judge no better than chance")
return min(1.0, max(0.0, (observed - fpr) / d))
Read it as: the function takes tpr and fpr as given inputs. It has no way to know whether those two numbers were measured precisely from 500 cases or noisily from 20. The formula's correctness and the inputs' reliability are two separate questions. The 20-label result being worse than raw is a story about the second question. It is not a flaw in the Rogan-Gladen formula itself.
Prevalence correction (Rogan-Gladen)
An estimator that turns an observed pass rate into an estimate of the true rate: (observed minus false positive rate) divided by (true positive rate minus false positive rate).
Example: the corrected estimate going from worse-than-raw at 20 labels to a 2-point error at 500 labels is the correction formula staying fixed while its inputs become trustworthy -- a reminder that "I corrected for the judge's bias" is only as credible as the calibration set behind the correction.