Part 1 of 4 · 1 min read
Overview
Open-ended agent outputs (summaries, explanations, plans) cannot be checked by string match, so teams use a model as the grader. A judge that is wrong in a consistent direction silently steers every prompt change. The interview answer must name the biases and how you measured the judge.
About the example, and what is real
Real: kappa, true and false positive rates, the prevalence correction (the Rogan-Gladen estimator), the position-swap logic and the seeded simulations in code/mini-judge, with 11 tests. Assumed: the judge is a coin with a 90 percent true-positive and 15 or 20 percent false-positive rate; position and length biases are mixed in with a fixed probability; humans flip a label with 8 percent probability; pass-rate prevalence is 30 or 60 percent. No language model is called. The simulation shows the mechanics of calibration and bias, not how good any real judge is.