Part 2 of 12 · 2 min read
A judge needs a written rubric before it can be trusted at all
Asking a model "is this good?" with no criteria gets you an opinion, not a measurement.
Think of hiring a teaching assistant to grade essays. With no rubric at all, the marks depend on the assistant's mood and come out inconsistent. If you hand them a scale with a worked example for each grade level, the marks become something you can actually audit.
In plain words: LLM-as-judge means using a language model to score or compare outputs, a rubric is written criteria and a scale (ideally with an example for each score), and a judge can be reference-based (comparing an output to a known good answer) or reference-free (judging without one) -- and every simulation in this module treats the judge as a measuring instrument with its own error rates, which only makes sense once a rubric has fixed what "correct" means.
A worked example. The simulation models the judge as a probabilistic rule: given a true label, the judge reports "pass" correctly 90% of the time (its true-positive rate) but also reports "pass" on 15-20% of truly bad items (its false-positive rate). This is the simulation's stand-in for a rubric-following judge that gets most calls right but has a systematic, measurable error pattern -- not random noise, but a consistent bias you can characterize and even correct for, which is the whole point of treating the judge as an instrument rather than an oracle.
The key lines. From code/mini-judge/judge.py:
# from code/mini-judge/judge.py
def judge_labels(truth, tpr, fpr, rng):
return [(rng.random() < tpr) if t else (rng.random() < fpr) for t in truth]
Read it as: the judge's output depends on the true label t. It depends on it through two fixed probabilities. One is tpr, used when the item is truly good. The other is fpr, used when it is truly bad. This one line is the whole model of "a judge with known, stable error rates." Everything the module measures later (kappa, bias, correction) is built on this structure.
LLM-as-judge, rubric, reference-based and reference-free
LLM-as-judge is using a language model to score or compare outputs. A rubric is written criteria and a scale, ideally with an example for each score. Reference-based: the judge compares an output to a known good answer; reference-free: it judges without one.
Example: the judge's 90% true-positive, 15% false-positive rates only mean anything because they're measured against a defined "truly good" and "truly bad" -- without a rubric drawing that line, there would be no fixed truth for the judge's error rates to be measured against at all.