CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 9 of 12 · 2 min read

Mistakes that make a judge untrustworthy

Common mistakes, and what the example does not do.

Mistakes that make a judge untrustworthy

  • No human labels at all, so the judge is unvalidated.
  • A calibration set of 20 cases.
  • Calibrating on easy cases, or cases where length tracks quality.
  • Judging with the same model that produced the answers, without checking self-bias.
  • A rubric with undefined terms ("good", "helpful").
  • A 1 to 10 scale with no anchors.
  • Changing the judge model silently, so trends break.
  • Reporting the raw judge rate as the quality.
  • Using an LLM where exact code works.

What the example does not do

About this example What the example does not do
  • No language model: the judge is a parametric coin, so it cannot show rubric effects or reasoning errors.
  • Biases are independent additions; real biases interact with content.
  • Humans make independent random errors; real annotators disagree systematically on hard cases.
  • A fixed prevalence; the correction would shift with a different mix.
  • Binary labels only. Ordinal scales need weighted kappa.
  • Optional real experiment: label 100 real outputs with two humans, run a judge in both orders, and compute kappa, true and false positive rates, the swap-inconsistency rate, and the correlation between length and judge score.

Check yourself

Your judge agrees with a human at kappa 0.60 and the humans agree with each other at 0.67. Is the judge bad?

Show answer

Not necessarily. Human agreement is the ceiling, and the judge is close to it. In the simulation the judge agreed with the truth at 0.73.

For the curious A judge with a position bias needs swapped order, at a cost

A judge with a 30 percent position bias fell from 85 to 75 percent accuracy. Asking twice with the order swapped and keeping only consistent verdicts raised accuracy on decided pairs to 92 percent, but 42 percent of pairs were undecided. Also, kappa from 20 labels ranged from 0.44 to 1.0 for a judge whose true kappa was about 0.75, so use at least 100 labels. The judge here is a probability model, not an LLM.

What you will use daily at work

Practice What it looks like
Write a rubric with an example per level Prefer pass or fail, or a 1 to 3 scale.
Measure human agreement first It is the most you can ask of the judge.
Swap the order in pairwise tests Keep only verdicts that survive the swap.
Pin the judge version Re-calibrate when the judge, rubric or task mix changes.