CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 8 of 12 · 2 min read

Answering judge-calibration questions in an interview

What are the failure modes of LLM-as-judge?

"Self-bias, favouring its own model's answers; verbosity bias, favouring long answers; position bias in pairwise comparisons; plus rubric ambiguity and drift when the judge model changes. I mitigate by swapping order and keeping consistent verdicts, controlling length, using a different model family, and calibrating against human labels with kappa and error rates." Follow-up: "How many labels do you need?" At least about 100: in the example's simulation the measured kappa ranged from 0.44 to 1.0 at 20 labels and 0.61 to 0.85 at 100.

How do you know a judge is good enough?

"I compare it with human labels on a held-out set, report kappa and the true and false positive rates, slice by length and source, and compare with human-human agreement, which is the ceiling. In the example's example the judge agreed with a noisy human at 0.60 and with the truth at 0.73, so a low judge-human kappa did not mean a bad judge." Follow-up: "What kappa is acceptable?" It depends on the decision; I tie it to how much a wrong gate costs, and I would rather report the error rates than a single number.

Your judge says 41 percent pass. Is that the quality of the agent?

"Not necessarily. If the judge has a false positive rate of 20 percent and a true positive rate of 90 percent, the true rate is (0.41 minus 0.2) over 0.7, which is 0.30. I can estimate those rates from a labelled set and correct, but the correction is only as good as the labelled set: with 20 labels it was no better than the raw rate." Follow-up: "Can you use it to compare two prompts?" Yes if the bias is stable, which I check by calibrating each version or on a shared labelled set.

Pairwise or pointwise judging?

"Pairwise is easier for a model and humans to be consistent on, but it has position bias and scales as pairs; pointwise scales linearly and gives absolute scores but is harder to calibrate. I use pairwise to compare versions, with swapping, and pointwise with a binary rubric for gating." Follow-up: "What about ties?" Count them as no preference and report the share; a high tie rate after swapping means the judge cannot tell the difference.