Part 7 of 12 · 1 min read
How to build it so it holds up
- Write the rubric with a scale, definitions and one example per level; ask for the score and a short reason; use a low temperature.
- Prefer binary or small scales (pass or fail, 1 to 3), which are easier to calibrate than 1 to 10.
- Calibrate on at least 100 human-labelled cases, balanced across outcomes and sliced by length and source.
- Measure human-human agreement first, and treat it as the ceiling.
- Use a different model family as judge where you can, to reduce self-bias, and strip model names from inputs.
- Randomise or swap order in pairwise comparisons; control for length (instruct, or compare within length bands).
- Use code where a check is exact (schema, arithmetic, state) and the judge only for fuzzy criteria.
- Re-calibrate when the judge model, the rubric or the task distribution changes; pin the judge version.
- Keep a human spot-check on a rolling sample.