CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 7 of 12 · 1 min read

How to build it so it holds up

  • Write the rubric with a scale, definitions and one example per level; ask for the score and a short reason; use a low temperature.
  • Prefer binary or small scales (pass or fail, 1 to 3), which are easier to calibrate than 1 to 10.
  • Calibrate on at least 100 human-labelled cases, balanced across outcomes and sliced by length and source.
  • Measure human-human agreement first, and treat it as the ceiling.
  • Use a different model family as judge where you can, to reduce self-bias, and strip model names from inputs.
  • Randomise or swap order in pairwise comparisons; control for length (instruct, or compare within length bands).
  • Use code where a check is exact (schema, arithmetic, state) and the judge only for fuzzy criteria.
  • Re-calibrate when the judge model, the rubric or the task distribution changes; pin the judge version.
  • Keep a human spot-check on a rolling sample.