CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 10 of 12 · 1 min read

Quick notes

  • An LLM judge is a measuring instrument with error rates you must measure: calibrate against human labels with Cohen's kappa plus true and false positive rates.
  • Known biases: self-bias, verbosity bias and first-position bias in pairwise comparison.
  • Swap the order and keep only consistent verdicts to counter position bias, at the price of undecided pairs. Length bias hides when length happens to correlate with quality in the validation set.
  • Agreement is capped by human agreement, so do not demand judge-human agreement above human-human agreement, and kappa needs a few hundred labels to be stable.
  • A judge's raw pass rate is biased; a labelled set with measured error rates can correct it, but only once it is large enough.