CSE Roadmap

Core tier · about 5 hours · 6 parts

Part 4 of 6 · 4 min read

A test set that does not lie

Build a set of questions and labels that does not flatter your system.

Building a test set that does not lie

  1. Collect real queries from logs, support tickets or your own use, and keep their wording (typos included). Invented, tidy queries flatter every system. Use generated questions to expand coverage, not to replace real ones.
  2. Label with grades, not just yes or no ("answers it", "useful context"). You will need them for nDCG, and they force you to decide what "relevant" means.
  3. Include questions that have no answer, and decide what the right behaviour is (return nothing, or return passages that a later step will refuse). The example has two. For both, the keyword scores are 2.77 and 1.48, while real answerable queries score from 0 to 9.5: no score threshold separates them cleanly, which is why the gate in module 5 looks at the question's words, not just a retrieval score.
  4. Slice by kind of question (exact term, paraphrase, multi-part, follow-up), as module 3 did. A good average hides a weak group.
  5. Watch for pooling bias. If people labelled only the passages that one system returned, passages that other systems find are marked "not relevant" by default, and the labelled system looks best.
Recall@3 when labels cover keyword vector hybrid hybrid + rerank
every passage (complete) 0.86 0.77 0.86 0.89
only what keyword returned 1.00 0.82 0.95 0.97
only what vector returned 0.88 1.00 0.95 0.95

Judging only one system's results gives that system a perfect recall by construction. Label from the union of several systems' top results, plus some random passages as a control.

  1. Count information, not only passages. A passage-level recall of 0.50 can still carry the whole answer. In the example, for three queries, label recall is 0.50 but the retrieved text contains every required answer phrase (fact recall 1.00), because the missed passage was only "related context". Judge-based context recall, as in common RAG evaluation toolkits, works at this level: does the retrieved text support each claim in the reference answer?

Practical advice

  1. Measure retrieval before generation. If recall@k is low, fix retrieval first (modules 1 to 4).
  2. Fix a test set and a metric before you change things. Re-run after every change.
  3. Use the same k the model really sees.
  4. Always show an interval or at least the number of queries.
  5. Keep a failures list. For each query where relevant passages were missed, write down why (chunking, vocabulary, ranking). That list guides the next module's work.
  6. Combine with an end-to-end check. Good retrieval metrics do not guarantee good answers (modules 5 and 6; full pipelines in Level 9).
  7. Refresh the set as documents and users change.

Interview angle

"How do you evaluate the retriever in a RAG system?" Build a labelled set of real queries with graded relevance, include no-answer cases, and slice by question type. For a system that feeds the top k to a model, lead with recall@k and hit rate, and add MRR or nDCG to check ranking. Say why you do not use precision alone (it has a ceiling that depends on how many passages are relevant). Then show maturity: you report confidence intervals or a paired comparison because small sets are noisy, you watch for pooling bias in the labels, and you validate that a retrieval gain actually improves the final answer.

Check yourself

Hybrid beats keyword search by +0.03 MRR with an interval of −0.05 to +0.16. Can you say hybrid is better?

Show answer

No. The interval includes 0, so with 22 queries the gap may be luck. You need more queries, or a bigger gap.

For the curious Undefined cases now raise an error

recall_at_k and average_precision divided by zero for a query with no relevant passages, and precision_at_k did the same for k = 0. They now raise a ValueError that says what to do (leave unanswerable queries out of recall and MAP, and judge them separately by whether the system refuses). The bootstrap functions also refuse an empty list and unequal lists, because a paired comparison on different queries means nothing.

What you will use daily at work

Practice What it looks like
A labelled query set in the repository Query, relevant passage ids, and a grade, reviewed like code.
Reporting recall@k and MRR with an interval A table in the pull request, not a single number.
Evaluation libraries Frameworks provide these metrics; you still supply the labels.