What is in the top k, and where the right ones sit.
What you need to measure retrieval
Retrieval evaluation set
Three things together: a collection of passages, a set of queries, and relevance judgments that say which passages are relevant to each query.
If any of the three is weak, every number computed from it is weak. The metrics are small formulas; the hard part is the set. Section 6 returns to it.
Set-based metrics: what is in the top k?
For one query, let ranked be the list your system returned and rel the set of relevant passages. Look at the first k results.
Metric
Question it answers
Formula
Precision@k
Of the k shown, what share is relevant?
relevant in top k ÷ k
Recall@k
Of all relevant, what share did the top k include?
relevant in top k ÷ number of relevant
Hit rate@k
Did the top k contain at least one relevant passage?
1 or 0
precision_at_k(["a","x","b","y"],{"a","b","c"},3)# 2/3: a and b are in the first 3recall_at_k(["a","x","b","y"],{"a","b","c"},3)# 2/3: found a and b, missed c
Two things to notice. Precision has a ceiling: if a query has two relevant passages and k = 3, precision@3 can never exceed 2/3, so a "low" precision may be as good as it gets (a test checks that the best precision@5 on this set is 0.4). And precision and recall pull against each other as k grows:
k
1
2
3
4
5
8
Precision@k (hybrid)
0.82
0.57
0.41
0.32
0.29
0.18
Recall@k (hybrid)
0.66
0.82
0.86
0.89
0.98
0.98
Choose k from how the results are used: if the model reads 3 passages, recall@3 is the number that matters; if a person scans a list of 10, precision@10 and ranking matter more.
Rank-aware metrics: where are the right ones?
Set-based metrics ignore order inside the top k. A model reading a prompt, or a person reading a list, cares whether the best passage is first.
Reciprocal rank (RR) and MRR
1 / rank of the first relevant passage (0 if none appears). MRR is its average over queries. First place scores 1, second 0.5, third 0.33. Use it when one good passage is enough.
Average precision (AP) and MAP
Walk down the ranking; at each position where a relevant passage appears, note the precision so far; average those values over all relevant passages (so missing ones count as zero). MAP is its average over queries. It rewards putting all relevant passages early.
nDCG (normalised discounted cumulative gain)
Give each result a gain from its grade (here 2^grade − 1), divide by log2(position + 1) so later positions count less, add up to k, and divide by the best total any ordering could reach. 1.0 is a perfect ordering. It is the one metric that uses graded labels.
Here is one query with the same two relevant passages in both lists. Only the order differs.
One query, two result lists, five metric values
The keyword list puts a relevant passage first and another third; the vector list puts them second and fourth. Hit rate would call them equal, yet RR drops from 1.00 to 0.50 and nDCG@3 from 0.92 to 0.39. Which metric you choose decides which behaviour you reward.
Which metric when
If your system...
Look at
passes the top few passages to a model
recall@k for that k, and hit rate
needs one good passage (a single fact)
MRR
should surface every relevant passage, early
MAP
has labels like "answers it" versus "related"
nDCG
shows a list to people
precision@k, nDCG
Report two or three, not all. And always say the k.