Part 3 of 5 · 2 min read
Candidates and practical advice
How many to fetch, how many to keep.
How many candidates, and how many to keep?
| Knob | Too small | Too large |
|---|---|---|
| Candidates per retriever (for example 20) | the right passage is never found, and no later stage can recover it | slower fusion, more to rerank |
| Shortlist for the reranker (for example 20 to 50) | the reranker cannot help | latency and cost rise linearly |
| Passages sent to the model (for example 3 to 8) | the answer is missing | noise and cost; important text may be lost in the middle |
Pick by measuring recall of the first stage at N (is the right passage inside the shortlist?) and precision after reranking (is it first?). Hard exercise 3 builds that sweep.
Practical advice
- Start with hybrid. It is a small step from vector-only and protects you against identifiers.
- Tokenise carefully for keyword search.
E-4021must stay one token; the example's tokenizer keeps it whole, and a test checks it. - Evaluate by kind of question. An average hides that one retriever wins one group and loses another.
- Rerank only if you measure a gain. It adds latency and a dependency.
- Filter by metadata in the database (plan, date, language) when you can, but remember that filtering after an approximate search can leave fewer than k results (module 2).
Interview angle
"Why hybrid search?" Keyword search is exact on rare terms and identifiers but blind to paraphrase; vector search handles paraphrase but can blur identifiers and always returns something. Merging both (RRF is simple and needs no score calibration) covers both failure modes. Follow with: "what does the reranker add?" A cross-encoder reads query and passage together, so it is more precise, but it is costly, so you apply it to a shortlist, and it cannot recover passages the first stage missed. Say that you would evaluate per question type and check that the added latency is worth the gain.
Check yourself
A query contains the error code E-4021. Which retriever do you expect to find the right passage, and why might the hybrid still not rank it first? (Keyword search keeps E-4021 as one token. The vector side may rank a wrong passage highly, and RRF rewards agreement. A reranker or a keyword boost can fix this.)
For the curious Edge cases fixed in the example code
Building a keyword index on an empty collection divided by zero, and merging two empty score lists crashed on min() of nothing. Both now return empty results. Scoring a document also looked up its position with a list search for each document; it now uses the loop position.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Hybrid search in the database | Many search engines and vector stores offer keyword and vector search with built-in fusion. |
| A reranking service | A hosted or self-run cross-encoder reorders the top 20 to 50. |
| Evaluating by question type | Report exact-term, paraphrase and mixed questions separately. |
Draft: confirm which hybrid and reranking options your search stack offers.