Part 2 of 8 · 1 min read
Keyword search
How BM25 finds exact words, and what it misses.
Think of asking a librarian for books that contain the exact words you give. Fast and precise, but if you say "car" it will never offer the book that only says "automobile".
BM25
A scoring formula for term-based search. A passage scores higher when it contains the query's words, when those words are rare across the collection (a rare word such as e-4021 is strong evidence, a common word such as plan is weak), and when a word repeats but with diminishing returns. It also adjusts for passage length so long passages do not win just by being long.
For each query word, BM25 adds idf × f(term frequency, length), where idf (inverse document frequency) is large for rare words. The example implements it in about twenty lines (search.py). A test checks that e-4021 has a higher idf than plan.
BM25(DOCS).rank("E-5503")[0] # 'd02', the passage about that error
BM25(DOCS).rank("my money back for the yearly subscription") # does not contain 'd06' (the refund passage)
Good at: exact identifiers, names, error codes, function names, quoted phrases, anything where the exact word is the signal. It is fast, needs no model, and is easy to debug (you can see which words matched). Weak at: paraphrase and synonyms ("money back" versus "refund"), and it returns nothing for words it has never seen.