Part 3 of 12 · 1 min read
The settings you pick change what counts as "the same"
Band and row counts for LSH aren't just a performance knob — they directly set the Jaccard similarity threshold where duplicate detection kicks in.
Think of a metal detector's sensitivity dial. Turn it up and it catches more buried objects, including some that are just bottle caps. Turn it down and it misses the faint signals along with the noise.
In plain words: the probability that a given pair becomes an LSH candidate rises sharply around a specific similarity value determined by the band and row counts — that threshold is a formula, not a guess, so the settings can be chosen deliberately for the similarity level you actually care about.
A worked example. For pairs in the 0.7-to-0.8 Jaccard range, the 8-bands-by-8-rows setting catches them as candidates only 49 percent of the time, while 16-bands-by-4-rows catches them 100 percent of the time — the theoretical predictions (57 percent and 99.6 percent) are close to what's actually observed. The formula's threshold sits at roughly (1/bands)^(1/rows): about 0.77 for 8x8 and about 0.5 for 16x4, which is exactly why 16x4 catches pairs in the 0.7-0.8 range far more reliably than 8x8 does.
The key lines.
# from code/mini-dataset/dedup.py
def candidate_probability(s, bands, rows):
"""Chance a pair with Jaccard similarity s becomes a candidate: 1 - (1 - s^rows)^bands."""
return 1 - (1 - s ** rows) ** bands
Read it as: this formula is computed from bands and rows alone, with no need to run the actual LSH process. Before touching real data, you can calculate exactly which similarity threshold a given band/row choice will catch. Then you can tune it to the duplicate definition you actually want.