CSE Roadmap

Specialist tier · about 5 hours · 12 parts

Part 6 of 12 · 4 min read

What the example found

Dedup recall, the LSH threshold curve, label-flag precision/recall, cleaning effects by model, split effects, distillation, and collapse over generations.

How to read the dedup-method table: all rows face the same 501 true duplicate pairs; "candidates" is how many pairs actually had to be checked.

Near-duplicate method (501 true pairs) Candidates Recall Precision after verification
All pairs 3,073,960 100% 100%
Exact after normalising n/a 62.9% 100%
LSH 8 bands x 8 rows 418 83.4% 100%
LSH 16 bands x 4 rows 499 99.6% 100%
LSH 32 bands x 2 rows 7,707 100% 100%

What to notice: precision is 100 percent in every LSH row because candidates are always verified against the true Jaccard similarity afterward. LSH's only job is finding candidates cheaply. It does not decide the final answer.

How to read the Jaccard-threshold table: each row is a different similarity range; the two theory columns come from the formula in Part 2, and the observed rate should sit close to its matching theory number.

Jaccard of pair Pairs Candidate rate 8x8 (theory) Candidate rate 16x4 (theory)
0.6 to 0.7 26 23% (28%) 92% (97%)
0.7 to 0.8 115 49% (57%) 100% (99.6%)
0.8 to 0.9 40 90% (87%) 100% (100%)
0.9 to 1.0 320 100% (100%) 100% (100%)

What to notice: 16x4 reaches 100 percent candidate rate a full band earlier (0.8-0.9) than 8x8 does. This is the S-curve threshold difference from Part 2 showing up directly in measured data, not just in the formula.

How to read the label-flag table: each row is a stricter flagging threshold; precision and recall move in opposite directions as more rows get flagged.

Label flag threshold (210 wrong labels) Flagged Precision Recall
below 0.05 218 74% 77%
below 0.1 263 68% 85%
below 0.2 353 53% 90%
below 0.5 568 35% 95%

What to notice: doubling the recall from 77 to 95 percent (loosening the threshold) costs more than halving the precision (74 to 35 percent). Flagged rows are leads for human review, not a list to delete automatically. That is exactly why precision this low is still usable.

How to read the cleaning-by-model table: each row is one pipeline stage; the two right-hand columns face the identical data at that stage, differing only by which classifier is scored.

Pipeline step (clean external test) Rows Naive Bayes 1-nearest-neighbour
Raw, no cleaning (full pool) 2,480 0.892 0.480
Markup stripped 2,480 0.894 0.480
Plus near-dedup 2,001 0.888 0.476
Plus label filter (below 0.2) 1,707 0.884 0.526
Oracle: true labels, deduped 2,001 0.888 0.504
First 300 rows: raw 300 0.824 0.426
First 300 rows: plus dedup and label filter 247 0.842 0.458

What to notice: Naive Bayes's score barely changes across the entire table (0.884-0.894), while 1-nearest-neighbour's changes by 10 points. The same cleaning pipeline reads as "did almost nothing" or "mattered a lot," purely depending on which model is measuring it.

How to read the split table: both rows face the same data; only how the train/test split handles duplicate clusters differs.

Split (mean of 10) Naive Bayes held-out 1-NN held-out 1-NN on external clean test
Random row split 0.870 0.652 0.494
Group split 0.856 0.481 0.507

What to notice: 1-NN's held-out score under a random split (0.652) is far above its true external-test score (0.494). The random split was measuring the model copying near-duplicate answers, not generalisation. The group split's held-out score (0.481) is a much more honest estimate.

How to read the distillation table: rows mix two kinds of comparison — human labels vs. teachers, and teachers vs. their students; the "rare intent 4" column is the one the blind teacher never saw.

Distillation (clean test) Overall Rare intent 4
150 human labels 0.784 0.745
Good teacher 0.866 0.847
Student of good teacher, hard / soft 0.884 / 0.888 0.867 / 0.878
Blind teacher 0.728 0.082
Student of blind teacher, hard / soft 0.742 / 0.762 0.153 / 0.245
Blind-teacher student plus 150 human labels 0.764 0.255

What to notice: the blind teacher's overall score (0.728) looks only modestly worse than the good teacher's (0.866). But its rare-intent score (0.082) reveals the real problem. A blind spot can hide inside a reasonable-looking overall number until you check the specific class it affects.

How to read the collapse table: each row is a different share of real data mixed in each generation; read across to see how fast the type count degrades without that anchor.

Real data per generation Types alive after 1 after 10 after 30
0% 38.9 16.6 8.7
10% 38.9 27.5 27.2
30% 38.9 34.5 34.6
50% 38.9 37.3 37.0

What to notice: even a modest 10 percent real-data anchor each generation roughly stabilises the type count by generation 10 (27.5 to 27.2). With 0 percent, the count keeps declining all the way to generation 30. A little real data each round seems to be enough to stop the slide. But this toy gives no evidence for exactly how much is enough on a real task.

{{svg:pipeline|A dataset pipeline and how LSH cuts near-duplicate comparisons.}} {{svg:effects|Leakage, imitation and collapse effects in the toy.}}