Part 7 of 12 · 1 min read
How to build it so it holds up
- Define a duplicate (level, exactness, threshold) before writing code, and verify candidates with the real similarity.
- Split by group (source document, user, conversation, duplicate cluster), never by random row.
- Check contamination between training data and every evaluation set, and report clean-subset results.
- Normalise, then deduplicate, then filter, ordering steps from cheapest to most expensive.
- Find label errors with out-of-fold predictions and review flagged rows by hand.
- Scrub personal data with broad patterns and a second check, and keep a record of what was removed.
- Verify synthetic data with tests, schema checks, judges or human samples, and keep only what passes.
- Mix real and synthetic data; track the rare cases explicitly.
- Keep lineage: record the source, the generating model and the filters for every row.
- Measure every step on a held-out set with the model you will train.