CSE Roadmap

Specialist tier · about 5 hours · 12 parts

Part 7 of 12 · 1 min read

How to build it so it holds up

  • Define a duplicate (level, exactness, threshold) before writing code, and verify candidates with the real similarity.
  • Split by group (source document, user, conversation, duplicate cluster), never by random row.
  • Check contamination between training data and every evaluation set, and report clean-subset results.
  • Normalise, then deduplicate, then filter, ordering steps from cheapest to most expensive.
  • Find label errors with out-of-fold predictions and review flagged rows by hand.
  • Scrub personal data with broad patterns and a second check, and keep a record of what was removed.
  • Verify synthetic data with tests, schema checks, judges or human samples, and keep only what passes.
  • Mix real and synthetic data; track the rare cases explicitly.
  • Keep lineage: record the source, the generating model and the filters for every row.
  • Measure every step on a held-out set with the model you will train.