Part 10 of 12 · 1 min read
Quick notes
- Data is the lever, judged on quality, coverage and quantity. A small set of carefully chosen examples can beat a large noisy one, and "high quality" means aligned with the task, not merely correct.
- Deduplicate before anything else: duplicates skew the distribution and leak test data into training. Normalised exact matching misses near duplicates; MinHash plus LSH banding catches them, with bands and rows chosen from the similarity you want and candidates verified.
- Find wrong labels with cross-validated predictions and review the flagged rows, and measure every cleaning step on the model you will train: a robust model hides data defects, a memoriser exposes them.
- Split by duplicate group and check contamination with n-grams, then report results on the clean subset; leakage inflates a memoriser most, which is why large models make contamination a real risk.
- Synthetic data needs verification, and a student inherits its teacher's blind spots. Mixing in real data stops model collapse in the toy, though there is no settled ratio.