CSE Roadmap

Specialist tier · about 5 hours · 12 parts

Part 8 of 12 · 2 min read

Answering dataset-engineering questions in an interview

How would you deduplicate a large training set?

"First define duplicate: exact after normalisation, then near-duplicate by shingle Jaccard similarity above a threshold. Comparing all pairs is quadratic, so I use MinHash signatures with LSH banding to generate candidates, choose bands and rows so the S-curve threshold sits near my similarity target, and verify candidates with the real Jaccard. In the example's toy, 16 bands of 4 rows found 99.6 percent of true pairs using 499 comparisons instead of 3 million." Follow-up: "What about paraphrases?" They need embedding similarity or clustering, with more cost and more false matches.

How do you prevent and detect train/test contamination?

"Split by group so copies stay on one side, deduplicate across the train and eval sets, and run an n-gram overlap check: GPT-3's analysis used a 13-token match. In the example's toy a random row split inflated a memorising model from 0.48 to 0.65 and the n-gram check found every leaked row. I report the score on the clean subset." Follow-up: "What if you cannot see the training data?" Use a perplexity test (low perplexity on an eval item suggests it was seen) and fresh, private evaluation sets.

What are the risks of training on synthetic data?

"Quality, because garbage in gives garbage out; superficial imitation, because the student copies style without the ability; model collapse, where rare cases vanish over generations; and lost lineage, which hides copyright or benchmark leakage. In the example's toy a student of a teacher that never saw one class scored 0.15 to 0.25 on it versus 0.75 from 150 human labels. I verify synthetic data with tests or judges and mix in real data." Follow-up: "How much real data is enough?" The research I know of has no definitive ratio; measure the rare-case metrics on your own task.

How do you find mislabelled examples?

"Train on folds, predict out of fold, and flag rows where the given label gets low probability, then review them. In the example's toy, flagging below 0.1 caught 85 percent of planted label errors with 68 percent precision. I treat flags as review candidates, not deletions, because rare but correct examples also look wrong." Follow-up: "Why out of fold?" A model that has seen a row can memorise its wrong label.