CSE Roadmap

Specialist tier · about 5 hours · 12 parts

Part 4 of 12 · 2 min read

A model that memorises exposes data defects a smooth model hides

The same dirty dataset can look almost harmless to one model and seriously damaging to another, depending on how much the model can memorise.

Think of two students taking the same messy, error-filled study guide into an exam. One studies the underlying concepts and barely notices the typos. The other memorises the guide word for word and reproduces its mistakes exactly.

In plain words: cleaning steps like deduplication and label filtering barely move a smooth model's score, but they can swing a memorising model's score substantially — which means you have to measure cleaning effects on the actual model you're training, not assume the effect generalises.

A worked example. Across the same cleaning pipeline (raw, then markup-stripped, then deduplicated, then label-filtered), Naive Bayes — a smooth model — moves by less than a point (0.892 to 0.884 across all four stages). The 1-nearest-neighbour model — a memoriser — barely moves from dedup alone (0.480 to 0.476) but jumps noticeably once the label filter is added (0.476 to 0.526). More strikingly, with a random row split (letting near-duplicate copies land on both sides), 1-nearest-neighbour scores 0.652 on held-out data; with a group split (keeping duplicate clusters together), it drops to 0.481 — the random split was silently measuring memorisation of near-copies, not real generalisation.

The key lines. From code/mini-dataset/models.py:

# from code/mini-dataset/models.py
class NN1:
    def fit(self, X, y): self.X = X / (np.linalg.norm(X, axis=1, keepdims=True) + 1e-9); self.y = y; return self
    def predict(self, X):
        Xn = X / (np.linalg.norm(X, axis=1, keepdims=True) + 1e-9); return self.y[(Xn @ self.X.T).argmax(1)]

Read it as: prediction is literally "find the single nearest stored training row and copy its label." If a near-duplicate of a test row sits in the training set, this model will find it and answer correctly. That happens regardless of whether it has learned anything general. It is exactly the leakage effect the random-split result shows.

Data contamination and leakage, label noise

Contamination (or leakage) means evaluation data, or copies of it, are present in the training data, inflating scores. Label noise means some training labels are simply wrong — detectable by training on part of the data and checking where the model disagrees with the given label on the rest.

Example: the 0.652-vs-0.481 split comparison above is contamination made visible — the random split let contaminated near-copies through, and the group split closed that gap.