CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 5 of 12 · 2 min read

Near-duplicates inflate a count, and leaked items test memory instead of ability

Counting twenty near-identical questions as twenty distinct test cases makes your suite look bigger and more thorough than it actually is.

Think of a survey that unknowingly sends the same question to the same person five times under slightly different wording. The tally would show five opinions where there was really only one.

In plain words: a near-duplicate is two items that differ only trivially, Jaccard similarity measures how much overlap two sets of word-trigrams share divided by their combined size, and leakage is an eval item appearing in the prompt, the few-shot examples or the training data -- deduplicating removes the first problem (an inflated, overweighted count) and a leak check catches the second (a score that measures memorized recall rather than genuine ability).

A worked example. A test batch of 43 items -- 20 templated order questions, 20 case-and-punctuation variants of those same 20, and 3 genuinely distinct items -- collapses to 23 items after deduplication at a similarity threshold of 0.5, and still 23 at a stricter 0.9 threshold. The 20 questions that differ only in an order number inside the template are not caught by this check at all, because swapping a number changes the trigrams enough to look like a different question -- a known limitation the module flags explicitly as needing normalization to fix. Separately, a leak check correctly flags one eval item that had been copied, nearly verbatim, into a few-shot example.

The key lines. From code/mini-traces/traces.py:

# from code/mini-traces/traces.py
def shingles(text, n=3):
    w = re.findall(r"[a-z0-9]+", text.lower())
    return {" ".join(w[i:i + n]) for i in range(max(1, len(w) - n + 1))}


def jaccard(a, b):
    sa, sb = shingles(a), shingles(b)
    return len(sa & sb) / len(sa | sb) if sa | sb else 1.0

Read it as: shingles breaks text into overlapping three-word chunks, and jaccard compares two texts by how many of those chunks they share. This is a purely lexical, word-level comparison with no understanding of meaning. That is why two questions differing only by an order number ("order 4471" vs "order 8832") still look different to it. The digit changes the chunk. The check has no concept that an order number is interchangeable.

Near-duplicate, Jaccard similarity, leakage

A near-duplicate is two items that differ only trivially. Jaccard similarity is the size of the overlap of two sets divided by the size of their union; here, of their word-trigram sets. Leakage is an eval item appearing in the prompt, the examples or the training data.

Example: 43 items shrinking to 23 is near-duplicates caught; the 20 order-number variants staying uncaught is Jaccard similarity's blind spot stated as a concrete count -- both facts belong in the same report, because a deduplication method that works is still a deduplication method with a known limit.