CSE Roadmap

Core tier · about 4 hours · 11 parts

Part 3 of 11 · 2 min read

Numbers and negations slip straight past a word-overlap check

"The warranty lasts 24 months" and "The warranty lasts 36 months" share every word except one digit.

Think of proofreading a contract by skimming for familiar phrases. A changed dollar figure or a flipped "not" can sail past a skim that is really only checking that the surrounding words look right.

In plain words: a claim is one checkable statement extracted from an answer, entailment means the context implies the claim, and contradiction means the context says the opposite or a different value -- and because a changed number or a flipped negation barely changes a claim's words, a check that only compares vocabulary needs separate rules to catch either kind of contradiction.

A worked example. "The warranty lasts 36 months" overlaps almost completely with the true "24 months" claim's context sentence, so word-overlap alone judges it correctly only 0% of the time across the module's wrong-number claims -- it's indistinguishable from the true claim by vocabulary. Adding an explicit numbers check (the claim's numbers must be a subset of the context's numbers) raises that to 100%. Negation flips behave the same way: "Water damage is covered by the warranty" shares every word with "Water damage is not covered", so overlap alone catches only 33% of negation flips, and only a dedicated negation check reaches 100%.

The key lines. From code/mini-faithful/faithful.py:

# from code/mini-faithful/faithful.py
def has_negation(s):
    return any(w in NEG for w in re.findall(r"[a-z']+", s.lower()))


    if check_negation:
        # a negation in the claim that is absent from the best-matching context sentence (or the reverse) is suspicious
        best = max(sentences(context), key=lambda s: support_score(claim, s))
        if has_negation(claim) != has_negation(best):
            return False

Read it as: the negation check first finds the single context sentence with the highest overlap to the claim (best). Then it compares only whether the two disagree on negation. It does not try to understand meaning. It just asks whether "not" appears in the claim but not in its best-matching source sentence, or vice versa.

Claim, entailment, contradiction

A claim is one checkable statement extracted from an answer. Entailment is the context implying the claim. Contradiction is the context saying the opposite or a different value.

Example: "The warranty lasts 36 months" contradicts a context that entails "24 months" -- the words are nearly identical, so only a numbers-specific check, not plain overlap, catches the contradiction.