CSE Roadmap

Common tier · about 4 hours · 11 parts

Part 4 of 11 · 2 min read

Verify every citation, don't trust the writer

Checking whether a cited page actually contains the claim catches what confidence alone cannot.

Think of fact-checking a news article, not by how confidently it is written, but by actually opening each footnoted source. You check that it says what the article claims it says.

In plain words: whether a statement sounds confident says nothing about whether its citation is real. The only reliable check is to mechanically confirm the cited source actually contains the claim being attributed to it.

A worked example. Simulating a careless writer that gets the citation or value wrong on 25 percent of statements, averaged over 30 runs: with no verification, the report contains 4.6 bad citations. Only 14.6 of 17 facts are correct. Adding a verifier that checks whether the cited page truly supports each claim brings bad citations to 0 and correctness to a full 17 of 17. It does this by regenerating the statement from evidence actually read whenever a citation doesn't hold up.

The key lines.

# from code/mini-research/research.py
def verify(statements, index=None):
    for s in statements:
        if s["status"] != "answered" or supports(s["cite"], s["need"], s["value"]):
            out.append(dict(s, verified=s["status"] == "answered")); continue
        alt = [d["id"] for d in DOCS if supports(d["id"], s["need"], s["value"])]
        if alt:
            out.append(dict(s, cite=alt[0], verified=True, repaired=True))
        else:
            out.append(dict(s, value=None, cite=None, status="unsupported", verified=False))

Read it as: every statement's citation is checked against the actual source text, not against how the writer phrased it. A wrong citation either gets repaired by finding a source that does support the claim. Otherwise the statement is downgraded to unsupported rather than left standing on a citation that doesn't hold up.

Citation precision

The share of cited statements whose cited source actually supports them — the measurable number behind "can this report be trusted."

Example: citation precision rising from roughly 73 percent (4.6 bad out of 17) with no verifier to 100 percent with one.