CSE Roadmap

Common tier · about 4 hours · 12 parts

Part 3 of 12 · 2 min read

Generate the numbers from a file; do not type them

A number typed into a document is a claim about the past; a number rendered from a results file is a claim that can be rechecked at any time.

Think of a thermostat display that shows the room's actual current temperature, versus a sticky note someone wrote down this morning. Both look like "72 degrees." But only one of them is still true an hour later.

In plain words: the provenance of a number is the file and command that produced it. A README template with placeholders like {{agent.accuracy|pct}} filled in from results.json guarantees every number in the document traces back to a specific run. So a checker can flag any number in the text that doesn't match a value actually in that file.

A worked example. The generated example README states "System accepted: 81% (75% to 87%)" — a number pulled directly from the evaluation's own bootstrap confidence interval, not typed by hand. A reviewer later changes one digit by hand ("81%" edited to "85%") to simulate a stale claim. The unverified-number check flags it immediately, because 85 no longer matches anything in results.json.

The key lines. From code/mini-portfolio/render.py:

# from code/mini-portfolio/render.py
def render(template, res):
    def sub(m):
        path, fmt = m.group(1).strip(), m.group(2).strip()
        if fmt not in FORMATS:
            raise ValueError(f"unknown format {fmt!r}; choose from {sorted(FORMATS)}")
        return FORMATS[fmt](lookup(res, path))
    return re.sub(r"\{\{\s*([\w.]+)\s*\|\s*(\w+)\s*\}\}", sub, template)

Read it as: lookup walks path through the results dictionary and raises immediately if any piece is missing. There is no silent fallback to a blank or a guessed value. A placeholder that doesn't resolve to something in the results file fails the render outright, which is what forces every number in the final README to have come from somewhere real.

Provenance of a number

Provenance is the file and command that produced a number; a number without it cannot be checked.

Example: "81% (75% to 87%)" has provenance the moment you can point to results.json's agent.accuracy field and the exact eval.py run that wrote it. A number with no such pointer is just an assertion.