Part 4 of 12 · 2 min read
A rerun changes the numbers, so a typed README goes stale within weeks
The moment you change the random seed or the sample size, last week's README is quietly lying about this week's system.
Think of a weather forecast printed on paper and pinned to the wall months ago. The paper doesn't update itself. Eventually it's describing a day that no longer exists.
In plain words: drift is a document's numbers falling out of step with the latest run of the code. An evaluation's exact numbers depend on things as incidental as the random seed or how many test items were used. So a README that isn't regenerated after every rerun will silently start making false claims, while one generated fresh from the new results file will always match.
A worked example. Re-running the fictional evaluation with a different random seed at the same sample size made 5 of the 9 claimed numbers in the old README wrong. Running it at 200 items instead of 150 made 3 of 9 wrong. Running it at 300 items with yet another seed made 5 of 9 wrong. In every case, regenerating the README from the new results file — rather than editing the old one by hand — left exactly 0 flags.
The key lines. From code/mini-portfolio/run_all.py:
# from code/mini-portfolio/run_all.py
for label, (n, seed) in {"other seed (n=150)": (150, 8), "more items (n=200)": (200, 7), "another seed and size (n=300, seed 9)": (300, 9)}.items():
new = E.main(n, seed, write=False)
stale = unverified_numbers(good, new)
regenerated = render((here / "templates/README.template.md").read_text(), new)
drift[label] = {"claim_numbers": len(claims), "stale_numbers": len(stale), "stale": stale, "flags_after_regeneration": len(lint(regenerated, new))}
Read it as: the same README text (good) is checked against three different reruns of the underlying evaluation. The point of the comparison isn't that any one rerun is "more correct." It's that the README's numbers are tied to one specific run. Any later run, however legitimate, makes the old text wrong until it's regenerated.
Drift
Drift is a document's numbers falling out of step with the latest run of the code.
Example: 5 of 9 numbers going stale from a seed change alone is drift happening without anyone touching the model or the data. It's a property of typing numbers into prose at all, not a sign that something went wrong with the evaluation.