CSE Roadmap

Core tier · about 8 hours · 15 parts

Part 8 of 15 · 2 min read

Planted defects

28 one-line defects, predicted and then tested, show which mistakes the suite would actually catch.

The suite has 131 tests. As in module 1, it was written down before the first run whether each of 28 one-line defects was expected to be caught or to escape.

25 of 28 were caught, 3 escaped, and every one of the 28 predictions held. That is a weaker claim than it sounds. The same author wrote the tests, so the thin spots were already known.

Caught: forgetting that only hides the fact (a soft delete), forgetting that keeps the history, secure_delete switched off, copies surviving by default, procedure text surviving a scrub, quarantined rows surviving, an old value left current beside the new one, and history erased by an update. Also caught: unsorted facts and episodes, procedure versions that never increase, a tool treated as trusted, a gate that guards facts only, blocked writes dropped without a trace, and a phone pattern that needs a capital letter. The last group caught: an extractor that suddenly understands the paraphrase, summaries that lose their decisions or are cut to 20 characters, and a replay that takes the oldest turns or overshoots its budget. It also caught typed recall without the fact table, every procedure or every decision shown whatever the question, and a judge that ignores a stale value or a leak.

Escaped, and each one is a real hole, not a quirk. First, the write function's default gate becomes "open," and no test calls apply() without the gate argument explicitly. So the safe default itself is untested. Second, the assistant's own lines are allowed to write memory as if the user had said them. No test feeds the extractor an assistant line containing a pattern. Third, an old procedure version stays marked current beside the new one. It escapes only by an accident of how SQLite and Python's dictionary happen to order results today. A change in ordering would break it silently.

All three are closable in a few lines, and they are left open on purpose, as in module 1, so the shape of a suite that passes and still is not enough stays visible.