Part 10 of 16 · 3 min read
Planted defects
Testing the tests: 33 one-line bugs, predicted caught or missed before the run.
The suite has 152 tests. They pin the token counter and the generated task's invariants (100 steps, 24 probes, one injection and it is in a tool turn). They pin each extraction pattern, and the budget after every step for each strategy at three budgets. They pin the behaviour of each strategy (FIFO order, stub contents, the recent turns that are never cleared, rolling feedback, caps). They pin the summariser (determinism, caps, oldest-first, source-agnosticism, drift), the gate for each role, the judge, and the headline claims of every experiment.
As in the earlier modules, it was written down before the first run whether each of 33 one-line defects was expected to be caught or to escape.
- 27 of 33 caught; 6 escaped. 5 of the 33 predictions were wrong (28 held). The tests and the predictions came from the same author, and five predictions were still wrong, which is the argument for running the mutants rather than trusting a prediction.
- Caught: the gate (tool treated as trusted, gate bypassed, refused writes not recorded), slot supersession (a revision not replacing the old value, a revision keyed by its value), the security pattern, constraint eviction, eviction order of slots, the slot cap, FIFO dropping the newest, the rolling summary not being fed back, free summariser calls, the round counter, the stub (never clearing, hiding its size), the hybrid missing either half, the question missing from the window, keep-all being reduced, digit-run counting, the summariser trimming the wrong end or not at all, the judge ignoring stale or forbidden values, refusal switched off.
- Predicted to escape, and did (3):
gate_default_open,fifo_strict_budget,drift_hits_noise. - Predicted to escape, but a test caught it (2):
rolling_input_not_charged(the charge test needs more than the output alone) androlling_one_turn_chunks(folding one turn per round makes far more rounds, which the budget-versus-rounds test sees). - Predicted caught, but escaped (3), each a real hole:
1.
rolling_cap_ignored: the summariser is told a cap five times larger. It escapes because the summariser keeps so few lines (3 percent of noise) that the summary never approaches either cap. The cap is never exercised by the task, only by the unit test of the summariser. 2.clear_includes_recent: the "keep the newest turns" guard is removed. It escapes because clearing runs oldest first and stops as soon as the request fits, so it never reaches the recent turns in this task. 3.clear_hits_user_turns: clearing is allowed to stub user and assistant turns. Same reason: it stops before it reaches any. Both guards are untested on a request that is far over budget.
The first three escapes (gate_default_open, fifo_strict_budget, drift_hits_noise) are real as well. The default gate has no test. An off-by-one in the budget loop is invisible unless the size lands exactly on the budget. Drift on noise lines is invisible because noise is almost never kept. All six are closable, and medium exercise 1 asks you to do it. They are left open on purpose, as in modules 1 and 2.