Part 9 of 16 · 2 min read
Planted defects
29 one-line defects, predicted and then tested, show which mistakes the suite would actually catch.
The suite has 187 tests. Before the first run, it was written down whether each of 29 one-line defects was expected to be caught or to escape, and then the suite was run against each.
25 of 29 defects were caught, 4 escaped, and all 29 predictions held -- a weaker claim than it sounds, since the same author wrote the tests, so the thin spots were already known.
Caught by the tests expected to catch them: wrong budgets (the wrapper not counted, the question not reserved, pinned cost not charged, 40 tokens of slack), dropping pinned items, wrong order, the oldest version of a fact winning, and a missing supersede step. Also caught: a window that skips instead of stopping, a packer that stops instead of skipping, summed instead of best-line relevance, flat word weights, trimming that keeps the first lines, and a lost marker. Also caught: the least relevant item put last, no stop words, zero-cost digits, and the first occurrence used for position. The last group caught: stale values never noticed, no position effect, numeric values that look like noise, items without a key thrown away, and keep-everything quietly losing items.
One catch is weaker than it looks: removing the escape for >> alone was caught only by a unit test that checks the escaped text directly. The behavioural test, which parses 200 random contexts, did not notice, because a lone >> cannot form a header on its own.
Four escaped, on purpose and by the prediction. The first is trimming that keeps one token fewer than allowed. The tests check the upper bound, not that the space is used. The second is ties in relevance breaking toward the oldest item. No test pins the tie rule, and this does change which items are chosen, so it is a real hole. The third is the parser accepting any role name. That is harmless today only because the escape stops all headers regardless. The fourth is the length penalty in the invented curve starting later than intended. That is a parameter of the curve, not a bug in the assembler.
What this says about the suite: the assembler's rules (budget, pinning, order, supersede, fencing) are well covered. The weakest point is selection's tie-breaking, and the first thing to do with the code is write the test that pins the tie rule and see which results move.