CSE Roadmap

Common tier · about 4 hours · 12 parts

Part 6 of 12 · 2 min read

What the example found

The three fictional READMEs, the planted faults, the drift under reruns, and the generated example's own numbers.

How to read the fictional-README table: each row is a README the module wrote on purpose, from deliberately bad to directly generated from results.

Fictional README Flags What was flagged
Weak 15 7 missing sections, 4 overclaim words, 8-command install, no baseline, no interval, a number not in results
Partial 4 missing cost and latency, missing limitations, two numbers not in results
Generated from results 0 none

What to notice: the gap between "partial" (4 flags) and "weak" (15) is mostly missing sections, which the linter catches reliably. The gap between "partial" and "generated" (0) is about numbers not tracing back to a results file, which is a narrower and more mechanical kind of check.

How to read the planted-fault table: each row is one deliberate change made to the generated README; "Detected" says whether the linter's flags changed as a result.

Planted fault (15 total) Detected
Remove any one of the eight sections 8 of 8
Add an overclaim word yes
Add a number not in results yes
Change one number (stale) yes
Make the quickstart 10 commands yes
Remove interval wording yes
Remove the baseline yes
Make the README very long yes

What to notice: the linter caught all 15 of 15 planted faults. But the module is explicit that this is circular evidence: the same author wrote both the faults and the detection rules. So a perfect score here says the code matches its own author's intent, not that the rules are the right ones.

How to read the drift table: each row is a rerun of the underlying evaluation with a different seed or sample size, compared against the README text generated from the original run.

Rerun of the evaluation Numbers now wrong (of 9) Flags after regenerating
Other seed, same size 5 0
200 items 3 0
300 items, other seed 5 0

What to notice: regenerating always brings the flag count back to 0, regardless of how many numbers went stale. The fix for drift isn't catching every stale number by eye. It's never typing numbers by hand in the first place.

How to read the generated-README table: this is the actual content of the README produced from one run of the fictional evaluation.

The generated example README (synthetic numbers) Value
Rules baseline accepted 61% (53% to 69%)
System accepted 81% (75% to 87%)
Gain 20 points (11 to 29)
Cost per ticket (assumed prices) $0.0092
Latency median / 95th percentile 3.2 s / 5.7 s
README length 203 words

What to notice: every figure in this table has a visible interval or an explicit unit next to it. There is no bare number anywhere. That is the entire point the README-generation approach is built to enforce.

{{svg:docs|The four documents and the README layout.}} {{svg:check|Linting a README and the drift check.}}