Part 9 of 12 · 2 min read
Mistakes that undercut a write-up
Common mistakes, and what the example does not do.
Mistakes that undercut a write-up
- Typing numbers into the README by hand.
- No baseline, so the number means nothing.
- No interval or sample size.
- Superlatives ("state-of-the-art", "production-ready") without evidence.
- A quickstart that has never been run on a clean machine.
- A demo that shows only the best case.
- Limits hidden or missing.
- A giant README that should be three linked documents.
What the example does not do
About this example What the example does not do
- No real project: TicketDraft is fictional and its evaluation is simulated.
- Crude text rules: the linter cannot judge clarity, honesty or whether the baseline is fair; it finds missing pieces.
- Circular validation: the planted-fault test uses faults and rules written by the same author.
- The number check matches claims to values by format and rounding; it misses numbers written as words and can flag harmless numbers.
- No video, diagram or screenshot generation; the demo is a script, not a recording.
- No reviewer data: the suggested reading order is a guess, not a study of how reviewers read.
- Optional real experiment: give your README to three people who have not seen the project, ask each to state what it does, the headline result and one limitation after 60 seconds, and fix what they miss.
Check yourself
Why generate the README numbers from results.json instead of typing them?
Show answer
A README that quotes last week's run is a false statement. When the fictional evaluation was re-run with another seed, 5, 3 and 5 of the 9 numbers in the old README were wrong. Regenerating left 0 flags.
For the curious A linter finds missing things, not good ones
The linter checks for missing sections, overclaiming words, a long quickstart, numbers without a baseline or interval, and numbers that match nothing in results.json. It caught 15 of 15 faults planted in it. But that is circular, because the same author wrote both the faults and the rules. The example project and its results are synthetic, and no real reviewer was studied.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Generate numbers from a results file | A template plus a short script. |
| State baseline, interval and n for each number | Or the number of runs. |
| Say what the agent cannot do | Put limits in their own section. |
| Keep the README short | About 200 words, detail in linked documents. |