Part 11 of 13 · 1 min read
Quick notes
- Answer the hardest-bug question in six beats: symptom, reproduction, localisation, hypotheses tested, cause and fix, and what you changed so it cannot recur.
- Agent failures are quiet. A tool succeeds but misleads, an answer sounds fluent but is wrong, a plan half works. Everything starts with a trace with spans, tool arguments and statuses.
- Localise before you theorise: compare the failing trace with a healthy run of the same task and find the first span that differs. A fluent wrong answer cannot be localised this way; it needs an evaluation.
- Rules catch structural faults, not quality faults, and precision is where they hurt. Read your false positives; they show which two causes share a symptom. A threshold is a trade-off between missed faults and wasted reviews.
- Change one thing, confirm on a held-out set, and turn the failure into a regression test. Read twenty failing traces before building tooling.