Part 6 of 13 · 3 min read
What the example found
Per-fault precision and recall, look-alike false-flag rates, the latency threshold sweep, and localisation accuracy by fault class.
How to read the per-fault table: each row is one of the eight injected fault classes across 6,000 synthetic traces; precision and recall are against the true fault labels.
| Fault class (6,000 traces) | True cases | Precision | Recall |
|---|---|---|---|
| Swallowed tool error | 169 | 1.00 | 1.00 |
| Retry storm (repeated failing calls) | 166 | 1.00 (v1: 0.48) | 1.00 |
| Plan loop | 177 | 1.00 | 1.00 |
| Empty retrieval, answer not hedged | 189 | 1.00 | 1.00 |
| Bad tool arguments | 189 | 1.00 | 1.00 |
| Latency spike (k = 4) | 165 | 0.63 | 0.99 |
| Context truncation | 191 | 0.75 | 1.00 |
| Fluent wrong answer | 156 | n/a | 0.00 |
What to notice: the six structural fault classes all reach near-perfect precision and recall. The two classes defined by continuous thresholds (latency, context) or by meaning rather than structure (fluent wrong) are exactly where the rules struggle. The pattern tracks whether the fault is a clean yes/no structural fact or a matter of degree and judgment.
How to read the look-alike table: each row is a healthy or benign trace pattern that was never meant to be flagged as faulty; the last column is how often some rule flagged it anyway.
| Look-alike that is not a fault | Traces | Flagged by some rule |
|---|---|---|
| Plain healthy | 3,416 | 51 (1.5%, all latency) |
| Transient error then a successful retry | 325 | 3 |
| Empty retrieval with a hedged answer | 287 | 1 |
| Two planning rounds | 284 | 4 |
| Prompt at 90 to 100% of the limit | 286 | 73 (26%) |
What to notice: near-limit prompts get flagged more than a quarter of the time by the context rule. That single row accounts for most of the false positives in this whole table. That is exactly why the per-fault table above shows context truncation's precision stuck at 0.75 rather than 1.00.
How to read the latency-threshold table: each row sets the latency-spike rule's threshold at a different multiple of the median duration.
| Latency rule threshold | Precision | Recall |
|---|---|---|
| 2 times median | 0.07 | 1.00 |
| 3 times | 0.27 | 1.00 |
| 4 times | 0.63 | 0.99 |
| 6 times | 0.94 | 0.94 |
| 8 times | 0.99 | 0.81 |
What to notice: there is no threshold that gives both high precision and high recall at once for this fault. Every row is a different point on the same trade-off curve. This is the table to point to when an interviewer asks how you would pick a threshold.
How to read the localisation table: each row is a fault class or group of classes; the share is how often the first-divergence check named the actually-injected fault span.
| Localisation (first divergence equals injected span) | Share |
|---|---|
| Swallowed error, plan loop, empty retrieval, bad arguments, truncation, retry storm | 0.96 to 0.98 |
| Latency spike | 0.81 (15% show no divergence) |
| Fluent wrong answer | 0.00 (92% show no divergence) |
What to notice: "no divergence found" for 92% of fluent-wrong-answer traces is the structural proof that this fault class needs a different tool entirely. Localisation is not failing at its job here. It is correctly reporting that nothing in the spans looks abnormal. The wrongness lives in the content of the answer, not its structure.
{{svg:loop|A swallowed-error trace and the six-step loop.}} {{svg:scores|Detector scores, threshold trade-off and localisation.}}