CSE Roadmap

Core tier · about 6 hours · 13 parts

Part 6 of 13 · 3 min read

What the example found

Per-fault precision and recall, look-alike false-flag rates, the latency threshold sweep, and localisation accuracy by fault class.

How to read the per-fault table: each row is one of the eight injected fault classes across 6,000 synthetic traces; precision and recall are against the true fault labels.

Fault class (6,000 traces) True cases Precision Recall
Swallowed tool error 169 1.00 1.00
Retry storm (repeated failing calls) 166 1.00 (v1: 0.48) 1.00
Plan loop 177 1.00 1.00
Empty retrieval, answer not hedged 189 1.00 1.00
Bad tool arguments 189 1.00 1.00
Latency spike (k = 4) 165 0.63 0.99
Context truncation 191 0.75 1.00
Fluent wrong answer 156 n/a 0.00

What to notice: the six structural fault classes all reach near-perfect precision and recall. The two classes defined by continuous thresholds (latency, context) or by meaning rather than structure (fluent wrong) are exactly where the rules struggle. The pattern tracks whether the fault is a clean yes/no structural fact or a matter of degree and judgment.

How to read the look-alike table: each row is a healthy or benign trace pattern that was never meant to be flagged as faulty; the last column is how often some rule flagged it anyway.

Look-alike that is not a fault Traces Flagged by some rule
Plain healthy 3,416 51 (1.5%, all latency)
Transient error then a successful retry 325 3
Empty retrieval with a hedged answer 287 1
Two planning rounds 284 4
Prompt at 90 to 100% of the limit 286 73 (26%)

What to notice: near-limit prompts get flagged more than a quarter of the time by the context rule. That single row accounts for most of the false positives in this whole table. That is exactly why the per-fault table above shows context truncation's precision stuck at 0.75 rather than 1.00.

How to read the latency-threshold table: each row sets the latency-spike rule's threshold at a different multiple of the median duration.

Latency rule threshold Precision Recall
2 times median 0.07 1.00
3 times 0.27 1.00
4 times 0.63 0.99
6 times 0.94 0.94
8 times 0.99 0.81

What to notice: there is no threshold that gives both high precision and high recall at once for this fault. Every row is a different point on the same trade-off curve. This is the table to point to when an interviewer asks how you would pick a threshold.

How to read the localisation table: each row is a fault class or group of classes; the share is how often the first-divergence check named the actually-injected fault span.

Localisation (first divergence equals injected span) Share
Swallowed error, plan loop, empty retrieval, bad arguments, truncation, retry storm 0.96 to 0.98
Latency spike 0.81 (15% show no divergence)
Fluent wrong answer 0.00 (92% show no divergence)

What to notice: "no divergence found" for 92% of fluent-wrong-answer traces is the structural proof that this fault class needs a different tool entirely. Localisation is not failing at its job here. It is correctly reporting that nothing in the spans looks abnormal. The wrongness lives in the content of the answer, not its structure.

{{svg:loop|A swallowed-error trace and the six-step loop.}} {{svg:scores|Detector scores, threshold trade-off and localisation.}}