CSE Roadmap

Core tier · about 6 hours · 13 parts

Part 10 of 13 · 2 min read

Mistakes that stall a debugging story

Common mistakes, and what the example does not do.

Mistakes that stall a debugging story

  • Reading prompts first before looking at the trace.
  • Changing several things at once.
  • Blaming the model without checking tool outputs and context.
  • Alerting on a rule without checking its false positives.
  • Fixing the one failing case and not testing on a held-out set.
  • Not storing the failing run, so it cannot be replayed.
  • Telling a bug story with no evidence, only the conclusion.
  • Treating a retry as harmless: it hides errors and adds tail latency.

What the example does not do

About this example What the example does not do
  • No real logs: the faults are injected, one per trace, with clean labels.
  • No overlapping causes: real failures often have two causes at once.
  • Circular scoring: the detector recall is about the example's own definitions.
  • Rules only: no model-based grading, so quality faults are invisible.
  • Simple localiser: it compares against one canonical healthy trace; real systems have many valid paths.
  • Optional experiment: take 20 real failing runs from your own agent, label each by hand, then see how many of your labels a trace rule could have found.

Check yourself

How do you answer "walk me through the hardest bug you found in an agent you built"?

Show answer

Six beats: symptom, how you reproduced it, how you localised it, the hypotheses you tested, the cause and fix, and what you changed so it cannot recur.

For the curious Rules catch structural faults, not fluent wrong answers

On the synthetic traces, seven simple rules reached recall near 1.00 on six fault classes, but 0 of 156 fluent-wrong answers, because every span looked healthy. That needs an evaluation, not a trace rule. Precision is where rules hurt: the first retry-storm rule had precision 0.48 because plan loops also repeat tool calls; requiring the repeated calls to be failures raised it to 1.00. The traces, faults and rules were all written for the example, so the recall figures are circular.

What you will use daily at work

Practice What it looks like
Start from the trace Spans, tool arguments and statuses.
Compare a failing run with a healthy one Report the first span that differs.
Turn each fixed failure into a test Keep the trace as a regression case.
Read your false positives They show which causes share a symptom.