Trace: the record of one request. Span: one step in it (a guard, retrieval, planner call, tool call or answer) with its duration, status, arguments and sizes.
Root cause
Root cause: the change that, if removed, makes the failure disappear. A symptom (a wrong answer) can have several, and two causes can share a symptom.
Localisation
Localisation: finding the first step where a failing run departs from a healthy one.
Regression case
Regression case: a stored failing input (or trace) that must pass after the fix and in all later versions.
Swallowed error
Swallowed error: a tool failure that the agent continues past as if it had succeeded.
Precision and recall of a detector
Precision: of the traces it flags, the share that truly have the fault. Recall: of the traces that have the fault, the share it flags.
The debugging loop
A swallowed-error trace and the six-step loop.
Reproduce. Store the input, retrieved documents, tool outputs, prompts and model version. Nondeterminism means one rerun proves little; repeat several times or replay stored tool results.
Localise. Compare with a healthy run of the same task and find the first difference.
Hypothesise. Write down at least three causes and what each predicts in the trace.
Test one change. Change one variable, rerun on the failing case and on a held-out set.
Fix and guard. Fix the cause; add a check earlier in the pipeline.
Swallowed error, plan loop, empty retrieval, bad arguments, truncation, retry storm
0.96 to 0.98
Latency spike
0.81 (15% show no divergence)
Fluent wrong answer
0.00 (92% show no divergence)
Detector scores, threshold trade-off and localisation.
A rule is a hypothesis about a symptom; check its false positives.
Shared symptoms hide causes: a retry storm and a plan loop both repeat tool calls.
Precision depends on how common the look-alikes are: the context rule's false flags came mostly from near-limit prompts that were fine.
Localisation is not diagnosis: it names the first departure, and my first label for the retry storm pointed to the second call until I noticed the first failing call was the real departure.
Across eight corpora of 2,000 traces, latency precision ranged 0.59 to 0.75 and context precision 0.71 to 0.80, so quote precision with its spread.
Design rules
Log spans with arguments, statuses, durations, token counts and versions from day one.
Return structured errors from tools and never let a loop treat an error as an observation silently.
Compare with a healthy run before reading prompts.
Write rival hypotheses and what each predicts, then choose the cheapest test.
Change one thing at a time.
Read the false positives of every alert.
Use rules for structure and evaluations for quality; a fluent wrong answer needs a grader or a human.
Put every fixed failure into a regression set.
Bound loops and retries and alert on the bound being hit.
Interview questions, with the follow-ups
Walk me through the hardest bug you found in an agent you built.
Use the six beats. Example (fictional, built from the example trace): "Customers were told a refund was issued when it was not. I pulled traces for the complaints: the refund tool span had status error, there was no later tool call and the answer did not hedge. Compared with a healthy refund trace, the first difference was that span. Hypotheses: tool outage, bad arguments, or the loop ignoring errors. Arguments were valid and the tool was up, so the loop treated an error string as an observation. Fix: tools return structured errors, the loop stops or hands off on an error, and the answer must say so. I added the trace as a regression test and an alert on a tool error followed by a non-hedged answer." Follow-up: "How did you know it was the cause?" Changing only the error handling made the failing cases pass and the held-out set did not regress.
An agent gives a confident wrong answer and every span looks fine. What now?
"Trace rules will not see this. I would sample the failing cases, label the error type (stale data, misread tool result, hallucination), build an evaluation set from them, and use a grader plus human spot checks. Fixes could be grounding, citations, a verification step or a handoff." Follow-up: "How do you know the fix worked?" A measured improvement with an interval on a held-out set.
Your alert for slow requests fires constantly. What do you do?
"Look at the false positives. In my example a threshold of 2 times the median had 7 percent precision; at 6 times it was 94 percent with 94 percent recall. I would pick the threshold by the cost of missing a spike against the cost of a review, and alert on a rate over a window rather than single spans." Follow-up: "What else could cause slow spans?" Provider latency, large context, retries, a cold cache.
How do you stop the same bug coming back?
"Add the failing trace to a regression corpus, add a guard that catches the pattern earlier, and monitor the rate. Keep good traces as golden paths too." Follow-up: "What if the trace contains personal data?" Redact or synthesise before storing it.
Common mistakes
Reading prompts first before looking at the trace.
Changing several things at once.
Blaming the model without checking tool outputs and context.
Alerting on a rule without checking its false positives.
Fixing the one failing case and not testing on a held-out set.
Not storing the failing run, so it cannot be replayed.
Telling a bug story with no evidence, only the conclusion.
Treating a retry as harmless: it hides errors and adds tail latency.
What the example does not do
No real logs: the faults are injected, one per trace, with clean labels.
No overlapping causes: real failures often have two causes at once.
Circular scoring: the detector recall is about my own definitions.
Rules only: no model-based grading, so quality faults are invisible.
Simple localiser: it compares against one canonical healthy trace; real systems have many valid paths.
Optional experiment (I did not run this): take 20 real failing runs from your own agent, label each by hand, then see how many of your labels a trace rule could have found.
Check yourself
How do you answer "walk me through the hardest bug you found in an agent you built"? (Six beats: symptom, how you reproduced it, how you localised it, the hypotheses you tested, the cause and fix, and what you changed so it cannot recur.)
For the curious Rules catch structural faults, not fluent wrong answers
On my synthetic traces, seven simple rules reached recall near 1.00 on six fault classes, but 0 of 156 fluent-wrong answers, because every span looked healthy. That needs an evaluation, not a trace rule. Precision is where rules hurt: my first retry-storm rule had precision 0.48 because plan loops also repeat tool calls; requiring the repeated calls to be failures raised it to 1.00. The traces, faults and rules were all written by me, so the recall figures are circular.
What you will use daily at work
Practice
What it looks like
Start from the trace
Spans, tool arguments and statuses.
Compare a failing run with a healthy one
Report the first span that differs.
Turn each fixed failure into a test
Keep the trace as a regression case.
Read your false positives
They show which causes share a symptom.
Draft: confirm which tracing tool your team uses and what it records, so your debugging story names real spans and logs.
What you must be able to say (one page)
The interview question is "walk me through the hardest bug you found in an agent you built." Answer in six beats: symptom, how you reproduced it, how you localised it, the hypotheses you tested, the cause and fix, and what you changed so it cannot recur. The six-step loop is my own summary.
Agent failures are quiet. A tool succeeds but cascades errors, an LLM output sounds fluent yet misleads, or a plan partially works but misses the goal. These rarely crash the system, so production observability is not optional. The uneven success pattern is sometimes called a jagged frontier: similar-looking tasks succeed or fail unpredictably.
Everything starts with a trace. With structured events you can tell whether a failure was a reasoning error, a tool call failure or an agent coordination issue, and re-evaluate any past run from its logged events. Without spans, tool arguments and statuses, you are guessing.
Localise before you theorise. I compared each failing synthetic trace with a healthy run of the same task and reported the first span that differs. It named the injected span in 96 to 98% of cases for six fault classes and 81% for latency spikes (noise hides some). It cannot localise a fluent wrong answer (0%, since every span looks healthy), which needs an evaluation, not a trace rule.
Rules catch structural faults, not quality faults. Seven simple rules reached 1.00 recall on six fault classes and 0.99 on latency spikes in my synthetic corpus (circular: I wrote both the faults and the rules). Recall on fluent-wrong answers was 0 of 156, and 1,082 of 1,237 failing traces were flagged by at least one rule (87%); the unflagged ones were the fluent-wrong class.
Precision is where rules hurt. The latency rule at 4 times the median had precision 0.63; the context-limit rule 0.75 (benign near-limit prompts trigger it: 26% of them were flagged). A first version of my retry-storm rule had precision 0.48 because plan loops also repeat tool calls; requiring the repeats to be failures raised it to 1.00. Read your false positives; they tell you which two causes share a symptom.
A threshold is a trade-off. Latency precision/recall at 2, 3, 4, 6, 8 times the median: 0.07/1.00, 0.27/1.00, 0.63/0.99, 0.94/0.94, 0.99/0.81. Choose by the cost of a missed spike against the cost of reviewing a false alert.
Change one thing, then confirm on a held-out set. A fix that works on the failing trace may hurt others; rerun your evaluation set and report the change with an interval (module 9 of level 13).
Turn failures into tests. Export failure traces into the test suite as a regression corpus, so that once a fix is in, rerunning the trace should pass. Keep exemplar successes as golden paths too.
Look at the data. Fifteen minutes of reading real traces often saves hours; read twenty failing traces before building tooling.
Honest limits: the traces are synthetic, the fault list is mine, the detectors are simple rules, and a real production bug is messier (several causes at once, missing logs).