Part 3 of 13 · 2 min read
Compare with a healthy run; the first span that differs is where to look
Reading a failing trace in isolation leaves you guessing at what's normal; a healthy trace of the same task gives you something to subtract.
Think of comparing two photos of the same street taken a day apart to spot what changed. Staring at just the "after" photo, you might notice anything or nothing. Lined up against the "before" photo, the one new pothole jumps out immediately.
In plain words: localisation means finding the first step where a failing run departs from a healthy one, which turns "something in this huge trace is wrong" into "span 3 is wrong" — and the root cause is whatever change, if removed, would make the failure disappear, which localisation narrows down to but does not by itself prove.
A worked example. For a trace with a retry-storm fault — five consecutive failing tool calls inserted at position 3 — comparing span-by-span against a healthy reference trace flags span 3 as the first point of divergence, because that's exactly where the healthy trace has one successful tool call and the faulty trace has the first of five failing ones. Across six structural fault classes (swallowed error, retry storm, plan loop, empty retrieval, bad arguments, context truncation), this first-divergence check lands on the actually-injected fault span 96 to 98% of the time.
The key lines. From code/mini-debug/detect.py:
# from code/mini-debug/detect.py
def first_divergence(good, bad):
"""Index in `bad` of the first span that differs from the reference run; None if the traces match."""
for i in range(min(len(good.spans), len(bad.spans))):
if differs(good.spans[i], bad.spans[i]):
return i
if len(bad.spans) != len(good.spans):
return min(len(good.spans), len(bad.spans))
return None
Read it as: the function returns the index of the first mismatch and stops looking. It deliberately does not try to find every difference or rank them. Localisation's whole job is pointing at the one place to start hypothesising. That is also why it can be wrong when a fault produces no structural difference at all. A fluent wrong answer localises at 0%, because nothing in its spans looks abnormal.
Root cause, localisation
The root cause is the change that, if removed, makes the failure disappear; a symptom can have several causes, and two causes can share a symptom. Localisation is finding the first step where a failing run departs from a healthy one.
Example: localisation pointing at span 3 for the retry storm is the start of the investigation, not the end of it — the actual root cause (the loop treating a tool error as something to retry rather than escalate) still needed a hypothesis and a test to confirm, which is Part 3's job.