CSE Roadmap

Core tier · about 6 hours · 13 parts

Part 2 of 13 · 2 min read

A trace of spans turns "it gave the wrong refund" into something checkable

A vague complaint about an agent is unanswerable until it is broken into the individual steps the agent actually took.

Think of a black-box flight recorder: after a crash, investigators don't start from theories, they start from the recorded sequence of instrument readings leading up to it.

In plain words: a trace is the record of one request, made of spans — one per guard, retrieval, planner call, tool call or answer — each with its own duration, status, arguments and sizes, and a swallowed error is the specific, common failure pattern where a tool span's status is "error" but the agent's final answer carries on as if nothing went wrong.

A worked example. In the synthetic support-agent traces, a swallowed-error fault is detected simply by checking whether the last tool span in a trace has status == "error" and the final answer was never hedged to reflect that failure. Across 6,000 synthetic traces, this single structural check caught all 169 true swallowed-error cases with zero false positives — because the pattern it's looking for ("a tool failed, and nothing downstream acknowledges it") is written directly into the structure of the spans.

The key lines. From code/mini-debug/detect.py:

# from code/mini-debug/detect.py
def tool_spans(t):
    return [s for s in t.spans if s.kind == "tool"]


def swallowed_error(t):
    ts = tool_spans(t)
    return bool(ts) and ts[-1].status == "error" and not t.hedged

Read it as: the check only needs two fields on two objects: ts[-1].status and t.hedged. This is the entire value of structured tracing. A bug that would take real effort to spot by reading prose output becomes a two-line check. That happens once the agent's steps are recorded as spans with a status field.

Trace and span, swallowed error

A trace is the record of one request; a span is one step in it (a guard, retrieval, planner call, tool call or answer) with its duration, status, arguments and sizes. A swallowed error is a tool failure that the agent continues past as if it had succeeded.

Example: "the refund tool span had status error, there was no later tool call, and the answer did not hedge" is a swallowed error stated entirely in terms of spans — which is exactly the vocabulary an interviewer wants to hear you reach for before any talk of causes.