Part 1 of 4 · 1 min read
Overview
An agent is a loop of model calls and tool calls, and most real failures look like success on the surface. The skill that interviewers test is not knowing a catalogue of bugs; it is how you narrow a vague complaint ("it gave the wrong refund") to a span, a cause and a test.
About the example, and what is real
Real: the trace data structure, eight fault generators, seven rule detectors, a first-divergence localiser, a regression-case converter and 23 tests in code/mini-debug. Assumed: the fictional support agent, the fault classes and rates (about 23% faulty, 20% benign look-alikes), latency noise (log-normal, sigma 0.5), the context limit of 16,000 tokens and the 12-times latency spike. Not real: no real agent logs were analysed; because I wrote the faults and the detectors, high recall shows that the code does what I intended, not that it would find real bugs.