CSE Roadmap

Core tier · about 6 hours · 13 parts

Part 9 of 13 · 2 min read

Answering debugging questions in an interview

Walk me through the hardest bug you found in an agent you built.

Use the six beats. Example (fictional, built from the example trace): "Customers were told a refund was issued when it was not. I pulled traces for the complaints: the refund tool span had status error, there was no later tool call and the answer did not hedge. Compared with a healthy refund trace, the first difference was that span. Hypotheses: tool outage, bad arguments, or the loop ignoring errors. Arguments were valid and the tool was up, so the loop treated an error string as an observation. Fix: tools return structured errors, the loop stops or hands off on an error, and the answer must say so. I added the trace as a regression test and an alert on a tool error followed by a non-hedged answer." Follow-up: "How did you know it was the cause?" Changing only the error handling made the failing cases pass and the held-out set did not regress.

An agent gives a confident wrong answer and every span looks fine. What now?

"Trace rules will not see this. I would sample the failing cases, label the error type (stale data, misread tool result, hallucination), build an evaluation set from them, and use a grader plus human spot checks. Fixes could be grounding, citations, a verification step or a handoff." Follow-up: "How do you know the fix worked?" A measured improvement with an interval on a held-out set.

Your alert for slow requests fires constantly. What do you do?

"Look at the false positives. In the example's example a threshold of 2 times the median had 7 percent precision; at 6 times it was 94 percent with 94 percent recall. I would pick the threshold by the cost of missing a spike against the cost of a review, and alert on a rate over a window rather than single spans." Follow-up: "What else could cause slow spans?" Provider latency, large context, retries, a cold cache.

How do you stop the same bug coming back?

"Add the failing trace to a regression corpus, add a guard that catches the pattern earlier, and monitor the rate. Keep good traces as golden paths too." Follow-up: "What if the trace contains personal data?" Redact or synthesise before storing it.