Part 8 of 12 · 2 min read
Answering tracing questions in an interview
How would you debug an agent that fails in production?
"From its trace: a tree of spans for the run with model calls, tool calls, arguments, errors and timings. I find the first span that went wrong, look at its inputs and the previous step's output, then reproduce it with recorded tool outputs and add it as an eval case. With OpenTelemetry's Gen-AI conventions the attribute names are standard, so the same tooling works across frameworks, although the conventions are still in development." Follow-up: "What if it is not in your traces?" Then my sampling dropped it, which is why I use tail sampling that keeps errors, slow runs and low-scoring runs.
How do you track cost per request?
"Record token counts on each model-call span and roll them up over the trace, then multiply by the price table, per user or feature. In the example's simulation the example run was $0.027 and 90 percent of its time was in model calls. I alert on cost per run and on the long tail, not just the mean." Follow-up: "Where do cached tokens go?" As separate attributes with their own price (Level 7 module 7), or the cost will be wrong.
Head or tail sampling?
"Head sampling is simple and cheap but random, so it keeps 1 percent of failures if it keeps 1 percent of traces, as in the example's simulation. Tail sampling decides after the trace completes, so I can keep all errors and slow runs for 5 percent of storage, at the price of buffering traces and running a collector." Follow-up: "What if volume is huge?" Combine them: sample the boring majority at the head, and use the tail rule for the interesting ones, accepting the buffering cost.
What should not go in a trace?
"Secrets, and by default raw prompts and outputs with personal data. The conventions treat message content as opt-in. I redact or hash sensitive fields, limit retention, and restrict access, because traces are a copy of user data that engineers can read." Follow-up: "How do you debug without content?" Capture content for a sampled or flagged subset with consent and a short retention, or log structured fields instead of free text.