Part 10 of 12 · 1 min read
Quick notes
- A trace is a tree of timed spans describing one run: a root span for the agent, child spans for each model and tool call, each with timing, status and attributes.
- OpenTelemetry is the vendor-neutral standard, and its Gen-AI semantic conventions name model calls, tool executions and agent workflows, but they were still in development status in mid-2026, so verify names against the specification.
- Per-step cost and latency fall out of a trace; "where does the time go?" is the first question it answers.
- Sampling is a decision about what you can debug later: head sampling throws away most error traces; a tail rule keeps every error and the slow runs while storing a small share of the rest.
- Instrumentation is cheap but not free, and prompts and tool outputs contain personal data: capture content only on purpose, redact it, and set retention.