Part 7 of 12 · 1 min read
How to build it so it holds up
- One trace per user request, with a span per model call, tool call, retrieval and handoff; carry the trace id across services and agents.
- Use standard attribute names where they exist, and record the convention version you used because names are still changing.
- Record tokens, model, tool name, status and error type; record prompt and output content only on purpose, redacted, with a short retention.
- Link evals to traces: store the trace id with each eval result (module 7), so a red build shows the run.
- Sample by outcome: keep errors, slow runs, low judge scores, flagged hallucinations and a random base rate.
- Alert on aggregates (error rate, p95 latency, cost per run) and use traces to explain them.
- Propagate context across async work and between agents, or the tree breaks.
- Measure the overhead of your own instrumentation and export asynchronously in batches.