CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 9 of 12 · 2 min read

Mistakes that make traces useless

Common mistakes, and what the example does not do.

Mistakes that make traces useless

  • Logs instead of traces, so steps cannot be tied to a run.
  • Head-sampling at 1 percent and then missing every rare failure.
  • No token or cost attributes.
  • Capturing full prompts with personal data and no retention.
  • Broken context propagation, so tool calls appear as separate traces.
  • Hard-coding attribute names from a draft convention without a version note.
  • Tracing only the model call, not tools and retrieval.
  • No link between eval results and traces.

What the example does not do

About this example What the example does not do
  • No OpenTelemetry SDK, exporter, collector or backend; the tracer is minimal and synchronous.
  • A simulated agent with simple durations; real runs have retries, parallel calls and streaming.
  • Self time assumes sequential children; parallel spans need the critical-path calculation.
  • The tail rule is evaluated offline on complete traces; a real collector must buffer.
  • The convention status comes from a secondary source; the specification was not read.
  • Optional real experiment: instrument a real agent with the OpenTelemetry SDK, export to a local backend, and compare your measured per-span overhead and attribute names with this module.

Check yourself

A run was slow. How do you find out whether the model or a tool was the problem?

Show answer

Read the trace: the span tree shows start, end and parent for each model and tool call. In the simulated run, model calls were 2.94 s of 3.28 s.

For the curious Sampling decides what you can debug later

Over 20,000 simulated runs, 3.9 percent had a tool error. Random head sampling at 1 percent kept only 1 percent of those error traces; deciding after the run (a tail rule) can keep the errors you need. The OpenTelemetry Gen-AI conventions were still marked Development in the secondary source read, so record which version of the names you used.

What you will use daily at work

Practice What it looks like
One trace per request Spans for each model call, tool call, retrieval and handoff.
Record tokens, model, tool and error type Content only on purpose, redacted, with a short retention.
Keep error traces Sample after the run, not only at the start.
Store the trace id with each eval result A red build then points to a run.