CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 7 of 12 · 1 min read

How to build it so it holds up

  • One trace per user request, with a span per model call, tool call, retrieval and handoff; carry the trace id across services and agents.
  • Use standard attribute names where they exist, and record the convention version you used because names are still changing.
  • Record tokens, model, tool name, status and error type; record prompt and output content only on purpose, redacted, with a short retention.
  • Link evals to traces: store the trace id with each eval result (module 7), so a red build shows the run.
  • Sample by outcome: keep errors, slow runs, low judge scores, flagged hallucinations and a random base rate.
  • Alert on aggregates (error rate, p95 latency, cost per run) and use traces to explain them.
  • Propagate context across async work and between agents, or the tree breaks.
  • Measure the overhead of your own instrumentation and export asynchronously in batches.