CSE Roadmap

Core tier · about 5 hours · 6 parts

Part 4 of 6 · 4 min read

Reading a trace

And a checklist.

Reading a trace

Raw JSON is for programs. The example renders any trace as a timeline:

TASK  Tell the customer when order A100 arrives.   [toy-model-1, prompt v1, seed 3]
  0  decide  "I need the order record first."  ->  get_order(order_id=A100)   (32 tokens)
     result  {"order": "A100", "customer": "[email protected]", "tracking": "TRK-1", ...
  1  decide  "Now ask the carrier where it is."  ->  carrier_status(tracking=TRK-1)   (32 tokens)
     result  TRK-1: in transit, estimated delivery 2026-11-04.   (1.2s)
  2  decide  "Tell the customer the carrier's date."  ->  send_email([email protected],
             body=Hello, your order A100 should arrive on 2026-11-03., key=A100-notice)
     result  Sent to [email protected].   (0.8s)
  3  decide  "The email went out, so I am done."  ->  finish
END   done after 3 steps, 144 tokens, 8.4s: Emailed the customer about order A100.

A stranger can see the bug at once: the carrier said 2026-11-04 and the email quoted 2026-11-03, the stale order date. That is what "every run leaves a trace a stranger could follow" means: each decision, what it was based on, and what happened, in order.

A checklist

  1. Events are the truth; state is derived. Append-only, one event per decision and per result.
  2. Run id and step number on every event, plus timestamps.
  3. Record the header: model, prompt version, seed, tool versions, limits.
  4. Record the result of every tool call, including errors, and what the model was shown (or its hash and version).
  5. Record intent before the effect, and result after, with an idempotency key on every effect.
  6. Redact before writing.
  7. Build replay early: exact replay, and new-prompt-on-old-results.
  8. Build a timeline view. If reading a trace is painful, nobody will read it.
  9. Link runs in a tree (a parent id) once sub-agents appear (Level 8), and send traces to a tracing tool when you need search and dashboards (Level 9).

Interview angle

"What do you store per step so a run can be replayed and debugged?" Say: an append-only event log with a run id and step number: the header (model, prompt and tool versions, seed, limits), and per step the model's output with what it was shown (or a hash), the tool call with its arguments and result, errors, tokens, latency and cost, and the ending status. Keep events as the truth and derive state by folding. Then say what it enables: exact replay (nothing random or external is left unrecorded), regression replay of new prompts on old tool results, resuming after a crash (with idempotency keys so a re-issued step cannot repeat a side effect), and diffing two runs for the first divergence. Add redaction, retention and access control for secrets and personal data. A good answer also admits that storing full model inputs is expensive and says how you trade it (hash plus version).

Check yourself

A run crashed after the email tool ran but before the result was saved. If you resume and issue the step again, what could happen? (The email is sent twice. Use an idempotency key for tools with side effects, and record the intent before the call, as the example does with its "pending" step.)

For the curious What the redaction does and does not catch

The original pattern only matched keys shaped like the example's fake key (sk-live- followed by hex digits), so a mixed-case key such as sk-ant-api03-AbC... or a Bearer token would have been written to the trace. The pattern now covers both shapes. It is still a short list of known shapes: a real system needs a rule for every kind of secret it handles, and should keep secrets out of tool results where it can. Redaction also changes the recorded model input, so an exact replay that compares inputs will report a difference for any message that held a secret.

What you will use daily at work

Practice What it looks like
Traces in a tracing tool Each run is a trace of model and tool spans with inputs, outputs, cost and timing.
Redaction before storage Secrets and personal data are masked before a trace is written.
Replay for regression tests Recorded tool results let you rerun a new prompt or model against old runs.
Version stamps Model, prompt and tool versions are stored in every trace.

Draft: confirm which tracing tools your team uses and what they redact by default.