Part 5 of 6 · 4 min read
Exercises
Easy
Easy 1: Run and read (20 minutes)
In code/mini-trace, run pytest and python demo.py. From section 2 of the demo, give the lowest detail level at which each of the nine questions can be answered.
Done when: you list the nine questions with a level for each and say which level you would choose for a production agent and why.
Hint
the matrix in section 2 gives the answer: 0, 1, 2, 2, 3, 4, 4, 4, 2.
Easy 2: Find the bug in a trace (15 minutes)
Using timeline on record_run("A101", 7, "v1"), find which step introduced a wrong date (if any), and write two sentences saying what you looked at to find it.
Done when: you name the step, and name the two events you compared.
Hint
compare the date in the carrier result with the date in the email body.
Easy 3: What would you add? (15 minutes)
A support agent searches a knowledge base, reads a ticket and writes a reply. List ten fields you would record per step, and mark each as essential, useful or optional.
Done when: the list has at least four essential fields, and at least one field you chose to store as a hash or a length instead of the full value, with the reason.
Hint
store a hash of a long document and its length; store the full text of a short instruction.
Medium
Medium 1: Add a new tool to the trace (60 minutes)
Add a lookup_policy(topic) tool to the world, a step to the scripted model that calls it before the email, and make sure the recorder, fold, replay and timeline handle it. Add tests for each.
Done when: all tests pass, exact replay still matches in every run, and the timeline shows the new step.
Hint
events for a new tool need no new code in fold if they use the same tool_call kind.
Medium 2: A diff between traces (60 minutes)
Extend first_divergence to return every differing event (up to ten) and a readable explanation such as "step 2 sent date 2026-11-03, the other sent 2026-11-04". Test with v1 against v2 runs.
Done when: a test shows at least two differing events with a text that names the changed field and both values.
Hint
compare field by field, skipping seconds and tokens.
Medium 3: Retention and sampling (90 minutes)
Add a policy: keep full level-4 traces for runs that failed or were slow, but only level 1 for 90% of normal runs and full traces for 10% (chosen by a hash of the run id so it is repeatable). Measure storage per 1,000 runs against keeping everything, and what questions you can still answer for a normal run.
Done when: you report the size saving and show one failing run that is fully traced and one normal run that is not.
Hint
take an integer from the first 8 hex digits of a hash of the run id, modulo 100.
Hard
Hard 1: Trace a real agent (150 minutes)
Add the recorder to your own Level 5 agent loop (real model, real tools), writing JSONL. Run 10 tasks. Build exact replay and check it reproduces each final answer. (I did not run this, so your numbers are the ground truth.)
Done when: replay matches on all 10 runs without any live call, and you list every source of nondeterminism you had to record to get there (time, random ids, model, tools).
Hint
record seeds, timestamps and generated ids; do not call time.time() inside replayed code.
Hard 2: Resume a real run (150 minutes)
Make your agent resumable: after each step write the events to disk before the next step. Kill the process at random points (including between a side effect and the save) 30 times and prove that with idempotency keys no side effect ever repeats, and without them find the cases that do.
Done when: a test kills the process at 30 random points, and a table counts duplicate effects with and without keys.
Hint
write the intent event, flush, then run the effect, then write the result.
Hard 3: A regression suite from traces (150 minutes)
Collect 30 recorded runs, replay them with a changed prompt, and flag runs whose final answer changed. Report how many replays asked for an unrecorded call and what that does to trust in the result.
Done when: a report lists changed answers, unrecorded-call counts, and a rule for when a replay result is too unreliable to use.
Hint
a replay that asks for a call not in the trace has left the recorded world; treat its result as a lead, not a verdict.