Part 5 of 6 · 5 min read
Exercises
Easy
Easy 1: Run and read (20 minutes)
In code/mini-agent, run pytest and python demo.py. From section 1 of the demo output, say which task needed the repeat guard to stop, and which needed the step limit.
Done when: you name both tasks and for each give the status shown (repeat or steps) and the number of steps it used.
Hint
stubborn stops on repeat after 3 steps; too-long stops on steps after 5.
Easy 2: Find the tool call in the SDK loop (15 minutes)
Open sdk_loop.py. Find (a) the line where the tool-call block is read, (b) the line where the tool result is appended, and (c) the line that decides the run is finished. Write the line numbers and one sentence for each saying what would break if it were missing.
Done when: you have three line numbers and three sentences, and one of them says why the tool result must carry the same id as the tool call.
Hint
the tool result is matched to its call by id, so the model can tell which call each result answers when it made more than one.
Easy 3: Name the pieces (15 minutes)
Take any framework you have heard of and look at its quick-start page. Fill in the table from section 3 with its name for each piece (loop, tool definition, limit, trace).
Done when: at least four rows are filled, and for each one you mark whether it was clear from the page or you had to guess.
Hint
where a page does not name a piece, that gap is an answer too.
Medium
Medium 1: Add a tool to all three shapes (60 minutes)
Add a list_files(prefix) tool that lists the document paths under a prefix, and a task "How many documents are there under teams/?" with the right answer. Make the model handle it, and add tests.
Done when: the task passes on the raw loop, the SDK-shaped loop and the graph, and same_behaviour() still reports 11 of 11 identical.
Hint
add the tool to toolbox, a pattern to model.py, and the task to TASKS with its expected text; the three runners share the same tools, so you change only one place.
Medium 2: Close a hidden default (60 minutes)
Subclass or configure AutoAgent so that every failure in section 4 is fixed except the policy difference on too-long: no truncation of the result, a repeat guard, retries only for tools marked idempotent, a visible error instead of (no output), and retries counted in the trace. Reuse as little of kernel.py as you can.
Done when: the tasks other than too-long pass, the duplicate-effect experiment gives 1 saved note, and you can say in two sentences how what you wrote differs from kernel.py.
Hint
retry only when the tool has effect=False, or when the call carries a key you have already seen succeed.
Medium 3: Add a human approval node (90 minutes)
In the graph version, add a node before tools that pauses when the tool is write_file and asks a callback for approval; a refusal returns the observation Error: the user did not approve this write. Use the checkpoint hook to save the state at the pause.
Done when: the write task passes with an approving callback, ends cleanly with a refusing one (and no file is written), and a test shows the saved state at the pause.
Hint
a node that returns a status and routes to a stop is easier than raising an exception; save copy.deepcopy(state) in the hook.
Hard
Hard 1: Port to a real SDK (150 minutes)
Replace FakeClient with a real client for one provider you have access to, using its current documentation for the request and response fields. Keep tools, limits and tasks unchanged. Run each task 5 times. (I did not run this, so your numbers are the ground truth.)
Done when: you report the pass rate per task across the 5 runs, the mean steps and the cost, and you list every field name or behaviour that differed from the shape in sdk_loop.py.
Hint
print the raw response of the first call and compare it with FakeClient.create field by field before you change anything else.
Hard 2: Port to a real framework and audit it (150 minutes)
Re-implement the agent in a real framework of your choice. Answer the five questions in section 5 from its documentation and by writing a test for each (for example, a tool that raises, a tool that returns 5,000 characters, a model that repeats itself). Then run the ten tasks. (I did not run this.)
Done when: you have a table of five questions with the answer, the page or test that supports it, and the result, and you report how many of the ten tasks differ from your raw loop, with the cause of each difference.
Hint
for truncation, return a long result containing a fact only at the end, and see whether the answer survives.
Hard 3: Keep the framework honest in CI (120 minutes)
Put the ten tasks, run against your framework version, into an automated test with a pinned version. Make a deliberate "upgrade" (change a default or a version) and show the suite catching a change in behaviour that no code change of yours caused.
Done when: a recorded run shows the suite passing on the pinned setup and failing after the change, with the first differing task and the exact difference shown.
Hint
a pinned version file plus a test that prints the version is the simplest way to prove which setup ran.