Part 4 of 5 · 4 min read
Exercises
Easy
Easy 1: Run and read (20 minutes)
In code/mini-plan, run pytest and python demo.py. From sections 2 and 3 of the demo, report the correct rate and the model calls of each strategy at p = 0 and p = 0.3.
Done when: you give a table of both and say which strategy loses the most between the two.
Hint
the loss for plan only comes from both a structural miss and from failures.
Easy 2: Draw the graph (20 minutes)
For the goal "Plan a team offsite: pick a date, book a room, order lunch for the confirmed headcount, send invitations", list the steps, mark which depend on which, and group them into rounds.
Done when: you have the steps, the dependencies and at least two rounds, and you name the critical path.
Hint
lunch needs the headcount, which needs the invitations answered, so it is late in the chain.
Easy 3: Plan or adapt? (15 minutes)
For each, say which strategy you would start with and why: (a) convert 12 CSV files to JSON; (b) debug why a test fails; (c) answer "what is the capital of France?"; (d) research three competitors and write a comparison.
Done when: each answer names a strategy and a one-sentence reason that mentions knowability, cost or failure.
Hint
(a) is knowable and parallel; (b) depends on every finding; (c) needs no plan.
Medium
Medium 1: A new task, three ways (60 minutes)
Add the question "Which two countries have capitals whose populations add up to more than 400000?" to the example. Decide whether it can be planned up front, extend make_plan and the finishing rules, and run all three strategies.
Done when: you have results for all three strategies and say which needed a replan and why.
Hint
you cannot know which capitals matter before reading the list, so expect a replan.
Medium 2: Add a plan validator (60 minutes)
Write a function that rejects a plan with an unknown tool, a missing argument, a dependency on a step that does not exist, or a cycle. Add a test for each rejection and one for a valid plan.
Done when: five tests pass and the executor refuses to run an invalid plan.
Hint
check cycles with the same rounds loop; if no step is ready but some remain, reject.
Medium 3: Sweep the replan cap (60 minutes)
Run plan with replanning at p = 0.3 with the cap set to 0, 1, 2, 3, 5 and 10. Plot or tabulate correct rate and average model calls.
Done when: you report the table and say where more replans stopped helping.
Hint
with the toy failures, extra replans beyond three recover very little.
Hard
Hard 1: A real planner (150 minutes)
Replace the rule-based planner with a real model that outputs the plan as JSON (steps, tools, dependencies). Validate it, run it, and compare with a real adaptive agent on 20 tasks you write. (I did not run this, so your numbers are the ground truth.)
Done when: you report success, calls and time for both, and list at least three ways real plans were wrong or incomplete.
Hint
ask for JSON only, validate, and on invalid output re-ask once with the error message (module 2).
Hard 2: Decision points and partial replans (150 minutes)
Extend the executor so a plan can mark steps as "decision points". After one finishes, ask the planner to extend or modify only the unfinished part, keeping completed results. Test with a task where the plan is wrong halfway.
Done when: completed steps are never re-run, a test proves it, and you report the saving against a full re-plan.
Hint
store results by step id; a replan returns new steps, never the finished ones.
Hard 3: Cost-aware strategy choice (180 minutes)
Build a router that looks at a task and picks adaptive, plan only, or plan with replanning. Use features you can compute (number of steps expected, whether a list is read first, tool failure rate seen recently). Evaluate against always using each fixed strategy.
Done when: your router beats every fixed strategy on a stated score (for example success minus a cost penalty) on a held-out set, or you explain honestly why it does not.
Hint
keep a final test set you never tune on.