Now each tool call fails at random with probability p ("service busy, try again"). Plan only stops at the first failure; adaptive just calls again; replanning spends a model call to repair, up to three times:
Tasks done correctly as the tools get flakier
p (chance a call fails)
adaptive
plan only
plan with replanning
0%
100%
90%
100%
10%
100%
69%
99%
20%
100%
50%
96%
30%
100%
42%
90%
Plan only decays quickly because every step must succeed: with k steps and failure p, the chance of getting through is roughly (1 - p) to the power k, so longer plans fall faster. Adaptive hides the failures, because the next model call simply retries. But look at the bill at p = 20%:
Strategy
Model calls
Tool calls
Prompt words
Time units
adaptive
5.4
4.4
564
20.5
plan only
1.6
1.9
112
6.1
plan with replanning
2.8
4.3
205
10.6
(Plan only looks cheap because it often gave up early; compare it with the success column, not on its own.) Replanning recovers most of the robustness for about half the time and a third of the words of adaptive.
Two honest limits of this toy. First, the repair here is "run it again", which is easy; real failures (wrong tool, wrong argument, a missing fact) need a new plan, and a real model may write a worse one. Second, adaptive had no step cap in this experiment beyond a generous one; with a failure that never goes away it would loop. Module 6 is about exactly that.
Choosing between them
Situation
Favours
The steps are knowable from the request, and some are independent
Plan: parallel rounds, fewer model calls
The next step depends on what the last one returned
Adaptive, or a plan with decision points
Steps are costly or risky and a person should approve them
Plan: show it before running anything
You must audit what the agent meant to do
Plan: it is a record you can store
Tools fail often or inputs are messy
Adaptive or replanning
Few steps (one to three)
Adaptive: a plan adds a call and saves little
A cheap model can do the steps but a strong one is needed to decide
Plan: strong model plans, cheap one executes
Making plans robust
Validate the plan before running it: known tools only, arguments present, dependencies exist, no cycles. A model-written plan is untrusted input.
Number the steps and name their dependencies, so you can run rounds in parallel and report which step failed.
Keep the plan short. More steps means more things that can be wrong and more places to fail.
Define what triggers a replan: a step error, an empty result, a surprise that contradicts an assumption, or a marked decision point. Without triggers the plan is a script.
Cap replans (three here). An agent that replans forever is a loop with extra cost.
Keep what is done. After a replan, finished steps and their results stay; only the unfinished part changes.
Check the answer, not just the plan. A plan that runs cleanly can still produce the wrong answer; module 4 adds a check step.
Interview angle
"When is an upfront plan better than adaptive planning?" Say it depends on how much of the path is knowable and what a mistake costs. An upfront plan wins when steps are largely known, some can run in parallel, the work is expensive or risky and needs approval, or you need an audit trail; it also reduces model calls and context growth. Adaptive wins when each step depends on what you find, when tools are unreliable, or when the task is short. Then give the hybrid: plan, execute, replan on defined triggers, with a replan cap. Mention that plans are brittle exactly because they are written before the facts, and that you would measure both on your own tasks (success, calls, time) rather than guess.
Check yourself
A plan has the steps "capital of France", then "population of that capital". Which strategy can write the whole plan before running anything, and which cannot? (A fixed chain can be written up front. "Which country has the capital with the largest population" needs the list of countries first, so the plan must be extended after that result, and only a replanning strategy can do it.)
For the curious A small crash found in the example code
Asking for the answer of an "argmax" plan before the list of countries had been expanded raised an IndexError that the code did not catch. It now returns no answer. The numbers in the tables come from tools that fail at a chosen rate, with a fixed seed per run, so they show the direction and not a measurement of a real tool.
What you will use daily at work
Practice
What it looks like
A plan stored as data
A list of steps with dependencies, so it can be shown, edited and resumed.
Parallel steps for independent work
Independent tool calls run together; dependent ones wait.
Replan on failure, with a limit
One replan on a failed step, then stop and report.
Draft: confirm how your framework represents plans and parallel calls.