Part 4 of 6 · 6 min read
Interview questions and limits
Covers: Interview questions, with the follow-ups; Common mistakes; What the example does not do.
Interview questions, with the follow-ups
How do you know when an agent is done?
"I don't rely on the agent saying so. I write success criteria as checks (tests, a schema, a rubric) and have something independent verify the claim, with a step and cost budget as a backstop and an escalation path for tasks that cannot finish. In my simulation an agent that over-claimed at 5 percent per step declared unfinished work done in 40 percent of runs; a verifier cut that to under 1 percent." Follow-up: "What if there is nothing to check mechanically?" Use a rubric with a separate judge model and measure its error against human labels; its error rate is then your false-completion rate, as the verifier-leak sweep shows (10 percent at a 30 percent judge error).
How do you stop an agent that is looping?
"I track progress explicitly, for example new subgoals completed or new information gathered, and count steps without progress. At a threshold the monitor replans with fresh instructions; after a couple of failed replans it escalates. I set a budget as well because the monitor can fail. The threshold is a trade-off: in my test a threshold of 2 escalated 30 percent of all tasks although they were possible, and a threshold of 12 never escalated and cost almost twice as much." Follow-up: "How do you tune the threshold?" From traces: the distribution of idle steps in successful runs against stuck runs.
What is a progress ledger?
"It is an explicit record, updated each step, of whether the request is satisfied, whether the agent is making progress or repeating itself, and what should happen next. Magentic-One's orchestrator keeps one alongside a task ledger of facts and plan, and uses it to decide whether to continue, replan or stop." Follow-up: "What does it cost?" Extra tokens per step; I would use a smaller model for it and measure the saving against the cost: in my simulation monitoring raised cost per task by 7 percent and cut cost per successful task by 11 percent compared with verification and a budget.
Your agent was given a task that cannot be done. What should happen?
"It should stop early and say so, with what it tried, rather than spend the whole budget or invent an answer. That needs a stall or infeasibility signal and an escalation path. In my test, without a monitor, correct outcomes fell to 0.45 when 40 percent of tasks were impossible; with one they stayed near 0.88." Follow-up: "How does the agent know?" It may not, which is why the monitor is outside the agent and based on measurable progress.
Common mistakes
- Treating "done" as proof.
- A budget with no escalation path, so every hard task ends as a timeout.
- Vague success criteria ("good", "complete"), which no verifier can check.
- The same model, prompt and context grading its own work.
- A stall detector with no measure of progress (it counts steps, not subgoals).
- A stall threshold chosen by feel rather than from traces.
- No cap on replans, so the agent is replanned forever.
- Not testing the failure paths with impossible and looping tasks.
What the example does not do
- No real agent. The over-claim, loop and drift rates are assumptions and real ones depend on model, task and prompt.
- Binary progress. Real progress is fuzzy; a "new subgoal" signal may itself be noisy.
- A perfect progress signal. The monitor sees subgoal completion exactly; a real monitor infers it.
- A fixed budget rather than one scaled to the task's value.
- No human in the loop. Escalation just stops the run; module 8 covers approval and handover.
- Optional real-agent experiment (I did not run this): run an agent on 50 tasks with 10 known to be impossible, log claims, verifier verdicts and idle steps, and compute false completions and wrong escalations for two thresholds.
Check yourself
An agent says "done" and the run stops. How do you know it is done? (A claim of done is a hypothesis. Something other than the agent has to check it, such as tests, a verifier or a person.)
For the curious A budget is a limit, not a goal
In the simulation, an agent allowed to declare itself done reported false completion on 40 percent of tasks; a verifier cut that to 0.9 percent. A budget alone stops runaway cost but cannot tell finished work from unfinished work. A stall detector that replans and then escalates handled the impossible tasks (15 percent of them), at the price of 7.5 percent wrongly escalated.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Success criteria something else can check | Tests passing, a schema valid, a person approving. |
| Limits on steps, cost and time | So a stuck run ends. |
| An escalation path | What happens when the task cannot be done. |
Draft: confirm who is paged or asked when an agent escalates in your setup.
What you must be able to say (one page)
- A goal for an agent is a contract with three parts: success criteria that something other than the agent can check, limits (steps, cost, time) and stop and escalation rules (what to do when the task cannot be finished).
- There are three ways an agent run ends: budget-based termination (a limit on time, cost or iterations), semantic termination (a model decides the task is complete) and external signals (a person or an API stops it). Production designs combine them, for example "stop after 10 iterations or when a model judges the quality threshold met, whichever comes first".
- A claim of "done" is a hypothesis, not a result. In my simulation an agent that over-claims in 5 percent of its steps once it is three-fifths of the way through declared unfinished work done in 40 percent of tasks; with an independent verifier checking the claim, false completions fell to 0.9 percent.
- A budget is a limit, not a goal. It stops runaway cost but cannot tell a finished task from an unfinished one: "verified plus budget" stopped 35 percent of tasks on the budget alone.
- Monitoring means tracking progress explicitly. A multi-agent orchestrator can keep a task ledger (facts and plan) and a progress ledger (is the request satisfied, are we in a loop, are we making progress, who goes next). The simulation's monitor counts steps without a new subgoal and triggers a replan, then an escalation.
- Quote these numbers, with the caveat that the agent is a stochastic stand-in (5 subgoals, 15 percent of tasks impossible, 20,000 tasks per policy):
| Policy | Finished | Said done, was not | Escalated rightly | Escalated wrongly | Cost per task | Correct outcomes |
|---|---|---|---|---|---|---|
| agent says done | 59% | 40% | 0 | 0 | 21.8 | 0.59 |
| verified + budget | 64% | 0.9% | 0 | 0 | 20.3 | 0.64 |
| verified + stall detector | 77% | 0.8% | 15% | 7.5% | 21.8 | 0.92 |
Correct means finished a possible task or escalated an impossible one. Monitoring cost 0.3 model calls per step, so cost per task rose, but cost per successful task fell from 36.8 to 31.7 to 28.3. - The stall threshold is a trade-off: with a threshold of 2 steps, 30 percent of possible tasks were escalated wrongly; at 5 steps, 7 percent; at 12 steps, none, but the agent never escalated an impossible task and cost rose from 14.2 to 25.7. - Impossible tasks are why you need an escalation path. Without a stall detector, correct outcomes fell from 0.75 to 0.45 as impossible tasks went from 0 to 40 percent; with one they stayed at 0.88 to 0.89.