CSE Roadmap

Common tier · about 4 hours · 11 parts

Part 7 of 11 · 2 min read

Goal setting and monitoring: interview questions

How do you know when an agent is done?

"I don't rely on the agent saying so. I write success criteria as checks (tests, a schema, a rubric) and have something independent verify the claim, with a step and cost budget as a backstop and an escalation path for tasks that cannot finish. In the example's simulation an agent that over-claimed at 5 percent per step declared unfinished work done in 40 percent of runs; a verifier cut that to under 1 percent." Follow-up: "What if there is nothing to check mechanically?" Use a rubric with a separate judge model and measure its error against human labels; its error rate is then your false-completion rate, as the verifier-leak sweep shows (10 percent at a 30 percent judge error).

How do you stop an agent that is looping?

"I track progress explicitly, for example new subgoals completed or new information gathered, and count steps without progress. At a threshold the monitor replans with fresh instructions; after a couple of failed replans it escalates. A budget is needed as well, because the monitor can fail. The threshold is a trade-off: in the example's test a threshold of 2 escalated 30 percent of all tasks although they were possible, and a threshold of 12 never escalated and cost almost twice as much." Follow-up: "How do you tune the threshold?" From traces: the distribution of idle steps in successful runs against stuck runs.

What is a progress ledger?

"It is an explicit record, updated each step, of whether the request is satisfied, whether the agent is making progress or repeating itself, and what should happen next. Magentic-One's orchestrator keeps one alongside a task ledger of facts and plan, and uses it to decide whether to continue, replan or stop." Follow-up: "What does it cost?" Extra tokens per step; I would use a smaller model for it and measure the saving against the cost: in the example's simulation monitoring raised cost per task by 7 percent and cut cost per successful task by 11 percent compared with verification and a budget.

Your agent was given a task that cannot be done. What should happen?

"It should stop early and say so, with what it tried, rather than spend the whole budget or invent an answer. That needs a stall or infeasibility signal and an escalation path. In the example's test, without a monitor, correct outcomes fell to 0.45 when 40 percent of tasks were impossible; with one they stayed near 0.88." Follow-up: "How does the agent know?" It may not, which is why the monitor is outside the agent and based on measurable progress.