Interview questions and limits: Human-in-the-loop

Covers: Interview questions, with the follow-ups; Common mistakes; What the example does not do.

Interview questions, with the follow-ups

How do you add human approval to an agent without blocking everything?

"Two separate decisions. First, what to gate: by consequence, so low-risk read-only tools run freely and irreversible or costly ones need approval. Second, how to wait: pause the run, save state durably, free the worker, and resume by run id when the decision comes. In my simulation a blocking worker took 1,663 minutes for 50 tasks at a 30-minute mean wait, against 272 with pausing." Follow-up: "What if the person never answers?" A time limit with a default of stop or re-plan, never approve; and on resume I re-check the premise, since 13 percent of approvals at that wait would be older than an hour.

Why not require approval for every action?

"Because the reviewer's attention is limited. In my model with attention halving at 20 requests per hour, gating all six actions per task caught 23 percent of harm and leaked 71 per 1,000 tasks, against 60 for gating only irreversible actions. That fatigue curve is my assumption, but the effect is well known: rubber-stamping.

How do you decide what needs approval?

"I classify each tool by reversibility and cost, and gate the irreversible and costly ones, optionally adding a rule on a risk score. The risk score from the agent itself is only useful if it is calibrated, so I test it against outcomes; my simulation's best policy depended on an assumed informative score." Follow-up: "How do you know the gate is right?" I sample auto-approved actions for hidden harm, track the approval and edit rates, and check missed risks in test cases.

How does the human role change as the agent matures?

"From executor to reviewer to governor: people first do the work, then review exceptions, then set policy and audit logs. In practice I lower the gate threshold gradually, with evidence from the metrics, and keep a sample of audits." Follow-up: "What stops that from creeping unsafe?" Thresholds change only through reviewed policy, with a log of escalations.

Common mistakes

Approving every step, which trains rubber-stamping.
Gating by prompt only, so the gate can be argued away.
Blocking a worker thread while a person decides.
Default-approve on timeout.
Resuming without re-checking the premise.
Vague requests ("Proceed?") that force the person to investigate.
No audit trail, or approvals not tied to the exact arguments that run.
Approving one action, executing another (changed arguments after approval).
Never measuring approval rate, edits or sampled misses.

What the example does not do

No real reviewer, and the fatigue formula is an assumption.
Independent actions and independent reviews; real harm clusters.
No "edit" outcome: a reviewer can only approve or reject here.
A fixed, informative risk score; an adversary can manipulate it.
One reviewer and one worker; queues, shifts and escalation chains are not modelled.
No interface effects: the quality of the request screen changes real catch rates.
Optional real experiment (I did not run this): show 100 recorded agent actions, 10 of them harmful, to reviewers in two conditions (all actions, or only flagged ones) and compare the harm caught, time per decision and approval rate.

Check yourself

You ask a person to approve every one of six actions in each task. Is that safer than approving only the irreversible ones? (Not necessarily. Attention is a resource the gate spends: in the model, gating everything caught 23 percent of harm because the reviewer was swamped. With a reviewer who never tires the order reverses.)

For the curious The default on timeout

With a 30-minute mean wait and a 60-minute timeout, 13.5 percent of approvals time out. The default must be stop or re-plan, never approve, and the premise must be re-checked on resume. A worker that waits for each answer took 1,663 minutes for 50 tasks, against 272 when the run is saved and the worker takes other work.

What you will use daily at work

Practice	What it looks like
Gate by consequence	Read-only never needs approval; irreversible or costly always does.
Show what will happen	The person approves an action they can see, not a vague summary.
Measure the gate	Approval rate, time to decide, and harm found in sampled auto-approved actions.

Draft: confirm who approves actions in your team and how long they have to respond.

What you must be able to say (one page)

Gate by consequence, not by step. Classify tools by risk: low (read-only, no side effects) never needs approval; high (irreversible or costly: send email, delete data, payments) always does. The right policy depends on the domain, and that a greater than 95 percent approval rate suggests over-cautious flagging, while missed real risks in test cases suggest the policy is too lenient.
More approvals are not more safety. In my simulation (6 actions per task, 30 percent irreversible, 93.5 harmful actions per 1,000 tasks, 10 tasks per hour), gating every action sent 6 requests per task; with reviewer attention that halves at 20 requests per hour, the reviewer's catch probability fell to 23 percent and 71 of 94 harmful actions per 1,000 tasks still got through. Gating only irreversible actions (1.8 requests per task) leaked 60; gating on the agent's own risk score of 0.5 or more (1.0 requests per task) leaked 45.
The same policies with a reviewer who never tires reverse the order: gate-everything leaked 9, irreversible-only 29, risk score 22. The lesson is that the reviewer's attention is a resource the gate spends, not that gating everything is wrong in principle.
The risk-score policy depends on the agent's self-assessment being informative. I assumed harmful actions score 0.7 and harmless ones 0.3 with standard deviation 0.2. Module 5 showed estimates can be poor; if yours are, the gate misses what it should catch.
Do not block the worker. With 50 tasks that each need one person's decision (mean wait 30 minutes), a blocking worker took 1,663 minutes; pausing the run, saving state and taking other work took 272. With a 120-minute wait: 5,904 against 713. This needs durable state (module 7).
Pauses end badly if they have no rule. With a 30-minute mean wait and a 60-minute timeout, 13.5 percent of approvals time out. The default on timeout must be stop or re-plan, never approve; and on resume the premise must be re-checked (module 7).
Roles shift as trust grows. People move from executors to reviewers to governors who set policy and audit escalation logs. The gate is a dial you turn with evidence.
Documented facts (31 October 2026): the Claude Agent SDK's default permission mode sends tool calls that need approval to your canUseTool callback and denies them if there is no callback; PreToolUse hooks can block a call; dontAsk mode denies everything that would prompt. The OpenAI Agents SDK lists human in the loop as a feature. LangGraph documents checkpointers as enabling human-in-the-loop.