Part 4 of 6 · 6 min read
Interview questions and limits
Covers: Interview questions, with the follow-ups; Common mistakes; What the example does not do.
Interview questions, with the follow-ups
How would you use extra compute to make an agent more accurate?
"Repeated sampling with a way to choose: generate N candidates and select with a verifier, ideally something deterministic like tests. In my simulation best-of-3 with a verifier took accuracy from 0.55 to 0.73 for 6 calls, and returns diminished after that (0.82 at 15 samples and 30 calls). Majority vote is cheaper but only helps when wrong answers are scattered; when half of them agreed it capped at 0.65." Follow-up: "What if you have no verifier?" Then I rely on self-consistency for short-answer tasks, and measure how correlated the errors are, because if wrong answers agree the vote gains little.
How does an agent decide which task to do first?
"With a rule that matches the objective: value per cost under a budget, with deadlines and dependencies as constraints. On my simulated queue it reached 99 percent of the optimal value against 58 percent for arrival order. But if an agent estimates the values itself, the estimate may be noisy: at 100 percent error, ranking by value per cost fell to 0.84, below simply taking the cheapest first at 0.88. So I calibrate the estimates against outcomes before relying on them." Follow-up: "What about dependencies?" Rank by the value unlocked, including downstream tasks, and respect the dependency order (module 1 of this level).
Explain exploration against exploitation for agents.
"If I have several prompts, tools or sub-agents and don't know which works best, always using the current favourite risks missing a better one, and trying them all equally wastes runs. A bandit policy such as epsilon-greedy or Thompson sampling balances them. In my simulation never exploring had regret growing linearly (32 at 100 runs), while adaptive policies grew slowly (8 at 100 runs). If the environment changes, for example a tool breaks, a policy that committed after an initial phase kept paying: 114 against 83 for UCB over 1,000 runs." Follow-up: "What do you need to use it?" A per-run success signal, enough volume (at 20 runs there is little to learn), and a reason to keep a small exploration share.
Why does majority voting fail sometimes?
"Because it assumes errors are independent. If a task has a tempting wrong answer, many samples converge on it and the vote amplifies the mistake: in my test the same N=15 gave 0.72 when wrong answers scattered and 0.58 when 90 percent shared one. A verifier does not have that problem, so it stayed near 0.82, but it has its own error: a verifier that approves everything gives no gain." Follow-up: "How would you detect it?" Measure agreement on a labelled set and check whether high-agreement answers are actually right (calibration).
Common mistakes
- Sampling N times and then picking the first.
- Majority-voting free-form answers that cannot be compared.
- Using the same sample settings every time, so the samples are copies.
- Trusting an agent's own value or priority estimates without calibration.
- Never exploring, so the first strategy that worked is used forever.
- Exploring with no outcome signal.
- Ignoring cost: reporting the accuracy gain without calls, tokens or latency.
- Committing once and never rechecking after a tool or model change.
What the example does not do
- No real attempts or answers. A sample is a draw with an assumed chance of success.
- Independent verifier errors across candidates; a real verifier has correlated errors.
- Stationary value and cost. Real task values and costs change as the agent works.
- No dependencies or deadlines in the queue.
- Standard bandit algorithms without tuning, and no contextual information (which strategy suits which task type).
- No tree search. Expanding and pruning branches (beam search, Monte Carlo tree search, Tree of Thoughts) is mentioned but not implemented.
- Optional real-model experiment (I did not run this): take 100 tasks with a checkable answer, draw 15 samples each, and compute best-of-N with a test-based verifier and majority vote for N from 1 to 15.
Check yourself
Five attempts, and the wrong answers mostly agree with each other. Does a majority vote help? (Not much. Voting needs wrong answers to scatter. When half the wrong answers were the same, N = 15 reached only 0.65; a verifier checking each of 15 candidates reached 0.82.)
For the curious Noisy estimates can make a smarter rule worse
Ordering tasks by value per cost reached 99 percent of the best possible value when the estimates were exact, but with 100 percent estimate error it fell to 0.84, below the 0.88 of simply taking the cheapest first. If an agent scores its own tasks, calibrate its estimates before trusting them. A zero-cost task used to crash the ratio rule; it is now handled and tested.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| A deterministic verifier | Tests or schema checks, not another opinion. |
| Value per cost under a budget | With deadlines and dependencies as extra constraints. |
| Try alternatives on purpose | A strategy that never explores keeps losing when the best option changes. |
Draft: confirm how many samples your budget allows per task.
What you must be able to say (one page)
- Three questions look alike but need different tools: how many attempts at one task (repeated sampling), which task or branch next (prioritisation), and which strategy to run next (exploration against exploitation).
- Repeated sampling (best-of-N, majority vote, self-consistency) is test-time compute: more attempts, more cost, diminishing returns. Two outputs cost about twice as much as one, and diversity in the candidates raises the chance that one is good. Self-consistency draws several reasoning paths and takes a majority vote; external verifiers and iterative generation with verifier feedback are the other common forms.
- How you choose among attempts matters more than how many you make:
- best-of-N with a verifier rose from 0.55 (one sample) to 0.73 at N=3, 0.78 at 5, 0.82 at 15, at twice the calls because every candidate is checked;
- majority vote rose only to 0.63 at N=5 and then flattened at 0.65, because wrong answers can agree: when half the wrong answers were the same attractor, N=15 gave 0.65; when 90 percent were, 0.58; when they scattered, 0.72.
- A verifier is only as good as its false-approval rate: best-of-5 fell from 0.83 (never approves a wrong answer) to 0.77, 0.69, 0.63 and 0.54 (approves everything, no better than one sample).
- Prioritisation under a budget: ordering by value per cost reached 99 percent of the best possible value; arrival order reached 58 percent; highest value first 86 percent. But when the agent's value estimates are noisy, ranking by value hurts: at 100 percent estimate error value-per-cost fell to 0.84, below the 0.88 of simply taking the cheapest first.
- Exploration: a policy that never tries alternatives has regret that grows linearly (32 at 100 runs, 160 at 500); adaptive policies (epsilon-greedy, Thompson sampling, explore-then-commit) grow far more slowly (8 at 100, 24 to 27 at 500). If the best strategy changes mid-run, committing forever is fragile (regret 114 against 83 to 102 for UCB, Thompson and epsilon-greedy over 1,000 runs).
- Honest limits: all rates are assumptions; tree search, Monte Carlo tree search and Tree of Thoughts are described from general knowledge and not tested here.