CSE Roadmap

Specialist tier · about 4 hours · 12 parts

Part 9 of 12 · 2 min read

Mistakes that waste extra compute

Common mistakes, and what the example does not do.

Mistakes that waste extra compute

  • Sampling N times and then picking the first.
  • Majority-voting free-form answers that cannot be compared.
  • Using the same sample settings every time, so the samples are copies.
  • Trusting an agent's own value or priority estimates without calibration.
  • Never exploring, so the first strategy that worked is used forever.
  • Exploring with no outcome signal.
  • Ignoring cost: reporting the accuracy gain without calls, tokens or latency.
  • Committing once and never rechecking after a tool or model change.

What the example does not do

About this example What the example does not do
  • No real attempts or answers. A sample is a draw with an assumed chance of success.
  • Independent verifier errors across candidates; a real verifier has correlated errors.
  • Stationary value and cost. Real task values and costs change as the agent works.
  • No dependencies or deadlines in the queue.
  • Standard bandit algorithms without tuning, and no contextual information (which strategy suits which task type).
  • No tree search. Expanding and pruning branches (beam search, Monte Carlo tree search, Tree of Thoughts) is mentioned but not implemented.
  • Optional real-model experiment: take 100 tasks with a checkable answer, draw 15 samples each, and compute best-of-N with a test-based verifier and majority vote for N from 1 to 15.

Check yourself

Five attempts, and the wrong answers mostly agree with each other. Does a majority vote help?

Show answer

Not much. Voting needs wrong answers to scatter. When half the wrong answers were the same, N = 15 reached only 0.65; a verifier checking each of 15 candidates reached 0.82.

For the curious Noisy estimates can make a smarter rule worse

Ordering tasks by value per cost reached 99 percent of the best possible value when the estimates were exact, but with 100 percent estimate error it fell to 0.84, below the 0.88 of simply taking the cheapest first. If an agent scores its own tasks, calibrate its estimates before trusting them. A zero-cost task used to crash the ratio rule; it is now handled and tested.

What you will use daily at work

Practice What it looks like
A deterministic verifier Tests or schema checks, not another opinion.
Value per cost under a budget With deadlines and dependencies as extra constraints.
Try alternatives on purpose A strategy that never explores keeps losing when the best option changes.