CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 4 of 12 · 2 min read

A critic only helps if it's more right about the draft than wrong

Adding a reviewer only pays off when the reviewer catches more real mistakes than it creates by rejecting good work.

Think of a proofreader who occasionally flags a perfectly correct sentence as wrong: if they do that often enough, the extra editing cycles they cause can outweigh the real mistakes they genuinely catch.

In plain words: a critic is an agent that evaluates another agent's output and accepts or rejects it, adding a revision loop that costs extra calls. Whether that loop helps or hurts depends entirely on two competing rates: how often the critic actually detects a wrong draft, against how often it wrongly rejects a draft that was already correct.

A worked example. A critic detecting 70% of wrong drafts while wrongly rejecting only 10% of correct ones raises the orchestrator's accuracy from 0.71 to 0.80. That is a real gain, for about 1.8 more calls per ticket. But a critic that detects nothing (0% of wrong drafts) while wrongly rejecting 30% of correct drafts lowers accuracy to 0.65. That is strictly worse than not having a critic at all. Every rejection triggers a fresh attempt that has its own chance of being wrong, with none of the benefit of actually catching a real error.

The key lines. From code/mini-shapes/shapes.py:

# from code/mini-shapes/shapes.py
calls += 1; tokens += p.system + p.ticket + p.draft
rejected = (r.random() < p.detect) if not good else (r.random() < p.false_reject)
if not rejected:
    break

Read it as: which probability governs rejection depends on whether the draft (good) was actually right. p.detect only ever applies to genuinely wrong drafts, and p.false_reject only to genuinely correct ones. A critic's net value is the balance of how often each branch fires, not a single "critic quality" number.

Critic

A critic is an agent that evaluates another agent's output and accepts or rejects it.

Example: "detects 0, rejects 30%" producing worse accuracy than no critic at all (0.65 against 0.71) is the critic definition's dark side made concrete. A critic that merely re-judges work without actually detecting errors is pure added risk, not added safety.