CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 6 of 12 · 3 min read

What the example found

Accuracy, calls and tokens for all five shapes, accuracy as task mix shifts, critic quality, and handoff under stress.

A rough ordering of the shapes by how much the developer controls the path: sequential, conditional and parallel workflows give the most control and the least autonomy. Plan-based orchestration gives medium autonomy and still high control. Handoff gives medium autonomy and medium control for little complexity. An open conversation between agents gives the most autonomy and the least control. A common rule is to use workflows for well-defined repeatable processes, plan-based orchestration where planning is complex, handoff where domain expertise decides, and conversation for exploratory work. Production systems are usually hybrids: workflows for the predictable parts, and autonomous patterns where flexibility matters.

Setup: tickets need one or two of billing, tech and legal; a ticket is correct only if every needed part is correct. 20,000 tickets per shape, seed 0.

How to read the main results table: each row is one of the five shapes, run on the same mixed ticket stream (30% two-domain).

Shape Accuracy Single-domain Two-domain Calls Tokens Critical path
single agent 0.63 0.69 0.48 1.0 1,300 1.0
route 0.53 0.76 0.00 2.0 2,270 2.0
orchestrator-workers 0.71 0.76 0.59 3.2 3,459 2.9
handoff 0.74 0.81 0.58 2.3 2,405 2.3
orchestrator plus critic 0.80 0.84 0.71 5.0 5,671 4.4

What to notice: orchestrator plus critic wins on accuracy but costs 5 calls per ticket against the single agent's 1. That is five times the calls for roughly 17 points of accuracy. It is the kind of trade-off a design review has to put an actual value on, not just admire.

How to read the task-mix table: each column is a different share of two-domain tickets, from none to all.

Share 0.0 0.3 0.6 1.0
single 0.70 0.63 0.57 0.49
route 0.76 0.53 0.30 0.00
orchestrator-workers 0.76 0.71 0.66 0.59
handoff 0.81 0.74 0.66 0.57
orchestrator plus critic 0.84 0.80 0.77 0.72

What to notice: route's accuracy falls off a cliff as the two-domain share rises (0.76 to 0.00), while every other shape degrades gradually. Route isn't just worse on average. It's the only shape whose failure mode is total rather than partial.

How to read the critic-quality table: rows are the critic's detection rate for wrong drafts; columns are its false-rejection rate for correct drafts.

Detects wrong drafts Rejects correct drafts 0% 10% 30% Calls
0.0 0.71 0.69 0.65 4.4 to 5.2
0.5 0.78 0.77 0.74 4.6 to 5.5
0.9 0.83 0.84 0.82 4.8 to 5.8

What to notice: moving along the detection axis (top row to bottom) always helps, but moving along the false-rejection axis (left to right) always hurts. At zero detection, rising false-rejection actively makes things worse than no critic at all (0.71 baseline falling to 0.65).

How to read the handoff-under-stress line: each entry is a different hop limit, under stressed conditions (specialists notice a misroute 90% of the time, confusion rate 0.7).

Hop limit 2: cuts off 3.3% of tickets, scores 0.737. Hop limit 4: cuts off 0.2%, scores 0.762. Hop limit 8: cuts off none. Genuine loops (revisiting an agent) happened in at most 0.6% of tickets.

What to notice: doubling the hop limit from 2 to 4 both raises the score and cuts the cutoff rate by more than 15x (3.3% to 0.2%). A tight hop limit's cost is almost entirely paid in cut-off legitimate handoffs, not in protecting against genuine loops, which stayed rare throughout.

{{svg:shapes|The five shapes and the failure each is known for.}} {{svg:results|Accuracy and tokens per shape; accuracy as the share of two-domain tickets rises; accuracy of the critic loop by critic quality.}}