Part 5 of 6 · 3 min read
Exercises
Easy
Easy 1: Run it and read the table (30 minutes)
In code/mini-shapes, run pytest -q and python run_all.py.
Done when: you report accuracy, calls and tokens for single, route and orchestrate, and say why route scores 0.00 on two-domain tickets.
Hint
shapes.run(shape, params, n, seed) returns every column.
Easy 2: Draw the path of one ticket (30 minutes)
For one two-domain ticket, write the sequence of calls in each of the five shapes.
Done when: you list the calls and count the critical path for each, and check it against path in the table.
Hint
the path value counts sequential calls for one ticket.
Easy 3: Change the task mix (30 minutes)
Run all shapes at composite 0.1 and 0.9.
Done when: you report which shape is best at each and explain why the ranking changes.
Hint
replace(Params(), composite=0.9).
Medium
Medium 1: Find the critic's break-even (2 hours)
Sweep detect from 0 to 1 and false_reject from 0 to 0.4.
Done when: you plot accuracy against detection for three false-rejection rates, mark the plain orchestrator's accuracy, and report the break-even detection rate for each.
Hint
replace(P, detect=d, false_reject=f) with n=8000.
Medium 2: A cheaper critic (2 hours)
Review only the parts the worker marks as low-confidence (assume the worker's confidence separates right from wrong with a given strength).
Done when: you report accuracy, calls and tokens against the full critic and say what confidence quality you need to win.
Hint
draw a confidence value from two distributions (right and wrong) and review when it is below a threshold.
Medium 3: Hierarchy (3 hours)
Add a second level: a top orchestrator that sends each domain to a sub-orchestrator with two workers.
Done when: you report calls, path and accuracy, and say at what task size the extra level pays.
Hint
add a call level and a second decomposition probability.
Hard
Hard 1: Real agents (5 hours)
Implement the single agent, router, orchestrator and handoff shapes with a real model and 100 labelled tickets. (I did not run this, so your numbers are the ground truth.)
Done when: you report accuracy, calls, tokens, the model and the date, and say whether the ranking matches the simulation.
Hint
freeze the labels before running, and use one model for all shapes.
Hard 2: Correlated critics (4 hours)
Make the critic's errors correlated with the worker's (the critic is more likely to accept a wrong draft when the worker was confidently wrong). (I did not run this.)
Done when: you report how the critic's value falls as correlation rises, and say what you would change in a real design.
Hint
draw the critic's verdict from a distribution that depends on the worker's result.
Hard 3: Group chat (4 hours)
Add a conversation-driven shape with a speaker-selection policy and a message limit. (I did not run this.)
Done when: you report calls, accuracy and how often the limit ended the conversation, and compare it with handoff.
Hint
pick the next speaker with a policy function, and count messages.