Part 4 of 6 · 2 min read
Evaluator-optimiser and choosing
A draft, a judge and a limited number of retries, then a rule for picking the cheapest pattern that works.
Evaluator-optimiser
One call writes a draft, a check judges it, and feedback goes back for another try, up to a maximum number of rounds.
final, history, passed = refine(generate, evaluate, "DRAFT\nreply to the refund ticket", max_rounds=5)
The imitation drafter fixes one problem per round. The first draft has three problems (no greeting, a promise, no order number), and the verdicts shrink from three problems to two, to one, and then PASS on round 4. So it uses 4 draft calls. The evaluator here is plain code. If it is a model call too, count double.
- Use when you have a clear way to judge quality, ideally one you can compute, and a draft that visibly improves with feedback.
- Fails by never converging, or by an evaluator that is too lenient or too strict. Always return the last draft and say that it did not pass (the function returns
passed=False). A model that grades its own work tends to be lenient, so a separate judge prompt, or code, is more robust.
Pick the cheapest pattern that works
Start with a single prompt. Add a chain if one prompt is too hard, a router if inputs differ, parallel checks if latency matters, an orchestrator only when subtasks are unpredictable, and a loop only when you can judge quality. Measure after each step.
| Pattern | Calls on the example | Main failure | Guard in the code |
|---|---|---|---|
| Chain | one per step | an early mistake carried forward | gates |
| Routing | 2 | wrong route | default route, tidy the label |
| Parallel | one per task | rate limits, bad merge | errors collected, partial result kept |
| Orchestrator-workers | 1 + subtasks + 1 | huge or bad plan | max_subtasks, failures marked |
| Evaluator loop | one per round (plus the judge if it is a model) | never converges | max_rounds, passed flag |
What you will use daily at work
| Practice | What it looks like |
|---|---|
| A call counter | Every workflow reports calls, tokens and cost per run, so a change in shape shows up on a dashboard. |
| Framework graphs | Teams often express these shapes in a workflow library, which gives retries, tracing and persistence. |
| A cheap model for routing | A small model classifies; a stronger one handles the hard branch. |
| A cap on every loop | Rounds, subtasks and total calls are limited in code. |
Draft: confirm what your chosen workflow library supports before relying on this.
Interview angle
"Walk me through the workflow patterns and when you would use each." Name all five, give the failure mode and a cost note for each (chain: compounding errors; routing: misclassification; parallel: rate limits; orchestrator: unbounded fan-out; loop: non-convergence). Then say that each of those is capped or gated by code.