Part 8 of 12 · 1 min read
Answering cost and routing questions in an interview
How do you keep cost per resolved task under control?
"Measure it per task from traces, cap each run, cache the stable prompt prefix, trim context and steps, then route: cheap model first with a verifier and escalation. In the example's simulation a cascade got about the same success as the large model, 91 percent, at roughly 47 percent of the cost. A per-run cap turned a $12 runaway into $0.48." Follow-up: "What do you watch for?" The quality floor and the p95 latency, because escalation adds both cost and delay for hard tasks.
Router or cascade?
"A router decides before the call, so a misclassified hard task fails with no recovery: mine was cheap but only 77 percent successful. A cascade checks the cheap answer and escalates, so quality stays near the large model. The price is extra latency on escalated tasks, plus the need for a good verifier." Follow-up: "What if verification is expensive?" Then the cascade's saving shrinks. Use a router with a conservative threshold or sample-based verification instead.
When is prompt caching worth it?
"When a long prefix repeats: system prompts, tool definitions, documents, multi-turn history. For agents, which resend history each step, savings grow with run length. In the example's model the cost of a 40-step run dropped from $2.45 to $0.45." Follow-up: "How do you make it hit?" Keep the prefix byte-identical and put changing content at the end.
Would you use a semantic cache?
"Rarely for agents. I would use exact caches and prefix caches first." Follow-up: "Where would it be safe?" Public FAQ-style answers with a high threshold and a human-reviewed cache.