Part 7 of 12 · 1 min read
How to build it so it holds up
- Track cost per resolved task with traces (Level 9), split by step, model and tenant.
- Set a quality floor first, then minimise cost above it.
- Cap every run (dollars, steps, wall time) and define what happens at the cap.
- Cache the stable prefix: system prompt and tools first, volatile data last; measure hit rate.
- Use exact caches for deterministic lookups; treat semantic caches as risky and keep thresholds high, scoped per tenant and excluded for sensitive queries.
- Prefer cascades over blind routers where you can verify answers cheaply; evaluate the verifier.
- Compress context: summarise, truncate tool output, drop stale steps (Level 7).
- Batch non-urgent work and keep interactive traffic separate.
- Budget latency too: TTFT, time to first progress event (module 4), total.
- Re-evaluate routing when models or prices change; keep the policy in config, not code.