Part 10 of 12 · 1 min read
Quick notes
- Measure cost per resolved task, not per call or per token, and pick the cheapest policy above a quality floor; a cheap model that fails is expensive per success.
- Agents cost more than their step count suggests, because each step re-sends the history. Fewer steps, shorter context and a stable cached prefix are the first savings.
- Semantic caches buy hits with wrong answers; never semantic-cache anything user-specific, time-sensitive or high-stakes.
- Route by cascade rather than by classifier: small model first, a cheap verifier, escalate on a failed check. The verifier's quality drives the result.
- Per-task budget caps bound the tail, and must end the run in a defined state. Latency is a separate budget: TTFT plus TPOT times output tokens.