CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 2 of 12 · 2 min read

Why cost per call is the wrong number

A cheap model that fails often is expensive per success, not cheap.

Think of two repair shops. One charges $20 and fixes the problem a third of the time. The other charges $60 and fixes it almost every time. The "cheaper" shop is the worse deal, because you often pay twice.

In plain words: what matters is the cost of a task that actually gets resolved, not the cost of any one call, because a failed call still has to be redone or escalated.

A worked example. On 20,000 simulated tasks, the large model costs $0.0150 per call and succeeds 91.1 percent of the time, for $0.0165 per resolved task. The small model looks ten times cheaper per call at $0.0013, but it only succeeds 64.4 percent of the time. Its real cost per resolved task is $0.0019 — still cheaper, but nowhere near the ten-times difference the per-call price suggested.

The key lines. From code/mini-cost/costlab.py:

# from code/mini-cost/costlab.py
def call_cost(model, in_tok, out_tok, cached_in=0):
    pi, pc, po = PRICE[model]
    return ((in_tok - cached_in) * pi + cached_in * pc + out_tok * po) / 1e6

def p_success(model, d):
    return max(0.0, min(1.0, (0.97 - 1.05 * d) if model == "small" else (0.99 - 0.25 * d)))

Read it as: every call has a price and a chance of actually succeeding. Dividing total spend by the number of successes, not the number of calls, tells you the true price of getting work done. The target is the cheapest policy that still clears a quality floor. It is never the cheapest price tag alone.

Cost per resolved task

Total spend divided by the number of tasks that actually succeeded, rather than the number of calls made.

Example: the small model's $0.0013-per-call price becomes $0.0019 per resolved task once its 64.4 percent success rate is factored in.