Part 5 of 12 · 2 min read
Router vs cascade: deciding before vs checking after
A router guesses difficulty before trying; a cascade tries cheap first and checks the answer.
Think of a receptionist who guesses how serious your problem is before you've said a word. Compare that to a nurse who has you describe the symptom first and only calls in a specialist if something doesn't check out.
In plain words: a router picks a model before making the call, so a wrong guess fails with no second chance. A cascade tries the cheap model first and verifies the answer, escalating only when the check fails. That costs more in tail latency (the slowest few percent of requests) but protects quality.
A worked example. A classifier router, 85 percent accurate, costs $0.0059 per task but succeeds only 77.2 percent of the time. Misclassified hard tasks are sent to the small model and simply fail, with nothing to catch it. A cascade (small model, then a cheap verifier, then escalate on a failed check) costs $0.0070 per task at 90.9 percent success. That nearly matches the large-only model's 91.1 percent, at roughly 47 percent of the large-only cost. The price is latency: the cascade's p95 is 14.3 seconds against 10.8 seconds for large-only, because escalated tasks pay for both calls.
The key lines.
# from code/mini-cost/costlab.py
elif policy == "cascade":
s, c, l = attempt("small", d)
v_c = call_cost("small", 700, 30); c += v_c
passes = (rng.random() < verifier_tpr) if s else (rng.random() < verifier_fpr)
if not passes:
s2, c2, l2 = attempt("large", d); s, c, l = s2, c + c2, l + l2
Read it as: try the cheap model, run a cheap check on the result, and only pay for the expensive model on the fraction of tasks the check does not trust. The cascade's whole result rides on that verifier being good. A verifier that wrongly passes bad answers silently lowers quality. It does not show up as an escalation.
Router, cascade
A router picks a model before calling it; a cascade tries a cheap model first and escalates to a stronger one only if a check fails.
Example: the cascade above escalates on roughly 1 in 4 tasks, which is why 90.9 percent success costs only 47 percent of the large-only price rather than saving nothing.