CSE Roadmap

Specialist tier · about 5 hours · 12 parts

Part 4 of 12 · 2 min read

A cascade is only as good as its confidence signal

Escalating to a bigger model "when unsure" only saves money if the unsure-ness is actually correlated with being wrong.

Think of a junior employee who is told to flag anything they are not sure about for a manager's review. This only saves the manager time if the junior's sense of "not sure" actually tracks their real error rate. It does not help if it just tracks how they happen to feel that day.

In plain words: a cascade calls the cheap model first and escalates to the expensive one only when a confidence signal suggests the cheap answer might be wrong — but if that signal carries no real information about actual correctness, the cascade pays for both calls on many requests and ends up worse than just always using the large model.

A worked example. With a confidence signal that carries real information (signal strength 4), a cascade can match the large model's accuracy within 2 points at a cost of 13.1 — 34 percent cheaper than always using the large model (cost 20). With no confidence signal at all, the same cascade costs 20.2 — slightly worse than always using the large model, because the small model's call gets paid for and then essentially ignored every time it escalates anyway. Random mixing (escalating a fixed fraction of requests regardless of confidence) only ever traces a straight line between the small and large model's cost and accuracy — it can't beat that line the way an informative confidence signal can.

The key lines. From code/mini-small/routing.py:

# from code/mini-small/routing.py
def cascade(cs, cl, delta, threshold, c_small=1.0, c_large=20.0, c_verify=0.0, seed=4):
    """Call small first; escalate when a noisy confidence score falls below threshold.

    score ~ N(+delta/2, 1) if the small answer is correct, N(-delta/2, 1) otherwise.
    delta = 0 means the confidence carries no information.
    """

Read it as: delta is the parameter that controls how much the confidence score actually differs between correct and incorrect answers. Setting delta = 0 directly simulates a useless signal. That is exactly the case that makes the cascade cost more than doing nothing at all.

Router, cascade

A router picks which model or tool handles a request, often a small intent classifier making the decision before any model call happens. A cascade calls a cheap model first and escalates to a stronger one only when a confidence signal says the first answer is unreliable.

Example: the 20.2-vs-20-cost comparison above (useless signal vs. no cascade at all) is the clearest possible demonstration that a router or cascade is a net cost, not a net saving, until its signal is actually shown to carry information about correctness.