CSE Roadmap

Common tier · about 5 hours · 13 parts

Part 3 of 13 · 2 min read

A vague preference becomes a break-even: the input value where the decision flips

"It depends" is not an answer until you can say exactly what it depends on and at what value the answer changes.

Think of deciding whether to buy a monthly transit pass or pay per ride. There is no universally right answer. But there is an exact number of rides per month above which the pass wins and below which paying per ride wins.

In plain words: a break-even is the value of one input at which two options tie, so that above or below it the decision flips -- turning "a multi-agent system is sometimes better" into "a multi-agent system is better once tool-heavy traffic exceeds this specific percentage," which is a claim that can actually be checked against your own measured traffic.

A worked example. Comparing a direct model against a multi-agent system on illustrative scores (9.7 vs 9.3 on simple reasoning, 6.8 vs 9.2 on tool-heavy tasks), the multi-agent system's mean score only exceeds the direct model's once tool-heavy traffic passes about 14.3% of the mix. Below that share, the direct model wins; above it, the multi-agent system does, and the gap widens sharply with share -- at 90% tool-heavy traffic the multi-agent system leads by 2.12 points, enough that even the extra 43x token cost (about $0.015 per task at $1/million tokens) is easily worth paying.

The key lines. From code/mini-tradeoff/matrix.py:

# from code/mini-tradeoff/matrix.py
def mix_breakeven(a_tool, a_simple, b_tool, b_simple):
    """Share of 'tool-heavy' traffic at which option B's mean quality equals A's: solve p*(b_tool-a_tool) = (1-p)*(a_simple-b_simple)."""
    gain, loss = b_tool - a_tool, a_simple - b_simple
    if gain == 0 and loss == 0: return None          # identical on both kinds of traffic
    if gain <= 0 < loss: return 1.0                  # B is worse (or equal) on tool-heavy traffic and worse on simple: never ahead
    if loss <= 0 < gain: return 0.0                  # B is ahead on both: ahead at every mix
    return loss / (gain + loss)

Read it as: the function reduces an entire comparison to one number, p, the exact traffic share where the tie happens. The three guard clauses before the final formula handle the cases where one option simply always or never wins. So the single returned number is only meaningful in the genuinely mixed case. That is the case where each option wins on a different kind of traffic.

Break-even

Break-even is the value of one input at which two options tie; above or below it the decision flips.

Example: "it wins above about 14 percent tool-heavy" is a decision sentence with a number you can actually place your own measured traffic against -- which is exactly the four-slot decision sentence this module asks you to produce, built from one break-even calculation.