CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 3 of 12 · 2 min read

Do the break-even before committing

A tuned model that works can still be a financial loss if volume never clears the up-front and recurring cost of maintaining it.

Think of buying a commercial espresso machine for a coffee cart. It makes better coffee per cup. But if you only sell twenty cups a day, the machine never pays for itself before it needs replacing.

In plain words: fine-tuning has an up-front cost, ongoing upkeep, and a recurring re-tune cost every time a better base model arrives. The saving per request only clears all of that above a certain volume.

A worked example. At 20,000 requests a month, tuning never pays back. The monthly saving doesn't even cover upkeep. At 50,000 it still never pays back, because the periodic re-tune cost eats the saving faster than it accumulates. At 100,000 requests it pays back in 8 months; at 300,000, in 2 months; at 1,000,000, in 1 month. For a 12-month payback target, the break-even volume is about 75,300 requests a month in the base case. Accounting for a 2-point quality loss pushes that to about 158,600, and a re-tune every 3 months instead of 6 pushes it to about 105,600.

The key lines. From code/mini-finetune/economics.py:

# from code/mini-finetune/economics.py
def payback_months(p):
    """Months until cumulative saving covers the one-time cost, including refreshes. None if it never does."""
    _check(p)
    cum = -p.one_time_cost
    for m in range(1, 121):
        cum += monthly_saving(p)
        if m % int(p.base_refresh_months) == 0:
            cum -= p.refresh_cost
        if cum >= 0:
            return m
    return None

Read it as: the refresh cost is subtracted on a schedule, inside the same loop that accumulates the saving. A break-even calculation that leaves out the refresh line would show a shorter payback than what actually happens once a new base model shows up.

Break-even, base-model refresh

Break-even volume is the request rate at which cumulative savings cover the up-front and recurring cost of tuning within a chosen horizon. A refresh is re-doing the tuning on a newer base model. A recurring cost, not a one-time one.

Example: the jump from 75,300 to 105,600 requests, when the refresh interval shortens from 6 to 3 months, shows how much of the break-even volume exists purely to cover a cost that recurs on a schedule. It is not a cost paid once.