Part 3 of 12 · 2 min read
Self-hosting cost is mostly a question of utilisation
The same GPU, running the same model, can cost ten times more or less per token depending purely on how busy it is.
Think of owning a taxi. Parked in the driveway all day, it costs you insurance and depreciation for zero revenue. Driving fares back to back all day, the same fixed cost gets spread over far more trips.
In plain words: self-hosting's cost per token is inversely proportional to utilisation — the share of a paid GPU's capacity that's actually serving requests — so the exact same hardware can look far cheaper or far more expensive than an API depending purely on how busy it is kept, independent of anything about the model itself.
A worked example. The same 8B model on the same GPU at batch 32 costs $0.55 per million tokens at 100 percent utilisation, but $5.46 per million at only 10 percent utilisation — a 10x difference from load alone, against an assumed $6 per million API price that barely changes with your own traffic volume. Adding engineering staff cost on top moves the picture further: with no engineering cost, break-even is 304 million tokens a month (9 percent utilisation of one GPU); with $15,000 a month of engineering cost added, break-even rises to 2,804 million tokens a month, requiring 84 percent utilisation — potentially more than one GPU can even serve.
The key lines. From code/mini-small/economics.py:
# from code/mini-small/economics.py
def cost_per_million(gpu_per_hour, tps, utilisation):
"""Dollars per million generated tokens at the given utilisation (0-1)."""
if not 0 < utilisation <= 1 or tps <= 0:
raise ValueError("utilisation must be in (0, 1] and tps positive")
return gpu_per_hour / (tps * 3600 * utilisation) * 1e6
Read it as: utilisation sits in the denominator alongside throughput. You pay the same gpu_per_hour whether the GPU is busy or idle. So halving utilisation doubles the effective cost per token, exactly as directly as halving throughput would.
Utilisation, break-even volume
Utilisation is the share of a paid GPU's capacity that's actually serving requests; self-hosting cost per token is inversely proportional to it. Break-even volume is the monthly token count above which a fixed self-hosting cost is lower than the API's per-token bill.
Example: the 304-million-to-2,804-million jump in break-even volume above, purely from adding engineering staff cost, shows that break-even isn't just a hardware question — the fixed cost side of the equation includes people, and people are often the larger fixed cost.