Part 4 of 6 · 3 min read
API or self-hosted, and monitoring
Renting a model by the token versus running one yourself, and why the decision is never final.
| Hosted API | Open-weight model you run | |
|---|---|---|
| Start-up effort | Minutes | Days to weeks |
| Cost shape | Pay per token | Pay for servers, even when idle |
| Control and privacy | Provider's terms | Full control |
| Latency tuning | Limited (a hard attribute) | You can optimise (a soft attribute) |
| Best frontier quality | Usually here | Usually a step behind |
| Staff needed | Little | Someone who can operate GPUs |
This table is my summary of common trade-offs, not a measurement. Most teams start with an API and move parts of the workload to self-hosted models only when volume, privacy or cost clearly justifies the operating effort.
Step 4: keep checking
Models behind an API name are updated, deprecated and sometimes quietly changed. Keep your evaluation set, rerun it on a schedule and whenever the provider announces a change, and log real outcomes (user thumbs-down, tool errors, retries). Treat the model name as a configuration value that you can change in one place, not a constant buried in code (Level 0, module 6).
For the curious Cost per correct answer
A model that is cheaper per call can be dearer per success if it is wrong more often, since you pay again for retries. A rough way to compare is cost divided by accuracy. For the four simulated models that gives about $25.53, $5.52, $0.49 and $0.24 per 1,000 correct answers. It ignores the cost of a wrong answer reaching a user, which for many tasks dominates.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| An evaluation set in the repository | A versioned file of cases and expected results that runs in CI against the current model. |
| A model name in configuration | One setting (Level 0, module 6) changes the model everywhere. |
| Side-by-side runs | Before switching, you run old and new on the same cases and compare accuracy, cost and latency with an interval. |
| Provider change tracking | You watch deprecation notices and rerun the set when a model is updated. |
| Cost dashboards | You track cost per task, not just per call, and alert when it rises. |
| A fallback model | If the primary model is down or rate-limited, a second one takes over (Level 11). |
Draft: to be checked against current provider documentation (deprecation policies, version pinning) before this page goes live.
Interview angle
Answer in your head first, then open it.
How do you choose a model for a new feature?
Filter on hard constraints, shortlist from public information, decide on my own test set with confidence intervals and cost, and monitor afterwards.
Why not just use the top of a leaderboard?
It measures other tasks, may be contaminated, and ignores cost, latency and my constraints.
What is a Pareto front?
The models for which no alternative is both cheaper and better.
Two models score 81% and 83% on 100 cases. Which is better?
I cannot tell: the intervals overlap heavily. I would collect more cases or decide on cost.
When would you self-host?
High and steady volume, strict data control, or a need to tune latency and fine-tune, and when we have the staff to operate it.