Part 8 of 12 · 2 min read
Answering small-and-open-model questions in an interview
Would you self-host a model or use an API?
"I start from hard constraints: data that cannot leave, regional rules, on-device needs. If none apply it is cost, quality and effort. In the example's toy an 8B model cost $0.55 per million tokens at full load and batch 32 but $59 at 10 percent load at batch 1, against an assumed $6 API price, so utilisation decides. Adding engineering time moved break-even from 304 million to 1.5 billion tokens a month." Follow-up: "What would change your mind?" Needing the strongest model, function calling or structured outputs out of the box, or low traffic.
How would you route between a small and a large model?
"Measure the small model on real traffic first. Then either a cheap difficulty or intent classifier before the call, or a cascade that escalates on a confidence signal. In the example's simulation a cascade with a good signal came within 2 points of the large model at 16 against 20, a useless signal cost 20.2, and a stronger small model cut it to 4.7. Random mixing only moves along the straight line." Follow-up: "How do you know the router is not hurting?" Sample cheap-path answers for review and track accuracy by route.
Will a 13B model run on my 24 GB GPU, and how fast?
"In 16-bit the weights are 26 GB, so no; in 8-bit they are 13 GB and fit with room for the cache. Decode is bandwidth-bound: bandwidth divided by weight bytes gives an upper bound, about 77 tokens per second for 8-bit at an assumed 1 TB/s, before the cache and overhead." Follow-up: "What else limits you?" The KV cache and batch size (module 6).
How many evaluation items do you need to switch to a cheaper model?
"It depends on the gap I must detect. In the example's simulation a 2-point difference needed about 3,650 items with a paired test, and 300 items gave only 13 percent power. If the cheaper model is only worse by 10 points, a few hundred are enough; for 2 points I need thousands, or I accept a pre-agreed margin and test non-inferiority." Follow-up: "Why paired?" Both models see the same items, so only the disagreements carry information.