Part 7 of 12 · 1 min read
How to build it so it holds up
- Write down the seven axes and which ones are hard constraints (privacy, region, on-device) before comparing models.
- Read each licence for commercial use, user-count limits and output-use restrictions.
- Size memory as weights plus cache, then check decode speed with a bandwidth bound before renting hardware.
- Compute cost at realistic utilisation, not full load, and include engineering and on-call time.
- Measure the small model on your own traffic before designing a router; the gain depends on it.
- Make routers and confidence checks cheap and measure their own accuracy.
- Keep an escape hatch: log escalations and sample the cheap path for review.
- Use a paired comparison on a fixed evaluation set and plan the item count for the gap you care about.
- Pin versions of open weights and record them with every evaluation.