Part 2 of 12 · 2 min read
Sizing a model is arithmetic before it's a decision
Whether a model fits on a given device, and how fast it can run there, is a calculation you can do before touching any hardware.
Think of checking whether a piece of furniture fits through a doorway before buying it: a tape measure settles the question in thirty seconds, no delivery truck required.
In plain words: weight size is just parameters times bytes per parameter, and decode speed has a hard upper bound of bandwidth divided by weight bytes — both numbers can be computed on paper before renting any hardware or downloading any weights.
A worked example. A 13B model needs 26 GB of memory in 16-bit precision — that only fits the 80 GB GPU in this device list, not the 32 GB laptop or the 24 GB GPU. Switching to 8-bit precision halves that to 13 GB, which now fits the laptop, the 24 GB GPU, and the 80 GB GPU as well. At 4-bit precision on a 1 TB/s GPU, that same 13B model's decode speed is bounded at roughly 154 tokens per second — an upper bound, since it ignores the KV cache and kernel overhead entirely.
The key lines. From code/mini-small/sizing.py:
# from code/mini-small/sizing.py
def fits(params_b, precision, device, headroom=0.2):
"""Weights fit if they use at most (1 - headroom) of device memory."""
mem, _ = DEVICES[device]
return weight_gb(params_b, precision) <= (1 - headroom) * mem
def decode_upper_bound_tps(params_b, precision, device):
"""Batch-1 decode tokens per second if every step reads all weights once.
Ignores the KV cache, kernel overhead and compute; it is an upper bound.
"""
_, bw = DEVICES[device]
return bw / weight_gb(params_b, precision)
Read it as: fits reserves 20 percent headroom before comparing. decode_upper_bound_tps's own docstring admits it is an upper bound, not a real measurement. Both functions are deliberately conservative placeholders for the real question. That question is whether the model fits comfortably with the KV cache and overhead also accounted for.
Open weights and open models, inference service and model API
Open weights means the trained parameters can be downloaded; an open model means both the weights and the training data are public — most "open" models are open-weight only. An inference service is the system that hosts a model and answers queries; the model API is its interface.
Example: the 26-GB-vs-13-GB sizing comparison above is independent of whether the model is open or closed — it's a property of the weights' precision and size, which is exactly why this arithmetic has to be done before deciding whether self-hosting an open model is even feasible on the hardware you have.