Local or API, and will it fit: Local models

When a local model is the right choice, and how to estimate its memory before you download anything.

	Local model	Hosted API
Data	Stays on your machine	Sent to the provider (check their terms)
Quality	Depends on the model you can fit	Usually the strongest models
Cost shape	Fixed (hardware, power), cheap per extra request	Pay per token, nothing when idle
Operations	You run, update and monitor it	The provider does
Latency	No network, but limited by your hardware	Fast hardware, plus network
Availability	Works offline; you are the on-call	Provider outages, rate limits

Interview angle

"API or self-host?" Good answers are about the workload: volume, data sensitivity, quality needed, team skills, latency, and what happens at 10 times the traffic. Offer a decision rule, not a preference, and mention the hybrid: a local model for easy or private steps and a hosted one for hard steps, behind one interface.

Will it fit?

A model's memory has three parts that you can estimate before downloading anything.

Quantisation

Storing the model's weights with fewer bits (for example 8 or 4 instead of 16) to save memory and often to run faster, at some cost in quality. How much quality is lost depends on the model and method, so measure on your task.

Weights. parameters x bits / 8. A 7-billion-parameter model is about 14 GB at 16 bits, 7 GB at 8 bits and 3.5 GB at 4 bits. Real 4-bit formats usually need a little more than 4 bits per weight, because they also store scaling values, so treat these figures as a floor.

KV cache. While generating, the runtime keeps the keys and values of every earlier token for every layer (Level 1, module 2 explained attention). Its size is:

2 x layers x kv_heads x head_dim x context_length x bytes_per_value   (x batch size)

For the illustrative shape in the example (32 layers, 8 key/value heads, 128 per head, 16-bit values), 4,096 tokens need about 0.54 GB, and the figure doubles whenever the context doubles.

KV cache against context length, from the example code. The shape is illustrative.

Overhead. The runtime itself, buffers, and the rest of your system. The example assumes 1 GB, which is a guess, so leave real headroom.

The same model at three precisions with an 8 GB budget. Only the 4-bit version fits.

fits(7, "int4", layers=32, kv_heads=8, head_dim=128, context=8192, memory_gb=8)
# {'weights_gb': 3.5, 'kv_gb': 1.07, 'need_gb': 5.57, 'fits': True}

Fitting is necessary, not sufficient. Speed depends on how fast the hardware can read the weights from memory, and on whether they sit in fast graphics memory or slower system memory. The formulas cannot tell you tokens per second. Only a measurement can.

Two common surprises

A model that fits at a short context can run out of memory at a long one, because the KV cache grows with context. And serving several users at once multiplies the KV cache by the number of simultaneous sequences.

Check yourself

Does a 13-billion-parameter model fit in 8 GB at 4 bits? (Weights alone: 13 x 4 / 8 = 6.5 GB. Add about 1 GB overhead and the KV cache, and you are close to or over 8 GB. It depends on the context, so it is not safe to say yes.)