Where the memory really goes, and how 4-bit storage attacks it.
Here is the catch with everything above. The adapter is tiny, but the frozen base is not. A 13-billion-parameter model stored in 16-bit numbers takes 13 billion x 2 bytes = 26 GB of GPU memory before training starts. Its adapter is small by comparison, and the size depends on which matrices you attach it to. Rank 8 on all four attention matrices in each of the 40 layers is 2 x 5120 x 8 x 4 x 40 = about 13 million numbers, about 26 MB in 16-bit. Rank 2 on the same matrices is about 6.5 MB. Either way, shrinking the adapter further changes nothing. the base dominates memory, so the base is what QLoRA shrinks.
Think of rounding prices. A shop that rounds every price to the nearest rupee loses a few paise on each item but needs far fewer digits on every label. Quantisation rounds every weight to one of a small set of allowed values, so each weight needs fewer bits.
A worked example. Take four weights from one matrix: 0.31, -0.12, 0.05, 0.47. With 4 bits there are only 16 allowed values. To use them well, first divide the group by its largest absolute value, 0.47, so the numbers fit between -1 and 1: 0.66, -0.26, 0.11, 1.00. Then round each to the nearest of 16 evenly spaced levels: 0.60, -0.20, 0.07, 1.00. To use the weights later, multiply back by 0.47: 0.28, -0.09, 0.03, 0.47. The errors are 0.03, 0.03, 0.02 and 0. Each weight now costs 4 bits instead of 16, plus a share of the one stored scale (0.47).
Quantisation, block and absmax
Quantisation stores numbers with fewer bits by rounding them to a small set of levels. A block is a group of weights that share one scale. Absmax scaling divides the block by its largest absolute value so everything fits in [-1, 1].
Example: with blocks of 64 weights and a 16-bit scale per block, each weight costs 4 + 16 / 64 = 4.25 bits. Blocks of 16 cost 5 bits but round more accurately, because one unusually large weight only squashes 15 neighbours instead of 63.
NF4 (NormalFloat-4)
A 4-bit format whose 16 levels are packed closer together near zero and spread out at the edges, matching the fact that trained weights cluster near zero. Fewer weights land far from their nearest level.
Example: in the toy quantiser, normal-shaped levels cut the round-trip error from 0.101 to 0.092 at block size 64, and from 0.182 to 0.126 when one block covered the whole matrix.
QLoRA
LoRA on top of a frozen base stored in 4-bit NF4. The base is de-quantised to 16-bit on the fly for each computation, and only the adapter (kept in 16-bit) trains.
Example: at 4.25 bits per weight, the 26 GB base above becomes about 6.9 GB, roughly 3.8 times smaller, which is why a 13B model fits a single consumer GPU for tuning. The QLoRA paper reports tuning a 65B model on one 48 GB GPU.
The price is rounding error in the base. In the toy, LoRA on a 4-bit base reached 0.0069 to 0.0088 test error against 0.0009 on a 16-bit base: most of the task was recovered, not all of the quantisation damage. The QLoRA authors report parity on their benchmarks. Check on your own evaluation.
Blockwise 4-bit error, toy QLoRA results, and alpha/r against learning rate.