Part 5 of 11 · 2 min read
What the toy experiment measured
The numbers behind the claims above, in four tables.
The tables below are the numbers quoted in the sections above. Lower error is better. 0.24 means "no better than the untuned base" and 0.001 means "almost perfect" on this toy.
How to read the first table: compare the LoRA rows with the "Full tuning of W2" row, once in each column. Notice that LoRA r=4 beats full tuning when the needed change is simple (0.0003 against 0.0061), but when the change needs full rank LoRA only gets close at r=32 or above (0.0059 against 0.0048).
The detail Test error by rank, both cases
| Test error (normalised MSE) | Params | Needed change rank 4 | Needed change full rank |
|---|---|---|---|
| Untuned base | 0 | 0.2378 | 0.1581 |
| Full tuning of W2 | 4,096 | 0.0061 | 0.0048 |
| Full tuning of all layers | 5,376 | 0.0102 | 0.0086 |
| LoRA r=1 | 128 | 0.0273 | 0.0767 |
| LoRA r=2 | 256 | 0.0121 | 0.0368 |
| LoRA r=4 | 512 | 0.0003 | 0.0135 |
| LoRA r=8 | 1,024 | 0.0009 | 0.0112 |
| LoRA r=16 | 2,048 | 0.0016 | 0.0069 |
| LoRA r=32 | 4,096 | 0.0027 | 0.0059 |
| LoRA r=64 | 8,192 | 0.0032 | 0.0057 |
| Validation error, rank 8 | lr 0.001 | lr 0.003 | lr 0.01 | lr 0.03 | lr 0.1 |
|---|---|---|---|---|---|
| scale alpha/r = 0.25 | 0.0118 | 0.0049 | 0.0015 | 0.0005 | 0.0005 |
| scale = 1 | 0.0067 | 0.0024 | 0.0011 | 0.0008 | 0.0013 |
| scale = 4 | 0.0074 | 0.0033 | 0.0014 | 0.0023 | 0.633 |
The second table is the alpha lesson. How to read it: find the smallest number in each row. That is the best learning rate for that scale. Notice that it moves left as the scale goes up (lr 0.03 to 0.1 at scale 0.25, lr 0.03 at scale 1, lr 0.01 at scale 4). At scale 4 with lr 0.1 the run diverged (0.633). alpha and learning rate are not independent. Tune one with the other fixed.
How to read the third table: rows are block sizes, from small to one block for everything. The last column is what each weight costs in storage. Notice that smaller blocks give lower error but cost more bits. The normal-shaped levels help most when blocks are large (0.126 against 0.182 for the whole matrix), but not at all at block 16.
| Round-trip error / weight spread | Uniform 16 levels | Normal-quantile 16 levels | Bits per weight |
|---|---|---|---|
| Block 16 | 0.080 | 0.081 | 5.00 |
| Block 64 | 0.101 | 0.092 | 4.25 |
| Block 256 | 0.118 | 0.098 | 4.06 |
| Block 4096 | 0.147 | 0.109 | 4.00 |
| Whole matrix | 0.182 | 0.126 | 4.00 |
How to read the fourth table: the top three rows are before any training and the bottom three after LoRA. Notice that 4-bit storage costs a little before training (0.25 to 0.26 against 0.24), and about eight times the error after it (0.007 to 0.009 against 0.0009). LoRA recovers the task but not all of the rounding damage.
| Toy QLoRA (rank-4 change, LoRA r=8) | Test error |
|---|---|
| 16-bit base, untuned | 0.2378 |
| Uniform 4-bit, untuned | 0.2639 |
| Normal-quantile 4-bit, untuned | 0.2541 |
| 16-bit base + LoRA | 0.0009 |
| Uniform 4-bit + LoRA | 0.0088 |
| Normal-quantile 4-bit + LoRA | 0.0069 |