CSE Roadmap

Common tier · about 4 hours · 11 parts

Part 5 of 11 · 2 min read

What the toy experiment measured

The numbers behind the claims above, in four tables.

The tables below are the numbers quoted in the sections above. Lower error is better. 0.24 means "no better than the untuned base" and 0.001 means "almost perfect" on this toy.

How to read the first table: compare the LoRA rows with the "Full tuning of W2" row, once in each column. Notice that LoRA r=4 beats full tuning when the needed change is simple (0.0003 against 0.0061), but when the change needs full rank LoRA only gets close at r=32 or above (0.0059 against 0.0048).

The detail Test error by rank, both cases
Test error (normalised MSE) Params Needed change rank 4 Needed change full rank
Untuned base 0 0.2378 0.1581
Full tuning of W2 4,096 0.0061 0.0048
Full tuning of all layers 5,376 0.0102 0.0086
LoRA r=1 128 0.0273 0.0767
LoRA r=2 256 0.0121 0.0368
LoRA r=4 512 0.0003 0.0135
LoRA r=8 1,024 0.0009 0.0112
LoRA r=16 2,048 0.0016 0.0069
LoRA r=32 4,096 0.0027 0.0059
LoRA r=64 8,192 0.0032 0.0057
Validation error, rank 8 lr 0.001 lr 0.003 lr 0.01 lr 0.03 lr 0.1
scale alpha/r = 0.25 0.0118 0.0049 0.0015 0.0005 0.0005
scale = 1 0.0067 0.0024 0.0011 0.0008 0.0013
scale = 4 0.0074 0.0033 0.0014 0.0023 0.633

The second table is the alpha lesson. How to read it: find the smallest number in each row. That is the best learning rate for that scale. Notice that it moves left as the scale goes up (lr 0.03 to 0.1 at scale 0.25, lr 0.03 at scale 1, lr 0.01 at scale 4). At scale 4 with lr 0.1 the run diverged (0.633). alpha and learning rate are not independent. Tune one with the other fixed.

How to read the third table: rows are block sizes, from small to one block for everything. The last column is what each weight costs in storage. Notice that smaller blocks give lower error but cost more bits. The normal-shaped levels help most when blocks are large (0.126 against 0.182 for the whole matrix), but not at all at block 16.

Round-trip error / weight spread Uniform 16 levels Normal-quantile 16 levels Bits per weight
Block 16 0.080 0.081 5.00
Block 64 0.101 0.092 4.25
Block 256 0.118 0.098 4.06
Block 4096 0.147 0.109 4.00
Whole matrix 0.182 0.126 4.00

How to read the fourth table: the top three rows are before any training and the bottom three after LoRA. Notice that 4-bit storage costs a little before training (0.25 to 0.26 against 0.24), and about eight times the error after it (0.007 to 0.009 against 0.0009). LoRA recovers the task but not all of the rounding damage.

Toy QLoRA (rank-4 change, LoRA r=8) Test error
16-bit base, untuned 0.2378
Uniform 4-bit, untuned 0.2639
Normal-quantile 4-bit, untuned 0.2541
16-bit base + LoRA 0.0009
Uniform 4-bit + LoRA 0.0088
Normal-quantile 4-bit + LoRA 0.0069