Part 9 of 11 · 1 min read
Quick notes
- LoRA in one line: freeze W, learn small B and A, use W + (alpha / r) B A. Only A and B train. B starts at zero. R is the rank.
- It is cheap because the adapter is tiny, and it works when the change a task needs is low-rank. A rank below the true one underfits. When the change is full rank, LoRA trails full tuning.
- In practice: a small rank on all attention (and often MLP) matrices, picked from the validation curve. alpha and learning rate move together.
- Merge B A into W for no extra latency, or keep adapters separate to serve many fine-tunes from one base.
- QLoRA stores the frozen base in 4-bit NF4 and trains LoRA on top. Memory is mostly the base, so quantising the base is what cuts it.