CSE Roadmap

Common tier · about 4 hours · 11 parts

Part 9 of 11 · 1 min read

Quick notes

  • LoRA in one line: freeze W, learn small B and A, use W + (alpha / r) B A. Only A and B train. B starts at zero. R is the rank.
  • It is cheap because the adapter is tiny, and it works when the change a task needs is low-rank. A rank below the true one underfits. When the change is full rank, LoRA trails full tuning.
  • In practice: a small rank on all attention (and often MLP) matrices, picked from the validation curve. alpha and learning rate move together.
  • Merge B A into W for no extra latency, or keep adapters separate to serve many fine-tunes from one base.
  • QLoRA stores the frozen base in 4-bit NF4 and trains LoRA on top. Memory is mostly the base, so quantising the base is what cuts it.