CSE Roadmap

Common tier · about 4 hours · 11 parts

Part 7 of 11 · 2 min read

Answering LoRA questions in an interview

What are LoRA's rank and alpha, and how do you choose them?

"Rank r is the inner dimension of the low-rank update B A, so it sets the adapter's capacity and size. alpha scales the update by alpha over r. I start at rank 8 to 16 on the attention matrices and sweep rank on validation data. In the example's toy the matching rank won and extra rank added nothing. I tune alpha together with the learning rate, because they trade off. In the example one run diverged at scale 4 with lr 0.1." Follow-up: "What if the task needs a big change to the model?" Rank may need to be high, or full finetuning may be better. In the example's full-rank case LoRA needed r of 32 or more to match.

Why initialise B to zero?

"So the adapter starts as an identity change: the model equals the base model at step 0 and training moves away from a known good point. With a random B the example began at error 0.344 instead of 0.238." Follow-up: "Why not both zero?" Then the gradients to both A and B are zero, so nothing would train. A is random and B is zero.

How do you serve many LoRA-tuned models?

"Keep one frozen base and swap adapters, which saves storage: for 100 customers and a 4096 x 4096 matrix at r=8 it is about 23 million parameters instead of 1.68 billion with merged copies. The cost is a little extra compute per token, which can be reduced with batched adapter kernels. For a single model I merge." Follow-up: "What can go wrong?" An adapter loaded on the wrong base version, or noisy-neighbour load on a shared base.

What does QLoRA change?

"It stores the frozen base in 4 bits with blockwise scales and de-quantises to 16-bit for compute, so a much larger model fits on one GPU. LoRA trains on top. The cost is quantisation error. In the example's toy LoRA on a 4-bit base reached 0.007 to 0.009 against 0.0009 for 16-bit, and the paper reports parity on its benchmarks, so I would check on my own eval." Follow-up: "Where does the memory go?" Mostly the base weights, activations and optimiser state for the adapter. The adapter itself is tiny.