CSE Roadmap

Common tier · about 4 hours · 11 parts

Part 2 of 11 · 4 min read

Why a small add-on can change a big model

What LoRA trains, what it leaves alone, and why the add-on is so small.

Think of a clip-on lens. You do not rebuild the camera to get a wider shot. You clip something small on the front. LoRA does the same with a model: it leaves every original weight alone and trains a small add-on that sits next to one weight matrix.

In plain words: a weight matrix is a big grid of numbers, and LoRA learns a correction to that grid without touching the grid itself. The trick is that the correction is built from two thin strips instead of a full grid, which is what makes it small.

A worked example with real sizes. One matrix inside a small network is 64 numbers by 64 numbers, so it holds 64 x 64 = 4,096 numbers. Full fine-tuning would change all 4,096. LoRA instead learns one column of 64 numbers and one row of 64 numbers, and multiplies them to make a 64 x 64 correction. That is 64 + 64 = 128 numbers to train, about 3 percent of the matrix, yet the product fills the whole grid.

Here is the same idea on a 3 x 3 grid so you can see the multiplication. Take the column (1, 2, 3) and the row (1, 0, 2). Column times row gives:

            row:   1    0    2
column 1  ->       1    0    2
column 2  ->       2    0    4
column 3  ->       3    0    6

Nine numbers made from six, and every row of the result is just a multiple of the same row. That is what rank 1 means: one column-row pair. Rank 8 means eight such pairs added together. rank 64 means sixty-four. More pairs can express more complicated corrections, and cost more numbers.

LoRA (low-rank adaptation)

Freeze the original weight matrix W. Learn two small matrices, B (as tall as W, r columns wide) and A (r rows tall, as wide as W). Use W + (alpha / r) B A in place of W. Only A and B train.

Example: for the 64 x 64 matrix at rank 8, B is 64 x 8 and A is 8 x 64. The adapter has 2 x 64 x 8 = 1,024 trainable numbers against 4,096 in W.

The key lines. This is the whole of LoRA as code, from code/mini-lora/lora.py, trimmed to what matters:

# from code/mini-lora/lora.py
def lora_init(r, seed=0):
    ...
    return {"A": rng.normal(size=(r, H)) / np.sqrt(H), "B": np.zeros((H, r))}   # A small random, B all zero

def lora_W2(P, L, scale):                          # scale = alpha / r
    return P["W2"] + scale * (L["B"] @ L["A"])     # W plus the correction, built on the fly

# in the training loop: only A and B get gradient steps; P["W2"] is never touched
opt.step(L, {"B": scale * G @ L["A"].T, "A": scale * L["B"].T @ G})

Read it as: keep W as it is. Every time the layer runs, add scale * B @ A to it. Train only A and B. Merging for serving is one more line, Q["W2"] = lora_W2(P, L, scale), which bakes the correction into W once and for all.

The size difference is what makes LoRA practical on real models. A 4096 x 4096 attention matrix holds about 16.8 million numbers. A rank-8 adapter for it has 2 x 4096 x 8 = 65,536 numbers, about 0.4 percent. A rank-64 adapter has about 524,000, about 3 percent. For GPT-3 the LoRA paper reports roughly 4.7 million trainable parameters out of 175 billion, which recomputes as 0.0027 percent.

Rank (r)

The number of column-row pairs in the adapter, which sets both its size and how complicated a correction it can express.

Example: on the 64 x 64 matrix, rank 1 is 128 numbers, rank 8 is 1,024, and rank 64 is 8,192, which is more than the matrix itself. Past that point LoRA has stopped being small.

alpha

A volume knob on the adapter. The correction B A is multiplied by alpha / r before it is added to W, so the same adapter can be turned up or down without retraining.

Example: alpha 16 at rank 8 is a scale of 2. Alpha 16 at rank 64 is a scale of 0.25. Common practice keeps alpha : r somewhere between 1 : 8 and 8 : 1.

Why B starts at zero

At the start of training, B is all zeros, so B A is zero and the model behaves exactly like the untouched base model. Training then moves away from a known good starting point.

Example: in the toy, starting with a random B put the model at error 0.344 before any training, against 0.238 for the base, so the first steps had to undo a random perturbation.

PEFT (parameter-efficient fine-tuning)

The family name for any method that trains a small number of added or selected parameters instead of all the weights. LoRA is the most used member. The Hugging Face library with that name implements several.