Part 8 of 11 · 2 min read
Mistakes that waste a tuning run
Common mistakes, and what the example does not do.
Mistakes that waste a tuning run
- Raising rank hoping for better quality without a validation curve.
- Applying LoRA to a single matrix without testing more.
- Random B initialisation and a worse starting point.
- Changing alpha without re-tuning the learning rate.
- Mixing adapters and base versions.
- Assuming 4-bit is free; it costs accuracy that you must measure.
- Training on 200 examples with full finetuning and calling LoRA "better" (small data favours small adapters).
- Copying parameter counts without recomputing them from the matrix sizes.
- Ignoring regression tests on tasks the base model did well.
- Treating a toy result as evidence about LLMs.
What the example does not do
About this example What the example does not do
- No real model: a 64-unit network, random base, and a target built by a low-rank change.
- The low-rank assumption is built in, so the rank-4 result is flattering by design. The full-rank case is the fairer stress test.
- Small data: 256 examples make full tuning overfit. With more data the gap may flip.
- One seed per configuration, with learning rates chosen from a small grid, so small differences (for example between rank 8 and 16) are not conclusive.
- The quantiser is an approximation (symmetric normal-quantile levels, no double quantisation, no paged optimiser).
- No attention, no real training dynamics, no mixed precision.
- Optional real experiment: with a small open model, train LoRA at ranks 4, 8, 16, 32 on one agent step with PyTorch and PEFT, and plot validation quality against rank. Then repeat with a 4-bit base and compare quality and memory.
Check yourself
Why is B initialised to zero in LoRA?
Show answer
So the adapter adds nothing at step 0 and training starts exactly from the base model. With a random B the toy started at error 0.344 instead of 0.238.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Pick rank by the validation curve | Start small (4 to 16), raise it only if validation error keeps falling. |
| Apply LoRA to all attention matrices, often the MLP too | Check what your library targets by default. |
| Merge for one model, or keep adapters separate | Separate adapters let one base serve many tenants. |
| Use QLoRA when memory is the limit | A 4-bit frozen base cuts memory about 3.8 times. Compare quality with 16-bit. |