Part 6 of 11 · 1 min read
How to set rank, alpha and the rest in practice
- Start with rank 8 to 16 on all attention matrices (add MLP matrices if quality is short) and sweep rank on a validation set.
- Initialise B to zero and A small random, so training starts at the base model.
- Tune alpha and the learning rate together; keep alpha : r within a sensible band and watch for divergence.
- Merge for one model. Keep adapters separate for multi-tenant or multi-task serving.
- Version adapters with the base model, since an adapter is only valid for the exact base weights it was trained on.
- Use 4-bit bases (QLoRA) when memory is the limit, and compare against a 16-bit run on your own eval.
- Smaller quantisation blocks cost extra bits; measure error against memory.
- Hold out data from a later period to check the adapter does not overfit.
- Run a regression eval on tasks the adapter should not change.
- Recompute any parameter count you copy (tables sometimes label the matrices wrongly).