CSE Roadmap

Common tier · about 4 hours · 11 parts

Part 8 of 11 · 2 min read

Mistakes that waste a tuning run

Common mistakes, and what the example does not do.

Mistakes that waste a tuning run

  • Raising rank hoping for better quality without a validation curve.
  • Applying LoRA to a single matrix without testing more.
  • Random B initialisation and a worse starting point.
  • Changing alpha without re-tuning the learning rate.
  • Mixing adapters and base versions.
  • Assuming 4-bit is free; it costs accuracy that you must measure.
  • Training on 200 examples with full finetuning and calling LoRA "better" (small data favours small adapters).
  • Copying parameter counts without recomputing them from the matrix sizes.
  • Ignoring regression tests on tasks the base model did well.
  • Treating a toy result as evidence about LLMs.

What the example does not do

About this example What the example does not do
  • No real model: a 64-unit network, random base, and a target built by a low-rank change.
  • The low-rank assumption is built in, so the rank-4 result is flattering by design. The full-rank case is the fairer stress test.
  • Small data: 256 examples make full tuning overfit. With more data the gap may flip.
  • One seed per configuration, with learning rates chosen from a small grid, so small differences (for example between rank 8 and 16) are not conclusive.
  • The quantiser is an approximation (symmetric normal-quantile levels, no double quantisation, no paged optimiser).
  • No attention, no real training dynamics, no mixed precision.
  • Optional real experiment: with a small open model, train LoRA at ranks 4, 8, 16, 32 on one agent step with PyTorch and PEFT, and plot validation quality against rank. Then repeat with a 4-bit base and compare quality and memory.

Check yourself

Why is B initialised to zero in LoRA?

Show answer

So the adapter adds nothing at step 0 and training starts exactly from the base model. With a random B the toy started at error 0.344 instead of 0.238.

What you will use daily at work

Practice What it looks like
Pick rank by the validation curve Start small (4 to 16), raise it only if validation error keeps falling.
Apply LoRA to all attention matrices, often the MLP too Check what your library targets by default.
Merge for one model, or keep adapters separate Separate adapters let one base serve many tenants.
Use QLoRA when memory is the limit A 4-bit frozen base cuts memory about 3.8 times. Compare quality with 16-bit.