Part 1 of 4 · 1 min read
Overview
Agents often need a small model that follows your tool-call format or house style. LoRA makes that a hours-long job on one GPU instead of a cluster job, and adapters let one served base model carry a different behaviour per tenant or per task (Level 11 module 7 on tenancy). QLoRA is why a 7B to 70B model is tunable on rented single-GPU hardware. Knowing the knobs (rank, alpha, target matrices, quantisation) is a common interview topic.
About the example, and what is real
Real: the 3-layer network with hand-written gradients (checked against finite differences in the tests), the LoRA training loop with Adam, full-tuning baselines, merging, the blockwise 4-bit quantiser, and the 19 tests in code/mini-lora. Assumed: the base network is random; the "task" is the same network with W2 changed by a rank-4 (or full-rank) matrix of 50 percent relative size; 256 training and 256 validation examples; learning-rate grids. Not real: no language model, no real data, no GPU; the quantiser's level table approximates NF4 and is not bitsandbytes; nothing here shows how LoRA performs on a real LLM.