Part 5 of 7 · 3 min read
Scaling laws
Why model quality improves predictably with size, how to fit the curve, and the two rules of thumb you will see quoted.
Researchers noticed that a model's training loss (the error it is trained to reduce) falls smoothly and predictably as you increase parameters, data or compute, following a power law.
Power law
A relationship y = a x^(-alpha). On a chart with logarithmic axes it is a straight line whose slope is -alpha. That makes it easy to fit: take logs and draw a line.
The code does exactly this. fit_power_law takes model sizes and losses, subtracts the part of the loss that cannot be reduced, fits the line in log space, and returns (a, alpha). A test generates synthetic losses 1.7 + 25 x N^-0.3 and recovers alpha = 0.3 and a = 25. The demo recovers (30.0, 0.25) from data built with exactly those values. That is the logic of "train several small models, fit the line, and predict a big one before paying for it". Real fits are noisier and need care.
Two rules of thumb you will see quoted:
- Compute: training takes about
6 x parameters x tokensfloating-point operations. For 7 billion parameters and 140 billion tokens that is 5.88 x 10^21. - Compute-optimal data: for a fixed compute budget, the "Chinchilla" study found that parameters and training tokens should grow together, at very roughly 20 tokens per parameter. For example, 70 billion parameters with 1.4 trillion tokens matches that ratio. Many recent models are deliberately trained on far more tokens than this, because a smaller model that has seen more data is cheaper to serve, and serving cost dominates for popular models.
Scaling laws are not guarantees
They describe average loss on a fixed kind of data. They do not promise that any particular ability appears, they say nothing about data quality, and the extrapolation breaks if the setup changes.
What to read next
- The original "Attention Is All You Need" (2017), then the RoFormer paper (RoPE), the GQA paper and the Chinchilla paper.
- Level 13 of this course, where fine-tuning puts this knowledge to work.
For the curious What happens to the residual stream?
I stacked 12 random blocks and compared input and output (random weights, width 16). With residual connections the output still correlates with the input (correlation 0.38), because each block only adds a change. With the two additions removed, the correlation fell to 0.08 and the output size shrank from 34 to 7. Residuals keep the original signal flowing through a deep stack. These are random weights, so only the direction of the effect is meaningful.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Reading a model card | You can tell what "8 key-value heads", "rotary positions" or "64 experts, 8 active" mean for memory and speed. |
| Estimating serving memory | You compute the key-value cache for your context length and number of users before choosing hardware. |
| Comparing total and active parameters | For MoE models you look at active parameters for speed and total parameters for memory. |
| Rough training budgets | You use 6 x parameters x tokens to size a fine-tune or a training run before asking for GPUs. |
| Treating scaling laws as estimates | You fit small runs to predict larger ones and check the prediction against a real run. |
Draft: to be checked against current model cards and inference-engine documentation before this page goes live.
Interview angle
Answer in your head first, then open it.
Draw a transformer block.
Pre-norm, causal multi-head attention, residual add, pre-norm, MLP, residual add.
Why residual connections?
They give information and gradients a direct path, which makes deep stacks trainable.
What does RoPE give you?
Attention scores that depend on relative distance, by rotating queries and keys according to position.
Why do modern models use GQA?
A much smaller KV cache, so cheaper long-context serving, with little quality loss.
What is MoE and what is its catch?
Many experts, few active per token, so more capacity per unit of compute; the catches are load balancing and memory for all experts.
What does the Chinchilla result say?
For a fixed compute budget, scale parameters and data together.