Part 1 of 7 · 1 min read
Who this is for
In this module you will
- Read the architecture section of a model announcement and understand every term.
- Build a transformer block in NumPy and explain each part: normalisation, attention, the feed-forward network and the residual paths.
- Explain rotary position embeddings (RoPE), grouped-query attention (GQA) and mixture of experts (MoE), and show what each buys.
- Use the rough scaling-law rules for training cost and data, and fit a power law to data.
This module is optional. Modules 1 to 7 are enough to build agents well. Take this one if you want to pass deep technical interviews, to read papers, to fine-tune models later (Level 13), or simply because you want to know what is inside.
Everything here is small enough to run, and each piece is checked by a test. But none of it is a full model: nothing is trained, the weights are random, and the sizes are tiny on purpose. It shows the mechanisms, not the quality of a real model. It builds on modules 1 and 2, so read those first.