Part 1 of 8 · 1 min read
Overview
In this module you will
- Explain what a token is, and why models read tokens instead of words or letters.
- Run byte pair encoding (BPE) by hand and in code, and say what it guarantees.
- Estimate the tokens and cost of a task before you run it.
- Explain several "strange" model behaviours (spelling, counting letters, arithmetic, higher cost in some languages) from tokenisation alone.
Everything you pay for, wait for and run out of is measured in tokens: the price of a call, how long it takes, how much fits in the context window, and how many sources your agent can read before it forgets the start. An agent that loops twenty times, re-sending a growing history each time, can use ten times the tokens you expect.
Builders who cannot count tokens get surprised by bills, by answers cut short, and by prompts that "mysteriously" stop fitting.