Part 2 of 8 · 3 min read
The problem, the words and the pieces
A model cannot read letters or words. It reads numbers. Here is how text becomes numbers, and why the pieces are word parts.
The problem: the model does not read words
Type this into a chat model: "How many times does the letter r appear in strawberry?" Some models get it wrong. A person finds this strange: a child can count letters.
The reason is that the model never sees the letters. A computer program can only work with numbers, so before the model reads your text, another program chops the text into pieces and replaces each piece with a number. The model reads only those numbers, and writes only numbers, which are turned back into text afterwards. This whole module is about that chopping step, because it decides what the model can see, what it costs, and how long it takes.
text -> tokenizer.encode -> [numbers] -> model -> [numbers] -> tokenizer.decode -> text
Token
A piece of text that a model treats as one unit. It may be a whole word, part of a word, a single character, punctuation, or a single byte. The model sees a list of token numbers, never letters or words.
Tokenizer
The program that turns text into token numbers (encode) and back into text (decode). It is separate from the model, built before the model is trained, and fixed afterwards. A model can only be used with the tokenizer it was trained with.
Vocabulary
The full list of tokens a tokenizer knows, each with its number (its ID). Real vocabularies hold from tens of thousands to a few hundred thousand tokens.
Special token
A reserved token that carries structure instead of text: "start of text", "end of text", and the markers that separate system, user and assistant turns. This is how a chat is laid out inside the model.
One detail matters later: the space in front of a word is usually part of the token, not a token of its own. So " tokens" (with a leading space) and "tokens" (at the very start of a line) are different tokens with different IDs.
Check yourself
- What does the model actually receive: letters, words or numbers?
- Why can a model only be used with the tokenizer it was trained with?
Why pieces of words?
There are four obvious ways to cut text up:
| Scheme | Idea | Strength | Weakness |
|---|---|---|---|
| Word tokens | One token per word | Short sequences | Huge vocabulary; cannot handle a new word; apology, apologise and apologetic become unrelated tokens |
| Subword tokens | Whole words if common, pieces if rare | Handles any word; related words share pieces | Needs a trained vocabulary |
| Character tokens | One token per character | Tiny vocabulary | Very long sequences, and each position carries little meaning |
| Byte tokens | One token per byte (256 values) | Can represent any text in any language | The longest sequences |
A byte is the unit a computer uses to store text: 256 possible values. English letters take one byte each; many other scripts take two to four bytes per character. Modern tokenizers use subwords built on top of bytes. Common words become one token and rare words are split into pieces. Anything the tokenizer does not know falls back to smaller pieces, and in the worst case single bytes. So a tokenizer can never fail, and there is no "unknown word" token.
Sizes of vocabularies reported for three well-known tokenizers: GPT-2 has 50,257 tokens, Flan-T5 has 32,100, and the code model StarCoder2 has 49,152. A bigger vocabulary gives shorter sequences (cheaper for the same sentence) but a larger table inside the model. Newer models tend to use larger vocabularies, partly to handle more languages efficiently. (These sizes are quoted from published descriptions; I did not re-measure them.)
Check yourself
- Why does a character-token scheme make long sequences?
- What happens when a subword tokenizer meets a word it has never seen?