Part 7 of 8 · 4 min read
Exercises
Try each exercise before opening its hint.
Easy
Easy 1: Run and predict (30 minutes)
In code/mini-bpe, run python demo.py. Before running, predict how many tokens "the agent reads the tokens" and "agentic" will need, and say why one is cheaper per character. Then run the tests with pytest.
Done when: you can explain the difference using the word "merge".
Easy 2: Round-trip by hand (15 minutes)
Take the word banana. Starting from single bytes, apply the rule "merge the most frequent pair" by hand for three merges, writing the sequence after each. Then decode it back. Compare with the trace in section 3.
Done when: you have four sequences (the start plus three merges), decoding gives banana again, and you have said which tie-break rule you used.
Hint
for banana both an and na occur twice. The example code picks na because it breaks ties by the larger pair; either choice is fine if you state it.
Easy 3: Estimate a bill (15 minutes)
Use the rough rule from section 5 (about four characters per token in English) to estimate the tokens in a 2,000-character email, then price 500 such emails at an input price you invent. Show your arithmetic.
Done when: you have a token count, a total price and one sentence on how wrong the estimate could be.
Hint
input and output are priced separately; keep the two in separate variables even if the output is zero here.
Medium
Medium 1: Compression versus vocabulary size (45 minutes)
Take 5,000 or more words of your own English text (an essay, a README). Train the example tokenizer with vocabulary sizes 260, 300, 400 and 600. For each size, record the number of tokens needed for a held-out paragraph that was not in the training text. Make a small table.
Done when: you can say where the returns start to diminish and why held-out text matters.
Hint
the training text must not contain the held-out paragraph, or you are measuring memory, not skill. The example stops early when no pair occurs twice, so tiny texts may stop before the size you ask for.
Medium 2: Measure the language gap (45 minutes)
Train on English text, then encode a sentence and its translation in a language you speak. Compute tokens per character for each. Repeat after training on a mix of both languages, using different sentences for training and testing (unlike the demo above). Explain the change.
Done when: training on the mix reduces the gap, and you can state the lesson for a multilingual product.
Hint
Devanagari characters take three bytes each in UTF-8, so a tokenizer that never saw them starts at three tokens per character.
Medium 3: Cost calculator (30 minutes)
Pick a real model and its current price page (Level 0, module 8). Use cost.py to price 1,000 and 100,000 requests of your own typical size. Add an agent factor: ten loop steps where each adds 500 tokens to the history. Compare the naive and the loop-aware estimates.
Done when: you have a two-line table and can say which assumption moves the total most.
Hint
the loop re-sends the growing history on every step.
Hard
Hard 1: Use a real tokenizer (30 minutes)
Install tiktoken, load two different encodings and count tokens for the same five texts: plain English, a Python function, a JSON object, a sentence in your language, and a long number. Note which encoding is cheaper for each. (I could not run this, so your numbers are the ground truth.)
Done when: you can name one text type where the choice of tokenizer changes the count by more than 20%.
Hint
compare cl100k_base and o200k_base if both are available in your tiktoken version.
Hard 2: Add word-boundary pre-splitting (75 minutes)
Modify BPE.train and encode so the text is first split into chunks with a regular expression (words with their leading space, single digits, punctuation) and merges never cross a chunk. Keep the round-trip tests passing. Compare token counts and the learned pieces before and after.
Done when: all existing tests pass, a new test shows no learned token spans two words, and you can describe one downside of splitting digits.
Hint
a working starting pattern is r" ?[^\W\d_]+|\d|[^\w\s]|_|\s+" (a word with its leading space, a single digit, punctuation, an underscore, whitespace). Be careful: the simpler pattern r" ?\w+|\d|..." does not split digits, because \w already matches them. Apply merges within each chunk and join the results.
Hard 3: Explain a strange behaviour with tokens (60 minutes)
Pick one behaviour from section 6 (counting letters, odd arithmetic, trailing spaces). Design a small experiment using your trained tokenizer and, if you have access, a real model: write the prediction from the token view, run it on at least ten inputs and report how many matched. (I did not run the real-model part, so your numbers are the ground truth.)
Done when: you have a written prediction, ten results and a short conclusion that says whether tokens explain the behaviour.
Hint
keep the inputs varied (short, long, with and without spaces) so one lucky case cannot decide the result.