Part 6 of 8 · 3 min read
Strange behaviours and rules for builders
Why models miscount letters and fumble numbers, and six rules that keep you out of trouble.
Now the strawberry puzzle has an answer. The model sees a few multi-letter tokens, not ten letters, so it must recall how those tokens are spelled instead of looking at them. This table collects similar cases:
| Behaviour | Tokenisation explanation | What to do |
|---|---|---|
| The model miscounts letters in a word | It sees a few multi-letter tokens, not letters | Give it code to count, or spell the word out with spaces |
| Arithmetic on long numbers is unreliable | Numbers split into arbitrary chunks | Use a calculator tool |
Different answers for Hello and hello |
Different tokens | Keep prompt formatting consistent |
| A rare name or ID is garbled | Split into many small pieces with little meaning | Quote it exactly; copy it, do not retype it |
| JSON and code are expensive | Brackets, quotes, long key names and indentation each cost tokens | Use short keys and minimal formatting |
| Reversing or rhyming fails | The model works on pieces, not letters or sounds | Use a tool |
| A user types text that looks like a special token | Some tokenizers turn it into the real marker | Strip or escape special-token strings in user input |
The model is not "bad at counting". It is reading a different alphabet than you.
These are explanations, not measurements
The table says why such behaviours are expected. Whether a particular model fails on a particular input is something to test (Hard 3). Models also differ, and newer ones may handle some of these better.
Six rules for builders
- Count tokens before you send, and log the provider's
usageafter (Level 0, module 6). - Budget the whole window: system prompt, history, retrieved text, tool results and the reply.
- Use the right tokenizer for the model. Do not use one model's counter for another.
- Test your language and your data, not just English examples.
- Never let a user string become a special token.
- Prefer compact formats for repeated structure, and trim what you re-send.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Count with the provider's own tool | Use the token-counting feature or the usage field of the API you call, since each model family has its own tokenizer. |
| A token budget per request | Code that checks "system + history + retrieved text + reply cap fits the window" before sending. |
| Cost dashboards | Logs of input and output tokens per request, grouped by feature and user, with alerts on spikes. |
| Prompt caching and trimming | Keeping the repeated start of a prompt identical so the provider can reuse it, and cutting history that no longer helps. |
| Language checks | Sampling real user text in every supported language to see its token count before setting prices. |
Draft: to be checked against the current documentation of the providers the course uses (token counting, caching and billing of reasoning tokens) before this page goes live.
Interview angle
Answer in your head first, then open it.
What is a token and why not just use words?
A unit of text from a learned vocabulary, usually a word or word part. Words give an unbounded vocabulary and cannot handle new words; characters make sequences too long; subwords balance both.
How does BPE work?
Start from bytes, repeatedly merge the most frequent adjacent pair into a new token, stop at the target vocabulary size. Encoding applies the learned merges in order.
Why can the same request cost very different amounts in two languages?
Different languages tokenise with different efficiency, and price and latency scale with token count.
Why do models struggle to count the letters in a word?
They see tokens, not letters.
How would you estimate the cost of 10,000 requests?
Average input and output tokens times the two prices, using the provider's tokenizer on a sample, with a margin for agent loops and retries.