Part 5 of 6 · 3 min read
Exercises
Try each exercise before opening its hint.
Easy
Easy 1: Softmax by hand (20 minutes)
For logits [2, 1, 0], compute the probabilities by hand with a calculator (temperature 1), then with temperature 0.5. Confirm with softmax. (The answers are in section 2, so do the hand work first.)
Done when: your numbers match to three decimals, and you can say which token gained probability at the lower temperature.
Hint
the denominator is e^2 + e^1 + e^0.
Easy 2: Pick the setting (15 minutes)
Choose a temperature range and explain in one line for each: (a) extract an invoice total from text, (b) write five names for a coffee shop, (c) answer a support question from a policy document.
Done when: each answer has a number and a reason.
Easy 3: Count the tail (20 minutes)
For the demo distribution at T = 1, how many tokens does top-p 0.9 keep? top-p 0.99? top-k 3? Verify with the functions.
Done when: you can explain why top-p 0.9 and top-k 2 gave the same result here but would not always.
Hint
sort the probabilities, then add them from the top.
Medium
Medium 1: Plot entropy against temperature (45 minutes)
Compute entropy for temperatures from 0.1 to 5 in steps of 0.1 for your own 8-token logit vector. Plot or tabulate it, and mark where entropy first exceeds 1 bit.
Done when: the curve rises steadily and you state the temperature where it passes 1 bit.
Medium 2: Test the sampler statistically (45 minutes)
Draw 50,000 samples with a fixed seed using top_p = 0.9, and show that the observed frequencies match the filtered, renormalised probabilities within 0.01. Show that tokens outside the nucleus are never drawn.
Done when: both assertions pass and are in a test file.
Hint
compare np.bincount(draws) / n with top_p_filter(softmax(logits), 0.9).
Medium 3: Add a repetition penalty (60 minutes)
Write apply_repetition_penalty(logits, history, penalty) that divides positive logits (and multiplies negative ones) of already-used tokens by penalty. Test that it lowers the probability of repeated tokens and that penalty = 1 changes nothing.
Done when: both tests pass and you can explain why negative logits are treated differently.
Hint
dividing a negative logit by a number above 1 would make it less negative, which would raise its probability.
Hard
Hard 1: A tiny generator (90 minutes)
Using a small bigram table you write (about 15 words), generate 10 sentences of 8 tokens each with greedy decoding and with temperature 1.5 plus top-p 0.9. Measure the number of distinct sentences and the share of repeated words in each setting.
Done when: you have a table of both measures and a sentence about the trade-off.
Hard 2: Best-of-n and majority vote (90 minutes)
Simulate a "model" that answers a question correctly with probability 0.6 and otherwise gives one of three wrong answers at random. Sample n answers for n = 1, 3, 5, 9 and take the most common one. Run 5,000 trials per n and report accuracy.
Done when: accuracy rises with n, and you explain why it would not rise if the model's errors were all the same wrong answer.
Hint
with majority voting, wrong answers are split three ways, so the correct one wins more often as n grows.
Hard 3: Grammar-constrained decoding (120 minutes)
Build a character-level generator that must produce a string matching ^[0-9]{3}-[A-Z]{2}$. At each step, mask out characters that would break the pattern, sample from the rest using random logits, and generate 1,000 strings. All 1,000 must match.
Done when: a test verifies all strings against the regex and a second test confirms that without the mask many strings fail.
Hint
keep a position counter; the allowed character set depends only on the position.