Cutting off the unlikely tail so a rare absurd token cannot be chosen.
Top-k and top-p
Even at temperature 1 the long tail matters. A vocabulary of 100,000 tokens means a small chance of something absurd on each step, and across hundreds of steps it adds up. Filtering removes the tail before sampling.
Top-k
Keep only the k most likely tokens, set the rest to zero, renormalise so they add up to 1 again, then sample.
Top-p (nucleus sampling)
Sort the tokens by probability and keep the smallest set whose probabilities add up to at least p. The number of tokens kept changes at every step.
Top-k keeps a fixed number of tokens; top-p keeps a fixed share of the probability.
For the Paris example at temperature 1: the probabilities sorted from the top add up like this: 0.808, 0.917, 0.983, 0.998, 1.0. So top-p 0.9 keeps 2 tokens (0.808 + 0.109 = 0.917 is the first sum to reach 0.9), top-p 0.99 keeps 4, and top-k 3 keeps 3. After top-k 2 or top-p 0.9 the probabilities are renormalised to 0.881 and 0.119.
Top-p adapts to how sure the model is. Sampling 1,000 times with top-p 0.9 gives:
temperature 1, top-p 0.9: Paris 881, Lyon 119 (2 tokens kept)
temperature 5, top-p 0.9: Paris 333, Lyon 260, France 239, a 168 (4 tokens kept)
When the model is sure (temperature 1), few tokens remain. When the distribution is flat (temperature 5), more remain. A test in the example proves this: a confident distribution keeps one token, and a flat one keeps all five. Top-k cannot adapt this way, because it always keeps the same number.
Check yourself
Why does top-p 0.9 keep 2 tokens here but 4 at temperature 5?
What does "renormalise" mean and why is it needed?
Choosing settings, and why 0 is not always repeatable
Practical starting points
Task
Temperature
Top-p
Why
Extract fields, classify, call a tool
0 to 0.2
leave default
You want the same, correct answer
Chat, summarise, explain
0.3 to 0.7
0.9 to 1.0
Natural but stable
Brainstorm, name ideas, creative text
0.8 to 1.2
0.9 to 0.95
Variety is the point
These ranges are common starting points from general practice. I did not test them on any real model; test them on your own task.
Change one thing at a time
Some providers advise adjusting temperature or top-p, not both, because they overlap and changing both makes behaviour hard to reason about. Some reasoning models fix or ignore these settings. Check the model's documentation.
Why temperature 0 may still differ
Even with greedy decoding, two identical requests can give different text. The reasons are ordinary:
Floating-point arithmetic is not associative. Adding the same numbers in a different order can give a slightly different result. The example shows 0.1 + 0.2 + 0.3 giving 0.6000000000000001 one way and 0.6 when added in reverse. On real hardware the order of additions can depend on how requests are batched together.
Near-ties. If two tokens are almost equal, a tiny difference flips the choice, and every later token then differs.
Model or infrastructure changes. A provider may update the model behind a name.
Consequence: never write a test that requires a model to return exactly the same text. Test properties (valid JSON, contains the key fact), and where you need repeatability, record and replay responses (Levels 2 and 9).
Check yourself
Which settings would you choose to extract an invoice total, and why?
Name two reasons "temperature 0" might not repeat exactly.