Part 5 of 7 · 2 min read
Other controls and best-of-n
Stop sequences, length limits, penalties, voting and beam search, one at a time.
Here are the other dials, one at a time.
- Stop sequences: strings that end generation early (for example
</answer>). Like telling a speaker "stop when you say 'thank you'". - Max tokens: a hard limit on output length. A cut-off answer is a common bug, so check the finish reason in the response.
- Repetition and frequency penalties: lower the score of tokens already used, so the model repeats itself less.
- Beam search: keeps several candidate sentences alive at once and picks the best at the end. Common in translation, rare in chat models.
- Best-of-n and self-consistency: sample several answers and pick the best or the most common one. Think of "ask the audience" in a quiz show: if most people agree, the answer is probably right. It trades cost for accuracy.
On best-of-n, here is a real simulation. A pretend model is right 60% of the time and otherwise wrong in one of three different ways. Majority vote over n answers gave 61% for 1, 71% for 3, 81% for 5 and 92% for 9. It only works because the wrong answers differ. If the model made the same mistake every time, voting would not help.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Low temperature for tools and extraction | Set temperature near 0 for steps whose output a program reads, so loops are stable. |
| Schema-enforced output | Use the provider's structured-output or tool-call feature, then validate with Pydantic anyway. |
| Properties in tests | Tests check that the answer parses and contains the right facts, never an exact string from the model. |
| Record and replay | Real model responses are saved once and replayed in unit tests. |
| Checking the finish reason | Code looks at why generation stopped (length, stop sequence, tool call) and handles a cut-off. |
| Self-consistency for hard questions | For questions where accuracy matters more than cost, several samples are voted on. |
Interview angle
Answer in your head first, then open it.
What does temperature do?
It rescales logits before softmax: low sharpens, high flattens; it never changes the ranking.
Top-k versus top-p?
Top-k keeps a fixed count; top-p keeps a fixed probability mass, so it adapts to how confident the model is.
Is temperature 0 deterministic?
Mostly, but not guaranteed: floating-point order, batching and model updates can change outputs.
How do you get reliable JSON from a model?
Schema-constrained decoding or tool calling, then validation. Prompts alone are not a guarantee.
How would you test code that calls an LLM?
Assert on properties, use recorded responses for unit tests, and run a separate evaluation set.