CSE Roadmap

Common tier · about 5 hours · 7 parts

Part 5 of 7 · 2 min read

Other controls and best-of-n

Stop sequences, length limits, penalties, voting and beam search, one at a time.

Here are the other dials, one at a time.

  • Stop sequences: strings that end generation early (for example </answer>). Like telling a speaker "stop when you say 'thank you'".
  • Max tokens: a hard limit on output length. A cut-off answer is a common bug, so check the finish reason in the response.
  • Repetition and frequency penalties: lower the score of tokens already used, so the model repeats itself less.
  • Beam search: keeps several candidate sentences alive at once and picks the best at the end. Common in translation, rare in chat models.
  • Best-of-n and self-consistency: sample several answers and pick the best or the most common one. Think of "ask the audience" in a quiz show: if most people agree, the answer is probably right. It trades cost for accuracy.

On best-of-n, here is a real simulation. A pretend model is right 60% of the time and otherwise wrong in one of three different ways. Majority vote over n answers gave 61% for 1, 71% for 3, 81% for 5 and 92% for 9. It only works because the wrong answers differ. If the model made the same mistake every time, voting would not help.

What you will use daily at work

Practice What it looks like
Low temperature for tools and extraction Set temperature near 0 for steps whose output a program reads, so loops are stable.
Schema-enforced output Use the provider's structured-output or tool-call feature, then validate with Pydantic anyway.
Properties in tests Tests check that the answer parses and contains the right facts, never an exact string from the model.
Record and replay Real model responses are saved once and replayed in unit tests.
Checking the finish reason Code looks at why generation stopped (length, stop sequence, tool call) and handles a cut-off.
Self-consistency for hard questions For questions where accuracy matters more than cost, several samples are voted on.

Interview angle

Answer in your head first, then open it.

What does temperature do?

It rescales logits before softmax: low sharpens, high flattens; it never changes the ranking.

Top-k versus top-p?

Top-k keeps a fixed count; top-p keeps a fixed probability mass, so it adapts to how confident the model is.

Is temperature 0 deterministic?

Mostly, but not guaranteed: floating-point order, batching and model updates can change outputs.

How do you get reliable JSON from a model?

Schema-constrained decoding or tool calling, then validation. Prompts alone are not a guarantee.

How would you test code that calls an LLM?

Assert on properties, use recorded responses for unit tests, and run a separate evaluation set.