Part 2 of 12 · 2 min read
More attempts only help if you can tell the good one from the rest
Drawing several answers to the same question is cheap compute; choosing well among them is what turns that compute into accuracy.
Think of a chef plating five versions of a new dish. Making five plates is easy. The real skill, and the real value, is in a taster who can actually pick the best one.
In plain words: repeated sampling draws several candidate answers to the same query, best-of-N chooses among them with a scorer or verifier, majority vote chooses the most common answer, and self-consistency is majority vote taken over several reasoning chains -- which of these you use matters more than how many samples you draw.
A worked example. Each task has its own hidden difficulty (a per-task chance of a correct sample, averaging 0.55), and every draw is independent given that difficulty. Going from one sample to three with a verifier choosing among them lifts accuracy from 0.55 to 0.73; majority vote over the same three samples only reaches 0.60, because a vote has no way to recognize a correct answer -- it can only count how often answers repeat.
The key lines. From code/mini-explore/sampling.py:
# from code/mini-explore/sampling.py
def sample(r, q, p):
if r.random() < q:
return 0
return 1 if r.random() < p.concentrate else r.randint(1, p.wrong_options)
Read it as: q is this task's own chance of a correct draw. So sample returns the correct answer (0) with that probability. Otherwise it returns a wrong answer, which is either the single "attractor" wrong answer or one of several scattered ones. Every candidate in a batch is drawn from this same per-task q. That is exactly why more draws help. Each one is an independent new roll against the same odds.
Repeated sampling, best-of-N, majority vote, self-consistency
Repeated sampling draws several candidate answers to the same query. Best-of-N chooses among them with a scorer or verifier. Majority vote chooses the most common answer. Self-consistency is majority vote taken over several reasoning chains from a chain-of-thought prompt.
Example: best-of-3 reaching 0.73 against majority-of-3's 0.60 on the exact same three draws is the clearest demonstration that "how many samples" and "how you choose among them" are two separate levers, and the second one is usually the bigger lift.