Few-shot examples: Examples and reasoning prompts

Show the model what a good answer looks like, choose examples that cover your cases, and check them for mistakes.

Shot and few-shot

Each example in a prompt is a shot (as in "one shot at the task"). Zero-shot means no examples. Few-shot means a handful of input and answer pairs. The model is not trained by this. It simply reads the pairs as part of the prompt, and nothing is remembered for the next call.

Examples are best for three things: format (show the exact reply shape), boundaries (show the tricky case) and labels the model would not guess (your own category names). In the example, a feature_request label has no keyword rule, so with no examples every feature request lands in other.

Accuracy for four example sets on six tickets. Real output of the example code.

prompt = build_prompt(LABELS, examples, ticket)

Reading the picture:

No examples: 50%. The three feature requests were missed.
Three unrelated examples: 50%. Examples about bugs, billing and accounts do not teach the new label. Examples must cover what you need.
Three on-topic examples: 100%.
Chosen per ticket: 100%. select_examples picks the pool examples most similar to the current ticket. This is dynamic selection. With a big pool it keeps the prompt short and relevant.

With six test tickets, each ticket is about 17 percentage points. Treat this as a demonstration of the mechanism, not as a measurement.

Check your examples

Bad examples teach bad behaviour. check_examples finds the common problems:

Problem	Why it hurts
Answer not in the label set	The model learns to invent labels
Same input, different answers	The model is told two things at once
One label dominates	The model learns to guess that label
An example equals a test input	Your measurement is inflated (leakage)

Keep a held-out set

Never measure on the same tickets you used as examples. Keep a separate test set (module 5) and let check_examples warn you about overlap. If you select examples dynamically from a pool, the pool must not contain your test tickets either.

Check yourself

Why did three unrelated examples score the same as none? (They were about other labels, so none of them showed the model what a feature_request looks like. Examples only teach what they cover.)