CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 2 of 12 · 2 min read

A trace becomes an eval case once you say what counts as success

A log of what an agent actually did is not a test until someone decides what the right answer should have been.

Think of a restaurant reviewing its own receipts versus testing recipes on imagined customers. The receipts show exactly what people actually ordered and complained about. A chef's imagination would never have covered that as thoroughly.

In plain words: a trace is the recorded input, steps and output of one real agent run, an eval case is an input plus what counts as success (an expected answer, a check or a rubric), and a golden set is a curated, labelled collection of such cases treated as ground truth -- turning raw traces into eval cases is what makes production experience reusable as a test, rather than something that only ever happened once.

A worked example. The simulated traffic has five categories with very different shares and failure rates: order-status (60% of traffic, 4% failure), refund (25%, 12%), account (10%, 10%), multi-step (4%, 45%) and injection (1%, 60%) -- combined, the true overall failure rate works out to 8.8%. A random sample of 50 traces, drawn the ordinary way, contains the rarest category (injection, 1% of traffic) only 39% of the time -- meaning more than half the time, a random eval set built this way would contain zero examples of the single riskiest category.

The key lines. From code/mini-traces/traces.py:

# from code/mini-traces/traces.py
CATS = {"order_status": (0.60, 0.04), "refund": (0.25, 0.12), "account": (0.10, 0.10),
        "multi_step": (0.04, 0.45), "injection": (0.01, 0.60)}


def true_failure_rate(cats=CATS):
    return sum(s * f for s, f in cats.values())

Read it as: the true overall failure rate is a traffic-weighted average. Each category's failure rate is multiplied by its share of traffic, and the results are summed up. This is the ground truth the module tests every sampling strategy against. It only exists because the simulation defines exact shares and rates up front. Real production logs never hand you this directly.

Trace, eval case, golden set

A trace is the recorded input, steps and output of one agent run. An eval case is an input plus what counts as success (an expected answer, a check, or a rubric). A golden set is a curated, labelled set of cases treated as ground truth.

Example: the injection category appearing in a random 50-case sample only 39% of the time is a warning about turning traces into a golden set carelessly -- the riskiest category can simply be missing most of the time if nobody deliberately makes sure it's represented.