Part 5 of 6 · 4 min read
Exercises
Try each exercise before opening its hint. Use current provider pages for any real model facts.
Easy
Easy 1: Write your hard attributes (20 minutes)
Pick an agent idea (for example, an assistant that drafts replies to customer emails). Write five hard attributes with a reason for each, and one soft attribute you would try to improve before rejecting a model.
Done when: each hard attribute has a concrete threshold or a yes/no question.
Easy 2: Read the table (15 minutes)
Using the four-model table above, which model is cheapest per correct answer? Compute cost divided by accuracy for each.
Done when: you give the four ratios and name the winner and the runner-up.
Hint
cost per correct answer is the cost per task divided by the accuracy.
Easy 3: Spot the dominated option (15 minutes)
Given the points a = ($2, 0.80), b = ($5, 0.78), c = ($9, 0.90), d = ($1, 0.60), decide which are on the Pareto front, then confirm with pareto_front.
Done when: you can say why b is dominated and by which model.
Hint
a model is dominated if another is at most as expensive and at least as accurate.
Medium
Medium 1: Build a 40-case test set (60 minutes)
Write 40 realistic inputs with expected outputs for a task you care about (classify support tickets, extract dates, and so on), tagged easy or hard. Write the scoring function and test it on five cases you score by hand.
Done when: the scorer agrees with your hand scores on all five, and the set contains at least 10 hard cases.
Medium 2: Two real models, one test set (90 minutes)
Run your set from Medium 1 against two real models from different price tiers (use whatever access you have, or free tiers). Record accuracy, cost and time per case. Compute a paired interval for the difference with paired_diff_ci.
Done when: you have a table of results and a sentence that says whether the gap is distinguishable from luck.
Medium 3: How many cases do you need? (60 minutes)
With the simulated models, hold the accuracies at 0.80 and 0.85. For case counts of 50, 100, 200, 400 and 800, run 100 repeated experiments (use a different seed each time) and count how often the paired interval excludes zero. Plot or tabulate the detection rate.
Done when: you can state roughly how many cases are needed to detect a five-point gap most of the time.
Hint
the detection rate climbs with the number of cases; use a log axis for case counts.
Hard
Hard 1: Contamination experiment (90 minutes)
Create 30 test questions. Build a "training" text that contains a verbatim copy of 10 of them and paraphrases of 10 others. Measure contamination_rate with n = 8, 5 and 3. Report which cases each setting detects, and how false alarms change with small n.
Done when: you have a table of detection against n and a conclusion about the trade-off.
Hint
shorter word runs catch paraphrases but also match common phrases.
Hard 2: Cost-aware routing (120 minutes)
Using the simulated cards, build a router: send "easy" cases to small-fast or open-14b and "hard" cases to frontier-large. Assume the router itself is right 90% of the time. Compute accuracy and cost per 1,000 tasks for the mix, and compare with using mid-size for everything.
Done when: you report both numbers and say when routing wins and when it does not.
Hint
routing accuracy for a case is 0.9 x accuracy of the right model + 0.1 x accuracy of the wrong one.
Hard 3: A regression gate (120 minutes)
Write a script that takes two result files (old model, new model) on the same cases and decides "ship" or "block". Ship only if the paired interval for the accuracy difference has a lower bound above -0.02 and the cost per task does not rise by more than 20%. Test it on three synthetic scenarios: clear improvement, noisy tie, and cheaper but worse.
Done when: all three scenarios produce the decision you expect and the rule is documented.
Hint
reuse paired_diff_ci; remember to check that the case ids match.