Part 1 of 4 · 1 min read
Overview
An eval set from imagination covers what you thought of; one from traces covers what happens. The interview answer is a pipeline, not a dataset: collect, scrub, dedupe, sample, label, freeze, and keep refreshing.
About the example, and what is real
Real: the sampling strategies, the post-stratified estimator, the shingle-based deduplication and leak check in code/mini-traces, and the 11 tests. Assumed: traffic shares 60, 25, 10, 4 and 1 percent with failure rates 4, 12, 10, 45 and 60 percent; a weak signal (thumbs-down or error) that flags 60 percent of failures and 5 percent of passes; the texts used for deduplication are templated. No real logs or labels. Annotation quality (module 4) is not modelled: here the failure label is given.