Part 5 of 12 · 2 min read
Pairing the same inputs through both versions detects a change faster than splitting traffic does
Comparing two tasters on the same bowl of soup tells you more about which one tastes better than giving them different bowls from the same pot ever could.
Think of two chefs tasting the exact same spoonful of soup, versus tasting from two different pots made from the same recipe. The first comparison isolates the chefs' palates. The second mixes in all the batch-to-batch variation between the pots.
In plain words: shadow deployment runs the candidate on live inputs without serving its output, which makes it possible to run both the old and new version on the exact same request and compare their outputs directly -- a paired comparison -- while a standard A/B test splits different requests between versions, an unpaired comparison that has to contend with how much requests naturally vary from each other on top of whatever the real version difference is.
A worked example. On the same 500 requests with a true 4-point quality gain, a paired shadow-style comparison detects the gain 34% of the time, while an unpaired split of the same total number of requests detects it only 14% of the time -- the paired approach is roughly 2.4 times as sensitive, using the identical amount of traffic, purely because it removes request-to-request difficulty as a source of noise in the comparison.
The key lines. From code/mini-rollout/rollout.py:
# from code/mini-rollout/rollout.py
def shadow_vs_ab(n=500, true_gain=0.04, base=0.7, trials=800, seed=4, spread=3.0):
"""Shadow mode runs both versions on the SAME requests; A/B splits requests between them.
Request difficulty is shared, so shadow results are paired. Returns power of (paired sign test, unpaired z test)."""
rng = random.Random(seed)
from statistics import NormalDist
zc = 1.959964
paired = unpaired = 0
for _ in range(trials):
diffs = [rng.betavariate(base * spread, (1 - base) * spread) for _ in range(n)]
Read it as: the docstring names the mechanism directly: "request difficulty is shared". Each request in diffs has its own intrinsic difficulty. Both versions face that difficulty identically under shadow mode. So the comparison only has to explain differences caused by the version change itself. It does not have to explain differences caused by one group of requests happening to be easier than the other. That is exactly the extra noise an A/B split cannot avoid.
Shadow deployment, guardrail metric
A shadow deployment processes live inputs but its outputs are not served. A guardrail metric is a metric that must not get worse (error rate, cost, latency, safety flags), even if the main metric improves.
Example: shadow mode's 2.4x sensitivity advantage comes at a real cost worth naming alongside it -- it shows no user reaction at all, so it has to be paired with guardrail metrics and, eventually, a canary stage, before a change that only ever looked good in shadow is trusted with real users.