Part 4 of 13 · 2 min read
Find the critical weight: how wrong would the weights have to be?
Asking for the one correct set of weights is the wrong question; asking how far off the weights can be before the answer changes is the right one.
Think of a seesaw balanced between two riders of almost equal weight. The interesting question is not "exactly how much does each one weigh." It is "how much would one of them have to gain or lose before the seesaw tips."
In plain words: a trade-off means improving one property costs another, so a weighted score always embeds a judgment call about how much each property is worth -- and the critical weight is the weight on one criterion at which the winning option changes, which tells you exactly how sensitive the decision is to that judgment call, rather than asking you to defend one single "correct" weight.
A worked example. Between two close-scoring design variants (0.520 vs 0.515), the decision flips if the weight on p95 latency falls below 0.37, or if the weight on quality rises above 0.43; the weight on cost never flips the decision at all. Running thousands of simulated weight draws from increasing confidence in the base weights, the winning option's win probability rises from 55% (weights quite uncertain) to 77% (weights quite certain) -- even at full confidence in the stated weights, there's a 23% chance the other option would actually be preferred, because the two options are genuinely close.
The key lines. From code/mini-tradeoff/matrix.py:
# from code/mini-tradeoff/matrix.py
def critical_weight(options, crits, base_weights, criterion, mode="fixed", steps=2001):
"""Raise or lower one criterion's weight (the others keep their ratios) and report the weights at which the winner changes."""
...
for wj in np.linspace(0, 1, steps):
w = np.insert(rest * (1 - wj), j, wj)
wn = names[int(np.argmax(Z @ w))]
if prev is not None and wn != prev[1]:
flips.append({"weight": round(float(wj), 3), "from": prev[1], "to": wn})
Read it as: the function sweeps one criterion's weight from 0 to 1 while holding the others' ratios fixed, and records every point where the argmax winner changes. It is not searching for the "true" weight at all. It is mapping out the entire range of weights that would support each possible answer. That is the honest version of "defending your weights" in a review.
Critical weight, trade-off
A trade-off means improving one property (say latency) costs another (say quality), so you must choose how much of each to buy. The critical weight is the weight on one criterion at which the winning option changes.
Example: "cost weight never flips it" while "p95 weight below 0.37 flips it" tells a stakeholder exactly which of their priorities actually matters to this decision and which one is along for the ride -- a far more useful answer than simply stating the chosen weights and the resulting winner.