CSE Roadmap

Common tier · about 5 hours · 13 parts

Part 5 of 13 · 2 min read

Fix the scale before you see the results, and write the reversal down

Rescaling a comparison to fit whichever options happen to be on the table can flip the winner for no reason connected to the options themselves.

Think of grading a race by stretching the finish times so the slowest runner present always gets a zero. Add a slower runner to the race later, and everyone else's grade changes too. Their own performance never did.

In plain words: a rank reversal is the order of two options changing when a third, irrelevant option is added or removed, and it happens when a criterion is normalised by min-max scaling against whichever options happen to be present -- fixing the scale to real-world targets in advance, rather than rescaling by what's on the table, is what keeps a comparison from depending on options that were never going to win anyway; the decision and this reasoning then belong in a decision record for others to check later.

A worked example. Comparing two options A and B at weights 0.6/0.4 with min-max scaling, A wins. Adding a third option, D, that is dominated by both A and B (so it should never be chosen) and rescaling min-max against all three flips the winner to B -- purely because D's presence stretched the scale underneath A and B. Using fixed targets instead (a (worst, best) range set in advance, independent of which options show up) gives B as the winner whether or not D is in the comparison at all; the answer doesn't move just because an irrelevant option was added.

The key lines. From code/mini-tradeoff/matrix.py:

# from code/mini-tradeoff/matrix.py
def normalise(X, crits, mode="fixed"):
    """Scores in [0,1], 1 = best. 'minmax' rescales by the options present (its result changes when options are added);
    'fixed' uses each criterion's (worst, best) target, so the other options do not matter."""

Read it as: the docstring states the exact failure mode in its own second clause. "Its result changes when options are added" is precisely what a rank reversal is, named before any flip happens. That is why the module defaults to "fixed" scaling. The fix for a rank reversal is choosing the mode that structurally cannot produce one, not catching it after the fact.

Rank reversal, architecture decision record

A rank reversal is the order of two options changing when a third, irrelevant option is added or removed. A decision record (ADR) is a short document stating a decision, its context, alternatives, evidence, consequences and the condition for revisiting it.

Example: a decision record's "Evidence" section stating the fixed (worst, best) targets used, rather than just the final scores, is what lets someone months later check that the comparison wasn't quietly vulnerable to a rank reversal -- which is exactly the kind of thing the module's own seven-field ADR template and linter check for.