Part 3 of 12 · 2 min read
A single reference path rejects valid alternatives
Demanding the exact same sequence of steps every time punishes an agent for doing two independent things in a different, equally fine order.
Think of grading a recipe by requiring the cook to chop the onions before the garlic, when the dish comes out identical either way. A rule that strict would fail a perfectly good cook on a distinction that never mattered.
In plain words: exact match requires the steps to equal the reference exactly, in-order match only requires the reference steps to appear somewhere in that order (extra steps allowed), and any-order match only requires every reference step to appear at all, in any sequence -- so the stricter the matching rule, the more valid variations it will wrongly reject.
A worked example. A run that swaps two genuinely independent steps (verifying identity before looking up the order, instead of after) is just as correct as the reference path -- neither order matters for the task. Yet exact match fails it, and in-order match fails it too, because the reference's specific ordering of those two steps isn't what the swapped run produced. Only a rule-based scorer that explicitly allows the two steps in any order accepts it.
The key lines. From code/mini-trajectory/trajectory.py:
# from code/mini-trajectory/trajectory.py
def exact_match(path, ref=REF):
return path == ref
def in_order_match(path, ref=REF):
"""Reference steps appear in order; extra steps allowed."""
it = iter(path)
return all(any(p == r for p in it) for r in ref)
def any_order_match(path, ref=REF):
return set(ref) <= set(path)
Read it as: each function is a progressively looser test. It uses == for exact, an ordered scan for in-order, and a plain set-containment check for any-order, which throws away ordering information entirely. The three functions form a spectrum. At one end is "accepts almost nothing but the exact reference." At the other end is "accepts any sequence that happens to include every needed step." Picking which one to use is picking how much legitimate variation you are willing to tolerate.
Exact, in-order and any-order match
Exact match: the steps equal the reference. In-order match: the reference steps appear in order, extra steps allowed. Any-order match: all reference steps appear, in any order.
Example: the swapped-order run failing both exact and in-order match while passing a rules-based check is the direct proof that a single reference path is often the wrong tool -- the task has more than one acceptable way to do it, and only a method that encodes rules rather than one fixed sequence can tell the difference between "wrong" and "different but fine."