CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 5 of 12 · 2 min read

Partial credit helps you debug, but it can't see a wrong final answer

A path-similarity score that ignores the final state will happily give full marks to a run that did everything right except get the number wrong.

Think of grading a math proof purely on how many of the expected steps appear in the right order, while never checking whether the final line is actually the number the problem asked for. A student who shows all the right working but copies the wrong final digit would still get full marks on the working alone.

In plain words: a milestone is a checkpoint the run should reach, and the share of milestones reached (measured here by the longest common subsequence against the reference) gives a partial-credit score -- useful for debugging where a run went off track, but blind to whether the final outcome was actually correct, since a path can hit every milestone and still end on a wrong value.

A worked example. A run that follows the exact reference path but refunds the wrong amount gets a milestone score of 1.0 (every reference step is present, in order) and a precision, recall and F1 of 1.0 on the tool set -- every single path-based diagnostic says this run is perfect. Yet the outcome, the actual refund amount, is wrong. None of the path metrics have any way to see that, because none of them look at the final state at all.

The key lines. From code/mini-trajectory/trajectory.py:

# from code/mini-trajectory/trajectory.py
def lcs_len(a, b):
    dp = [[0] * (len(b) + 1) for _ in range(len(a) + 1)]
    for i in range(len(a)):
        for j in range(len(b)):
            dp[i + 1][j + 1] = dp[i][j] + 1 if a[i] == b[j] else max(dp[i][j + 1], dp[i + 1][j])
    return dp[-1][-1]


def milestone_score(path, ref=REF):
    """Share of reference steps reached in order (partial credit)."""
    return lcs_len(path, ref) / len(ref)

Read it as: milestone_score only ever looks at the two lists of step names, path and ref. It never receives or checks the final refund amount, the outcome, at all. That is not an oversight. It is the nature of a path metric. It can only answer "did the steps look right?" Answering "was the actual result right?" requires a separate outcome check layered on top.

Milestone

A milestone is a checkpoint the run should reach; the share reached is a partial-credit score.

Example: the wrong-amount run scoring a perfect 1.0 on milestones while being outright wrong on the outcome is the clearest possible demonstration that milestone scores are a diagnostic for locating where a run went off track, never a substitute for checking whether it actually got the right answer.