Evaluating Two Context Strategies for the Same Agent Task

Your team built two versions of the same code-migration agent: Version A pre-loads the full target module's documentation and every file it might touch at session start. Version B uses just-in-time reads and a sub-agent to isolate the test-suite run. On a set of 20 representative migration tasks, both versions reach a correct, passing final state on 18/20 tasks. A teammate concludes "they're equivalent, ship whichever is simpler to maintain."

1. Explain what this outcome-only comparison is missing, and what additional metrics you'd collect before agreeing they're equivalent.
2. Describe a plausible result (with made-up but reasonable numbers) where both hit 18/20 correctness but one is clearly the better choice for production. What would that result look like?
3. Why is a single run of these 20 tasks per version weaker evidence here than it would be for evaluating two versions of a single-turn prompt on 20 examples?