Match a job Paths Subjects Questions Quizzes Pricing
Advanced Open Pro

Credit Assignment in a Three-Stage Research Pipeline

A research pipeline has three agents in sequence: Agent 1 (search and retrieve source documents), Agent 2 (extract and summarize key claims from the retrieved documents), Agent 3 (write a final report citing the claims). On a failed trajectory, the final report cites a statistic that is wrong. Investigation shows: Agent 1 retrieved the correct source document containing the correct statistic; Agent 2's summary of that document states the statistic correctly; Agent 3's final report states a different, incorrect number for the same claim, attributed to the same source.

  1. Using the per-agent isolated-check method, identify which agent is responsible for the failure, and show how you'd have arrived at that conclusion mechanically from trace data alone (not guessing from the final wrong answer).
  2. A teammate says "Agent 1 is at fault, because if it had retrieved a source with the number pre-highlighted more clearly, Agent 3 wouldn't have gotten it wrong." Evaluate this claim.
  3. Propose the specific check you'd add to this pipeline's regression suite so this exact failure mode can't silently recur.

Share this question

← Back to Evaluating Agents and Multi-Agent Systems practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.