Match a job Paths Subjects Questions Quizzes Pricing
Advanced Open Pro

Comparing Two Proposals with Cost-Per-Successful-Task

A team runs 2,000 tasks/day through a document-processing agent. Proposal A: switch to a faster, cheaper model, dropping per-attempt cost from $0.40 to $0.22 but also dropping first-attempt success rate from 88% to 81% (failed tasks are retried once automatically at the same per-attempt cost, and if the retry also fails, the task is escalated to a human reviewer at $6 per case; assume retry success rate matches first-attempt rate for simplicity). Proposal B: keep the current model but add a validation step that catches and auto-corrects a class of formatting errors before they reach the user, raising first-attempt success from 88% to 93%, at an added cost of $0.05 per attempt.

  1. Compute cost-per-successful-task today (baseline, before either proposal), and for Proposal A and Proposal B.
  2. Which proposal actually wins on this metric, and does it match the proposal that looks better on raw success rate or on raw per-attempt cost alone?
  3. Name one thing cost-per-successful-task still doesn't tell you that you'd want to check before committing to either proposal.

Share this question

← Back to Evaluating Agents and Multi-Agent Systems practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.