Comparing Two Proposals with Cost-Per-Successful-Task
A team runs 2,000 tasks/day through a document-processing agent. Proposal A: switch to a faster, cheaper model, dropping per-attempt cost from $0.40 to $0.22 but also dropping first-attempt success rate from 88% to 81% (failed tasks are retried once automatically at the same per-attempt cost, and if the retry also fails, the task is escalated to a human reviewer at $6 per case; assume retry success rate matches first-attempt rate for simplicity). Proposal B: keep the current model but add a validation step that catches and auto-corrects a class of formatting errors before they reach the user, raising first-attempt success from 88% to 93%, at an added cost of $0.05 per attempt.
- Compute cost-per-successful-task today (baseline, before either proposal), and for Proposal A and Proposal B.
- Which proposal actually wins on this metric, and does it match the proposal that looks better on raw success rate or on raw per-attempt cost alone?
- Name one thing cost-per-successful-task still doesn't tell you that you'd want to check before committing to either proposal.
Share this question