Intermediate
Open
Pro
Propose a Cost-Reduction Plan for a Runaway Agent Loop
Your company's coding agent has grown from an internal prototype to a product feature, and its LLM bill has grown with it. Current shape, per completed task:
- Average task takes 25 agent-loop steps to complete.
- Each step resends the full system prompt + tool schemas (3,500 tokens, currently NOT cached — the team never enabled it) plus the entire growing transcript of the task so far (starts at 0, grows by ~600 tokens/step) plus that step's new tool call/result (~400 tokens) plus ~150 tokens of model output per step.
- All steps use one large model at $10/1M input, $40/1M output.
- 8,000 completed tasks/day.
- Compute the current daily cost (input and output separately) for one representative task, then scale to 8,000 tasks/day. Show the arithmetic — you can approximate the growing transcript as an arithmetic series.
- Propose a concrete plan using at least three levers from this subject, and estimate the new cost per task after applying them. State your assumptions for any prices/ratios you introduce.
- Which lever from your plan would you implement first, and why — what makes it the highest-leverage, lowest-risk first move for an agent loop specifically, versus for a single-shot chatbot?
Share this question