Controlling the Cost and Capacity of an LLM Review Tier

Your cascade sends the top 4 percent of 5 million daily posts (200,000 posts) by expected loss to an LLM tier. Assumptions, not measurements: 350 input and 40 output tokens per call; a small model priced at USD 0.50 per million input tokens and USD 2.00 per million output tokens; a large model at USD 3 and USD 15; a human review costs USD 25 per hour at 45 seconds per item. Without the LLM tier the system would send 71,480 items a day to humans; with it, 21,010.

Simulated marginal savings (cost units per extra call, placeholder scale): 1.05 going from 2 percent to 4 percent of posts, 0.41 from 4 to 8 percent, 0.089 from 8 to 16 percent, 0.018 from 16 to 32 percent.

1. Daily token cost for the small and large models at 4 percent, and if you routed every post.
2. How does that compare with the review labor saved?
3. Compute average and peak throughput and the concurrency you need at a 1.2 second latency.
4. How do you choose the routing budget, and what controls do you add?