Small Fine-Tuned Model or LLM: Decide With Numbers

A team classifies 2,000,000 customer messages a month into 30 intents. Proposal A: call an LLM for every message. Proposal B: label 8,000 messages with humans, fine-tune a small encoder, and serve it.

Assumptions (all are placeholders for this exercise, not real prices or measurements):

• Proposal A: a 500-token prompt (instructions, label definitions, examples) plus 150 tokens of message and 10 output tokens per call; USD 0.80 per 1M input tokens, USD 3.20 per 1M output tokens; 1.5 s per call.
• Proposal B: labels at USD 0.25 each, 3 engineer-weeks at USD 4,000 per week to build, serving at USD 400 per month, 30 ms per message.
• Optional prompt caching for A: 80 percent of the 500-token prompt is a shared prefix billed at a 90 percent discount.

1. Compute the monthly cost of A without and with caching.
2. Compute the one-off cost of B and the break-even month against A in each case.
3. List what the numbers leave out, and the factors besides cost that could decide it.
4. State the hybrid design you would actually propose and what you would measure before committing.