Advanced
Open
Free
Prompt, RAG or Fine-Tune?
A B2B SaaS company wants an assistant that answers questions about its product. Three complaints have come in from a prototype built with a plain system prompt:
- (a) It confidently describes settings that no longer exist since last quarter's release.
- (b) Answers are in the wrong tone and ignore the required "Summary / Steps / Related links" layout about 30% of the time even with examples in the prompt.
- (c) It gives generic advice about "your admin panel" instead of the customer's actual configuration.
- For each complaint, decide whether prompt engineering, RAG, or fine-tuning is the right lever, and justify it.
- Explain why fine-tuning is the wrong fix for complaint (a).
- Sketch the order in which you would apply the fixes and how you would know each one worked.
Solution
1. Lever per complaint
- (a) Stale facts → RAG. The failure is missing/outdated knowledge. Index the current documentation (with version metadata), retrieve the relevant sections per query, and instruct the model to answer only from them. Freshness then follows the index, not the model weights.
- (b) Tone and layout unreliability → prompt first, then fine-tune. This is a behaviour/format problem. Tighten the instructions and add 2–3 explicit few-shot examples; if reliability still sits at ~70% after that, a small SFT/LoRA run on a few thousand well-formatted answers is the tool that makes format adherence a property of the model rather than of the prompt. It also lets you drop the few-shot examples and shorten the prompt.
- (c) Generic advice → RAG over per-customer data (plus tool calls). The model needs the customer's configuration in context. Retrieve or fetch it (with strict tenant filters) and put it in the prompt. Nothing about training helps here; the data is per user and changes.
2. Why fine-tuning is wrong for (a)
Fine-tuning shapes behaviour; it does not reliably store facts. A model fine-tuned on documentation still hallucinates specifics, cannot cite a source, and goes stale the moment the docs change — you would have to retrain on every release. RAG updates instantly when the index does and supports citations and an "I don't know" path.
3. Order and verification
- Build the RAG pipeline for (a) and (c); verify with retrieval recall@5 on a gold set of (question, correct doc section) and a judge/human faithfulness score.
- Improve prompt and few-shot examples for (b); measure format validity rate on the offline eval set (target >95%).
- Only if (b) still fails, fine-tune (LoRA) on curated examples; re-run the full regression set to confirm no drop in correctness or groundedness, then A/B online on resolution rate and CSAT.
Share this question