Picking the Right Technique for Three Different Requests
Product brings you three separate requests in the same week:
- "Our JSON tool-call outputs are schema-valid only 85% of the time even with five few-shot examples in the prompt. We need 99%+."
- "We want the model to consistently prefer shorter, more direct answers over long hedging ones — we have pairs of past answers where our reviewers picked the better one, about 6,000 pairs."
- "We want to teach the model our company's refund policy so it stops asking customers to check the help centre."
For each, name the technique (SFT, LoRA/QLoRA, RLHF, DPO, or "none of these — use something else instead") you would recommend, and justify it in one or two sentences per case.
1. JSON tool-call formatting → SFT via LoRA/QLoRA.
This is a target-completion problem: for a given prompt there is a single correct shape of output (schema-valid JSON). SFT on a few thousand (prompt, correctly-formatted completion) pairs — sourced from real requests with a human or script correcting the malformed ones — directly optimizes for the pattern you want. LoRA is the right implementation: the target behaviour is narrow, the base model's other capabilities must not regress, and cheap iteration matters while tuning rank and data mix. RLHF/DPO would be the wrong tool: there is no comparative preference being expressed, just a correct target.
2. Preferring shorter, more direct answers → DPO.
This is exactly DPO's shape: 6,000 (prompt, chosen, rejected)
preference pairs already exist from real reviewer judgments, and the
target is comparative ("prefer this over that"), not a single gold
completion. DPO gets the RLHF-style preference-driven behaviour shift
directly from this data with a single training loop — no reward model,
no RL infrastructure needed. RLHF would work too but is unjustified
extra cost and instability for a well-defined preference-pair dataset
this size; plain SFT would require picking one "ideal" answer per
prompt, which discards the comparative signal you already collected.
3. Teaching the refund policy → none of these; use RAG (or a tool call).
This is a knowledge problem, not a behaviour problem — the policy is a fact that can and will change. Fine-tuning (of any of the four kinds) would bake the policy into weights with no citation, no way to tell a correct recall from a confabulated one, and no update path short of retraining every time the policy changes. Retrieve the current policy article at answer time and cite it; that is faster to update, cheaper to maintain, and auditable in a way none of the fine-tuning techniques can be. If the complaint is really "the model phrases the retrieved answer awkwardly by pointing customers to the help centre instead of answering directly," that narrower behaviour problem could later be an SFT/LoRA target layered on top of the RAG pipeline — but the policy content itself stays in retrieval.
Share this question