Diagnosing a Fine-Tuned Model That Memorized the Wrong Things

A team fine-tuned (full-parameter SFT, not LoRA) their support-bot base model on 50,000 resolved tickets, hoping to improve both tone and factual accuracy on common policy questions, and retired their RAG pipeline to cut latency and cost. Three weeks after launch:

• The model's tone and formatting are noticeably better.
• Support is fielding complaints that the bot confidently states outdated pricing on two plans that changed since the training data was collected.
• QA can't tell, by reading an answer, whether a given factual claim is currently correct or stale, because every answer reads equally confident.

1. Explain why this happened, mechanically — why did fine-tuning improve tone but not keep facts current?
2. What would you change architecturally to fix this, and why would you keep some form of fine-tuning rather than dropping it entirely?
3. What evaluation gate, if it had existed before launch, would have caught this?