The Cheapest First Lever Against Catastrophic Forgetting
A team is about to full-parameter SFT a base model on a narrow, high-volume internal dataset (200,000 examples of one task type) and is worried about catastrophic forgetting of the model's general capabilities. Before reaching for more complex mitigations, what is the single most standard, lowest-cost first lever to reduce catastrophic forgetting in this setup?
A. Mix a broad-coverage general-instruction dataset into the training batches alongside the narrow task data, rather than training on the narrow data alone. B. Switch from full-parameter fine-tuning to LoRA, since LoRA mathematically guarantees zero forgetting on any task. C. Increase the learning rate so the model converges on the narrow task faster, spending less total training time exposed to drift. D. Train for more epochs on the narrow dataset so the loss fully converges before evaluating for forgetting.
Share this question