Why DPO Keeps a Frozen Reference Copy of the Policy

DPO's loss function includes a frozen reference-model copy of the policy alongside the model being trained, and the loss compares chosen/rejected log-probabilities under both. A teammate asks: "Why not just drop the reference model and directly maximize the log-probability gap between chosen and rejected under the model we're training — wouldn't that converge faster?" What is the primary role the frozen reference model plays in DPO's objective?

A. It acts as an implicit KL-divergence anchor, penalizing the policy for drifting too far from the reference distribution as it fits the preference pairs. B. It supplies the reward signal, since DPO trains a reward model internally using the reference copy's outputs as negative examples. C. It is only used to initialize the trained policy's weights before training starts, and plays no role in the loss afterward. D. It generates additional synthetic preference pairs during training to augment the 15,000 labeled pairs.