Choosing Between RLHF and DPO for a Mid-Size Team

1. RLHF vs DPO for the core loop: DPO. The team has no dedicated RL infrastructure engineers, which is precisely the constraint that makes RLHF-with-PPO risky to run reliably — PPO is hyperparameter-sensitive and prone to instability without practitioners experienced in tuning it, and the RL loop requires serving a reward model plus running sampling-heavy on-policy rollouts, an operational burden a 15-person team without RL specialists is likely to under-resource. DPO needs the same preference data (40k comparisons/month is a reasonable, if modest, input) but trains in a single supervised-style pass — comparable operational complexity to the SFT step they're presumably already running. Given the constraints stated, DPO is the lower-risk choice for reliable monthly delivery.

2. Does best-of-N change the answer: It's the one requirement that pulls against DPO, and it's worth naming explicitly rather than ignoring: best-of-N re-scoring at inference time needs a standalone, continuously-queryable scorer that can rank arbitrary draft responses — exactly what a trained reward model is and what DPO, by construction, doesn't produce (DPO's preference signal is implicit in the policy's own log-probabilities, not a separately callable scoring function). This is a genuine trade-off, not a reason to default back to full RLHF-with-PPO — the team can still avoid the RL loop's instability while getting a usable scorer, which is the third part of the answer.

3. What to build instead: Train a standalone Bradley-Terry reward model on the same preference data used for DPO (the preference store is method-agnostic, so this doesn't require new data collection), and use it purely as an inference-time scorer for best-of-N — never feeding it into an RL loop. This gets the team a purpose-built scorer without taking on PPO's operational risk. The trade-off versus a full RLHF setup: this reward model is trained once per cycle on the same preference snapshot DPO uses and isn't continuously refined against an evolving on-policy distribution the way an RLHF pipeline's reward model implicitly is retrained/reused across RL iterations — so its ranking quality on drafts from a policy that has moved further from the training distribution (later in the cycle, or after several cycles without a reward-model refresh) will degrade faster than an RM embedded in an active RL loop. Mitigate by refreshing the reward model each cycle alongside the DPO policy and monitoring its ranking agreement with human judgment on a calibration slice, the same discipline used for LLM-judge calibration.