DPO, IPO, and offline preference optimization

PPO-RLHF requires a trained reward model and a value model, making the training pipeline complex and memory-intensive. Direct Preference Optimization (DPO, Rafailov et al. 2023) shows that the optimal policy under the RLHF objective can be derived in closed form, and the reward model can be eliminated entirely -- the policy is trained directly on preference pairs with a simple classification-like objective[1].

Deriving DPO

Start from the KL-penalized RLHF objective[2]:

The optimal policy has a closed-form solution (derived by setting the functional derivative to zero):

where $Z(x) = \sum_y \pi_{\text{ref}}(y|x) \exp(r(x,y)/\beta)$ is the partition function.

Rearranging: given the optimal policy, we can express the reward in terms of the policy:

Key insight: $\beta \log Z(x)$ does not depend on $y$, so it cancels in the Bradley-Terry preference probability:

Replace $\pi^*$ with the learned policy $\pi_\theta$ and maximize likelihood:

This is the DPO objective: binary cross-entropy on preference pairs, using the log-ratio of the policy to the reference model as an implicit reward.

What DPO does intuitively

The quantity $\log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$ measures how much more (or less) likely the policy makes response $y$ relative to the SFT baseline.

DPO maximizes $[\text{implicit reward of } y_w] - [\text{implicit reward of } y_l]$:

• It increases the relative probability of preferred responses.
• It decreases the relative probability of dispreferred responses.
• The $\beta$ controls the scale -- large $\beta$ keeps changes close to the reference.

The loss is binary cross-entropy on the margin $\hat r_w - \hat r_l$, where $\hat r = \beta \log \frac{\pi_\theta}{\pi_{\text{ref}}}$. Watching that margin makes the scale concrete (with the per-pair loss $-\log\sigma(\text{margin})$):

• At initialization, $\pi_\theta = \pi_{\text{ref}}$, so both log-ratios are $0$, the margin is $0$, and the loss is $-\log\sigma(0) = -\log\tfrac12 \approx 0.693$ nats. Every pair starts at the same value, the entropy of a coin flip.
• A modest correct nudge. Suppose training raises $\log\pi_\theta(y_w)$ by $+3$ nats over the reference and lowers $\log\pi_\theta(y_l)$ by $2$ nats. With $\beta = 0.1$ the implicit rewards are $\hat r_w = 0.3$ and $\hat r_l = -0.2$, a margin of $0.5$, and the loss drops to $-\log\sigma(0.5) \approx 0.474$. The same probability shifts at $\beta = 1$ would give a margin of $5$ and loss $\approx 0.0067$: $\beta$ rescales how loudly a fixed change in token probabilities registers.
• Saturation. Once the margin reaches $\approx 4$, $\sigma(4) \approx 0.982$ and the loss is $\approx 0.018$; pushing further yields almost no gradient. The objective stops driving a pair apart once the preferred response is already clearly ahead, which is what keeps DPO from collapsing the policy onto the chosen samples.

The confusion to name: the implicit reward $\hat r$ is not a log-probability of $y$, but $\beta$ times a log-ratio against the frozen reference. A response can have low absolute likelihood and still earn a high implicit reward if the policy raised it far above where the SFT model put it.

Implementation

import torch
import torch.nn.functional as F

def dpo_loss(policy, ref_model, prompts, y_w, y_l, beta=0.1):
    """
    policy, ref_model: callable (prompt, response) -> log-probability
    prompts, y_w, y_l: batched inputs
    """
    def log_prob(model, prompt, response):
        # In practice: compute log p(response | prompt) by summing token log-probs
        return model.log_probs(prompt, response)   # (B,)

    log_pi_w   = log_prob(policy,    prompts, y_w)
    log_pi_l   = log_prob(policy,    prompts, y_l)
    log_ref_w  = log_prob(ref_model, prompts, y_w)
    log_ref_l  = log_prob(ref_model, prompts, y_l)

    implicit_r_w = beta * (log_pi_w - log_ref_w)
    implicit_r_l = beta * (log_pi_l - log_ref_l)

    loss = -F.logsigmoid(implicit_r_w - implicit_r_l).mean()
    return loss

IPO: identity preference optimization

DPO assumes the Bradley-Terry model, which may not hold for all preference datasets. IPO (Azar et al., 2024) derives a variant that is more robust to reward hacking and works without the BT assumption by directly fitting the log-ratio:

This is a regression target: push the difference in log-ratios to $1/(2\beta)$, a fixed margin rather than maximizing the BT objective without bound.

KTO: Kahneman-Tversky optimization

KTO (Ethayarajh et al., 2024) does not require preference pairs $(y_w, y_l)$ -- only binary labels (good/bad). This is valuable when pairwise preference collection is expensive.

The objective aligns with human decision-making under prospect theory: losses are felt more acutely than equivalent gains, matching Kahneman-Tversky's reference-point model.

ORPO: odds ratio preference optimization

ORPO (Hong et al., 2024) removes the reference model entirely by combining SFT and preference optimization in a single objective:

where $\text{odds}_\theta(y|x) = p_\theta(y|x) / (1 - p_\theta(y|x))$.

ORPO uses the ratio of odds rather than log-ratios to a reference, making it a true single-stage training method with no SFT warm-up required.

Comparison

Exercises

Exercise 1. During DPO training with $\beta = 0.1$, a chosen response has its policy log-probability raised by $+3$ nats over the reference and a rejected response lowered by $2$ nats. Compute the two implicit rewards, the margin, and the per-pair DPO loss $-\log\sigma(\text{margin})$ in nats.

Exercise 2. Derive the gradient of the DPO loss with respect to the margin $u = \hat r_w - \hat r_l$, and show that it scales with $\sigma(-u) = 1 - \sigma(u)$. Use this to explain the saturation noted in the lesson: that a pair with margin $\approx 4$ gets almost no gradient.

Exercise 3 (code). Implement the DPO loss in numpy directly from policy and reference log-probabilities (four scalars per pair). Verify the Exercise 1 numbers, and check that adding the same constant to both chosen log-probs (policy and reference together) leaves the loss unchanged, since only the log-ratio enters.

import numpy as np

def logsigmoid(z):
    # numerically stable log sigmoid
    return -np.logaddexp(0.0, -z)

def dpo_loss(logp_w, logref_w, logp_l, logref_l, beta=0.1):
    r_w = beta * (logp_w - logref_w)        # implicit reward, chosen
    r_l = beta * (logp_l - logref_l)        # implicit reward, rejected
    margin = r_w - r_l
    return -logsigmoid(margin), margin

# Exercise 1: chosen log-ratio +3, rejected log-ratio -2, beta = 0.1.
logref_w, logref_l = -1.0, -1.0
logp_w = logref_w + 3.0                      # raised by +3 nats
logp_l = logref_l - 2.0                      # lowered by 2 nats

loss, margin = dpo_loss(logp_w, logref_w, logp_l, logref_l, beta=0.1)
assert np.isclose(margin, 0.5)
assert np.isclose(loss, -np.log(1 / (1 + np.exp(-0.5))))

# Only the log-ratio matters: shift policy and reference chosen log-probs together.
c = 2.5
loss_shift, _ = dpo_loss(logp_w + c, logref_w + c, logp_l, logref_l, beta=0.1)
assert np.isclose(loss_shift, loss)
print("DPO loss:", round(float(loss), 4), "margin:", round(float(margin), 4))

DPO and its variants are now the dominant preference optimization methods for open-source LLMs due to their simplicity, stability, and competitive performance with PPO.