LLM Reinforcement Learning Algorithms Compared: PPO, DPO, GRPO, and Their Variants
During alignment training of large language models, policy optimization determines how preference signals are converted into parameter gradients that can be updated stably. In recent work on long-horizon tasks, some teams have leaned toward proximal policy optimization in complex multi-step generation settings rather than relying only on variants that use within-group relative rewards, prompting practitioners to re-examine the assumptions, variance sources, and applicability boundaries of each algorithm. This article takes a neutral technical perspective: starting from problem modeling, it walks through proximal policy optimization, direct preference optimization, group relative policy optimization, and subsequent refinements, and closes with common engineering considerations for production use.
1. Problem Formulation and Notation
Let the current policy be a conditional language model $\pi\theta(y{t+1}\mid x,y{\leq t})$, where $x$ is the prompt, $y{t+1}$ is the $(t+1)$-th generated token, $y{\leq t}$ is the prefix generated so far, and $\theta$ holds the policy parameters. In reinforcement learning terms, the state at step $t$ is $s_t=(x,y{\leq t})$ and the action is the next token. Trajectories are produced by a sampling policy, and rewards may come from a reward model, rule-based checks, process supervision, or outcome verification.
The central difficulty in long-horizon tasks is that the useful signal often appears only sparsely at the end of the sequence; credit assignment across intermediate steps is hard; and updates that are too large easily damage language ability the model has already acquired.
For this reason, most algorithms have to address three things at once: expressing “the better action” through an advantage or a preference margin; controlling update magnitude through importance sampling or within-group normalization; and reducing variance while preventing policy drift through a value function, an entropy term, or a relative-entropy (KL) constraint.
2. The Full PPO Objective
Proximal policy optimization, widely used in early instruction-alignment practice, generally combines a policy term, a value term, an entropy bonus, and a KL penalty. The training objective minimizes:
[\mathcal{L}{\mathrm{PPO}}(\theta)=\mathcal{L}{\mathrm{policy}}(\theta)+cv\mathcal{L}{\mathrm{value}}(\theta)-ce\mathcal{L}{\mathrm{entropy}}(\theta)+\beta\mathcal{L}_{\mathrm{KL}}(\theta).]
The policy loss raises the probability of high-advantage actions while clipping restricts deviation of the ratio:
[\mathcal{L}_{\mathrm{policy}}(\theta)=-\mathbb{E}_t\left[\min\left(r_t(\theta)A_t,\operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)A_t\right)\right],]
where $rt(\theta)=\frac{\pi\theta(at\mid s_t)}{\pi{\theta_{\mathrm{old}}}(a_t\mid s_t)}$.
The purpose of clipping is twofold: when the advantage is positive, the ratio is not allowed to grow so large that a single update becomes too aggressive; when the advantage is negative, the ratio is not allowed to shrink so far that the probability is crushed. The policy therefore makes local improvements within a trust-region-like neighborhood, which matters especially for long sequences.
The advantage $At$ can be estimated in several ways. If per-step rewards $r_t$ exist, one can use the discounted return $R_t=\sum{k=t}^{T}\gamma^{k-t}r_k$ and set $A_t=R_t-V(s_t)$. The value network $V$ depends only on the state and serves as a baseline that lowers variance; as long as the subtracted quantity is independent of the next action, the expectation of the policy gradient is unchanged. It is worth stressing that the advantage in the policy loss is typically detached from the graph, while the value network learns through a separate regression term. The value loss is usually a squared error against the return or against a generalized advantage estimation target, and clipping is sometimes applied there as well to avoid violent swings in the value function. The entropy term encourages exploration, and the KL term pulls the policy back toward the reference model, guarding against reward hacking and collapse of language quality.
For long-horizon tasks, PPO’s advantage is that an explicit value function can score intermediate states, yielding finer credit assignment, while clipping and the KL term jointly limit drift. The cost is an extra value head or value model, more complex implementation and tuning, and the fact that bias in value estimates contaminates the advantage.
3. Direct Preference Optimization: Skipping Explicit Rewards and Values
Direct preference optimization starts from pairwise preference data and merges reward modeling with policy optimization. Its key observation is that if the reward is defined as the log ratio of the policy to a reference policy, then under the Bradley-Terry preference model there exists a closed-form optimal policy. One can therefore directly maximize the log odds of “chosen response over rejected response” within each preference pair, without online sampling or a value network. The method is simple to implement and friendly to offline pipelines, and suits settings with ample preference data and moderate sequence lengths. Its limitations are equally clear: generalization to out-of-distribution prompts depends on data coverage; without per-step advantages, process supervision is weak under sparse long-horizon rewards; and the implicit reward may overfit annotation bias.
In engineering practice this is usually combined with length normalization, a margin term, and a reference policy that is either frozen or refreshed periodically. If only outcome-level correct/incorrect signal is available and no pairwise preferences exist, pseudo-preferences must be constructed first, which imports construction error into the optimization.
4. Group Relative Policy Optimization and Its Motivation
Group relative policy optimization targets the setting where a group of responses is sampled per prompt and their relative performance inside the group serves as the advantage. For each prompt, $G$ complete trajectories are sampled; the group’s reward mean and standard deviation are computed; and the standardized scores become the advantage shared across the whole sequence or across steps. The result is then fed into a PPO-like ratio-clipping loss. The benefits are practical: no value network needs to be trained, so the implementation is lighter; the within-group comparison forms a natural baseline that reduces sensitivity to the absolute reward scale; and it works very well with verifiable rewards such as correctness or rule satisfaction.
For long-horizon tasks, however, the weaknesses of group relative methods become clearer. First, the advantage is often broadcast at the sequence level to every token, so an erroneous intermediate step and a correct one share the same sign, making credit assignment coarse. Second, when within-group variance is too small or too large, standardization becomes unstable, which calls for clipping the standardized scores and filtering out all-correct or all-incorrect groups. Third, without a state value, it is hard to assess intermediate decisions such as “the prefix is already good, so continue conservatively.” This also explains why some long-horizon efforts return to PPO with a value function: when the process is long and local decisions are dense, a per-step baseline carries more information than a single group-level scalar.
5. Later Variants: Dynamic Sampling, Decoupled Clipping, and Sequence-Level Optimization
Follow-up work on top of the group relative framework mainly addresses three issues: sample efficiency, clipping symmetry, and the mismatch between token-level and sequence-level objectives.
Dynamic sampling methods focus on the fact that not every prompt has learnable within-group variation. If the rewards in a group are nearly identical, the gradient is close to noise. Such methods increase the sampling ratio for hard cases, discard zero-variance groups, and allocate compute dynamically according to how unsolved a prompt is. This improves throughput effectively on verifiable tasks.
Decoupled clipping and adaptive constraint methods focus on the fact that standard symmetric clipping treats positive and negative advantages identically, yet in a language model the numerical ranges of “raising the probability of the correct path” and “lowering the probability of the wrong path” differ. Decoupling the upper and lower clipping thresholds, or making the KL coefficient adaptive, helps reduce the coexistence of over-updating and under-updating. Higher-order loss aggregation, such as a more robust reduction over token losses, can also ease the situation where a few tokens dominate the gradient in long sequences.
Sequence-level policy optimization emphasizes that what is actually evaluated is the entire output, while a product of token-level importance ratios becomes numerically unstable over long sequences. Defining the ratio on the sequence probability, or on a length-normalized sequence probability, and then clipping against a sequence advantage makes the objective consistent with how results are measured, and mitigates ratio explosion or decay across steps. Such methods align more closely with outcome-level “right or wrong” rewards, but for tasks that need fine-grained process supervision, it is still advisable to combine them with per-step rewards or a process value.
6. How to Choose: Criteria That Matter for Long-Horizon Tasks
Choosing an algorithm does not require taking sides; the signal structure should drive the decision. If a reliable process reward or a trainable value function is available and the sequences are long with critical intermediate decisions, PPO with clipping is usually the more stable option, because the advantage can vary with the state. If the reward is only given at the end and a distinguishable group of samples can be drawn for the same prompt, group relative methods are cheap to implement and fast to iterate. If large-scale pairwise preference data already exists and alignment must happen offline, direct preference optimization is the better fit. If many prompts in training carry no discriminative signal, introduce dynamic sampling. If sequence lengths vary widely and token-level ratios are unstable, consider a sequence-level objective or length normalization.
Common engineering points include: the refresh frequency of the reference policy and the old policy; generation temperature and group size; reward scale and handling of outliers; gradient clipping and learning-rate warmup; and periodic checks on a held-out set for fluency, instruction following, and reward hacking. If a value function is used, avoid overfitting it to noisy returns. If group relative optimization is used, handle all-correct and all-incorrect groups and bound the standardized scores. Whatever the algorithm, a KL term or an explicit deviation penalty should be treated as a default safety valve.
7. Assembling the Loss in Practice
In practice, the training loop can be split into four stages: sampling, rewards and advantages, policy update, and diagnostics. In the sampling stage, keep the old policy fixed so that the importance ratio is well defined. In the advantage stage, make the gradient cutoff explicit. In the update stage, also record the clip-trigger rate: if it almost never triggers, the step size is too small or $\epsilon$ is too large; if it triggers constantly, updates are too aggressive. In the diagnostics stage, look beyond mean reward at response length, repetition rate, KL divergence, value error, and within-group standard deviation. For long-horizon tasks, it is also advisable to score key intermediate checkpoints separately, so that optimizing the end metric does not come at the expense of process reliability.
Illustrative pseudocode:
for batch in dataloader:trajs = sample(policy_old, prompts, n=G)rewards = score(trajs) # rule-based or reward model, gradients stoppedadv = estimate_advantage(rewards, values=optional)ratio = pi_theta / pi_oldpolicy_loss = -mean(min(ratio*adv, clip(ratio)*adv))total = policy_loss + cv*value_loss - ce*entropy + beta*kloptimize(total)
Here estimate_advantage can be swapped for generalized advantage estimation, within-group standardization, or sequence-level standardization, which makes it possible to switch algorithm families within a single framework.
8. Summary
Proximal policy optimization offers a complete online optimization paradigm with a value baseline and trust-region clipping, and fits long-horizon, process-intensive tasks. Direct preference optimization absorbs the preference model into the policy likelihood and fits offline pairwise data. Group relative policy optimization uses within-group comparison to eliminate the value network and fits verifiable rewards and fast iteration. Dynamic sampling, decoupled clipping, and sequence-level objectives are refinements to stability, sample efficiency, and objective consistency. Understanding the statistical meaning of each loss term matters more than memorizing the names: what really determines long-horizon performance is whether the advantage assigns credit to the correct time steps, and whether updates are confined to a neighborhood in which the policy will not collapse.
