LLM RL Algorithms from PPO to GSPO: Principles, Trade-offs, and How to Choose

    As large-model post-training shifts from "can it align" to "can it learn long-horizon capability," algorithm choice has moved back into the spotlight. This article walks through five reinforcement learning methods for LLMs—PPO, DPO, GRPO, DAPO, and GSPO—starting from a unified notation for treating autoregressive generation as a Markov decision process. It breaks down each method's objective, gradient behavior, and cost: PPO's four-loss structure and value network, DPO's closed-form bypass of explicit rewards and online sampling, GRPO's group-relative advantage that removes the critic, DAPO's targeted fixes for GRPO's failure modes, and GSPO's sequence-level importance ratio. It closes with a decision table for choosing a method by reward structure and task length, plus a monitoring checklist for large-scale training. Readers will gain a clear picture of the trade-offs behind each algorithm and practical guidance on when to use which.

    1. Why “which algorithm to choose” is back in focus

    Over the past year, the focus of large-model post-training has shifted from “can it align” to “how do we train long-horizon capability.” Across tasks such as multi-turn tool use, repository-level code edits, and long chains of reasoning, one pattern has been repeatedly confirmed: small differences at the algorithm level get amplified over long sequences into large differences in training stability. Some teams have revived the classic PPO route with a value network (critic) for long-horizon agent tasks instead of the lighter group-relative approach. That choice alone shows that algorithm selection is not a matter of style—it is an engineering problem jointly determined by the reward structure, sequence length, and the difficulty of credit assignment.

    To compare PPO, DPO, GRPO, DAPO, and GSPO side by side, you first need a unified notation. Otherwise, since every paper uses different notation, the discussion easily stays at the level of terminology.

    2. A unified notation: modeling language generation as a Markov decision process

    Let the policy be an autoregressive language model π_θ(y_{t+1} | x, y_{≤t}):

    • x: the prompt or task instruction;
    • y_{≤t}: the first t tokens already generated;
    • θ: the current policy parameters being optimized.

    Mapped to standard RL notation, the state at step t is s_t = (x, y_{≤t}), and the action is the next token a_t = y_{t+1}. The full generation trajectory τ = (s_0, a_0, ..., s_{T-1}, a_{T-1}) forms an episode, and its probability is the product of the per-step conditional probabilities:

    1. P(τ) = Π_t π_θ(a_t | s_t)

    This definition has three direct consequences, which are also the root of all the algorithm disagreements that follow:

    1. The state space is nearly infinite and cannot be reused. Every token produces a new state; two trajectories from the same prompt diverge from the very first token, so you cannot rely on large-scale state revisiting to reduce variance the way classic RL does.
    2. Rewards are extremely sparse. In most alignment settings there is only a single scalar reward at the end of the sequence (a reward-model score, or whether a rule check passes), the intermediate token rewards are zero, and credit assignment depends entirely on the advantage estimation method.
    3. Sampling cost is asymmetric between the new and old policies. The token-level probability ratio π_θ/π_θ_old, multiplied over hundreds or thousands of steps, accumulates enormous variance—and this is the key to understanding GSPO.

    3. PPO: laying out all four losses

    3.1 Overall objective

    The PPO widely used in the InstructGPT/ChatGPT generation of RLHF minimizes a total loss made up of four terms:

    1. L_PPO(θ) = L_policy(θ) + c_v · L_value(θ) - c_e · L_entropy(θ) + β · L_KL(θ)
    • L_policy: the policy gradient term, which decides “which direction to update in”;
    • c_v · L_value: the value network’s regression error, which ensures the advantage estimate has a reliable baseline;
    • - c_e · L_entropy: the entropy bonus, which suppresses premature collapse into determinism (note the minus sign—we are maximizing entropy);
    • β · L_KL: a trust-region constraint against the reference model, preventing the policy from drifting off.

    3.2 Policy loss and the clipping mechanism

    The goal of the policy loss is to increase the probability of high-advantage tokens. Since we want the objective to be larger-is-better but the framework minimizes, we add a minus sign:

    1. L_policy(θ) = - E_t [ min( r_t(θ)·A_t , clip(r_t(θ), 1-ε, 1+ε)·A_t ) ]

    where the importance sampling ratio is r_t(θ) = π_θ(a_t|s_t) / π_θ_old(a_t|s_t), and A_t is the advantage estimate for that token.

    The gradient behavior of clipping deserves a separate explanation, because it directly determines “which samples actually produce a gradient.” Differentiating the expression above:

    • When A_t > 0 and r_t > 1+ε: the upper bound is hit, and the gradient is 0 (the token has already been pushed high enough and is no longer updated);
    • When A_t > 0 and r_t ≤ 1+ε: the gradient is -A_t · r_t · ∇_θ log π_θ(a_t|s_t);
    • When A_t < 0 and r_t < 1-ε: the lower bound is hit, and the gradient is 0 (the token has already been pushed low enough);
    • When A_t < 0 and r_t ≥ 1-ε: the gradient is -A_t · r_t · ∇_θ log π_θ(a_t|s_t).

    These can be combined into:

    1. ∇_θ L_policy = - E_t [ 𝟙{not clipped} · A_t · r_t · ∇_θ log π_θ(a_t|s_t) ]

    Two practical implications: first, using min rather than the raw ratio provides a pessimistic-lower-bound style of conservative update; second, ε controls the width of the trust region per iteration—too small an ε zeros out the gradient for many samples and causes sample efficiency to collapse, while too large an ε degrades into unconstrained policy gradient.

    3.3 Advantage estimation: GAE and discounted returns

    If the reward model gives a per-token reward r_t, the simplest advantage is an exponentially decayed return followed by baseline subtraction:

    1. R_t = Σ_{k=t}^{T} γ^{k-t} · r_k , A_t = R_t - V(s_t)

    More commonly, Generalized Advantage Estimation (GAE) introduces λ to trade off between bias and variance:

    1. δ_t = r_t + γ·V(s_{t+1}) - V(s_t)
    2. Â_t = Σ_{l≥0} (γλ)^l · δ_{t+l}

    When λ → 0, this degenerates to single-step TD (high bias, low variance); when λ → 1, it approaches the Monte Carlo return (unbiased, high variance). In long-horizon tasks, because effective credit spans multiple rounds of interaction, you usually lean toward a larger λ—but at the cost of demanding very high fit quality from the critic.

    3.4 Value loss, entropy regularization, and KL penalty

    The value loss is a regression on the critic:

    1. L_value(θ) = E_t [ ( V_φ(s_t) - R_t )^2 ]

    In practice this is often paired with value clipping, which constrains V_φ to a neighborhood of the old value estimate, preventing the critic from overfitting on a few outlier-return samples.

    The entropy regularizer is written as L_entropy = E_t [ H(π_θ(·|s_t)) ]. Its coefficient c_e is usually taken to be very small; its role is to maintain exploration early in training and prevent the policy from quickly degenerating into deterministic output.

    There are two ways to place the KL penalty, and they are not equivalent:

    1. Put it in the reward: r̃_t = r_t - β·log( π_θ(a_t|s_t) / π_ref(a_t|s_t) ). Here the KL accumulates through γ discounting, so the effective constraint strength is coupled with γ.
    2. Put it in the loss: directly add β·L_KL to the total loss above. The constraint strength is decoupled from the discount factor and is therefore more controllable.

    In addition, there are three common forms of the Monte Carlo estimate of KL, and they differ substantially in engineering practice:

    1. k1 = log(π_θ/π_ref) # unbiased but high variance, and can be negative
    2. k2 = 0.5 · (log(π_θ/π_ref))^2 # always non-negative, low variance, biased
    3. k3 = π_ref/π_θ - log(π_ref/π_θ) - 1 # unbiased and always non-negative, usually the most practical

    3.5 The real cost of PPO

    PPO requires maintaining four models simultaneously: policy, reference, reward, and value. This means memory and compute overhead roughly double that of other approaches, plus you must tune six or seven hyperparameters such as γ / λ / c_v / c_e / β / ε. Its payoff: the critic provides a dense value baseline, so even when the reward appears only at the end of a trajectory, it can refine credit assignment down to intermediate states; and the value head can naturally be adapted into a process reward model (PRM) that provides an intermediate signal at every step of interaction in multi-turn tasks. This is precisely the technical reason long-horizon tasks have embraced PPO again.

    4. DPO: bypassing explicit rewards and online sampling with a closed-form solution

    DPO’s observation is that the KL-constrained optimal policy has an analytic form. Solving the optimum of

    1. max_π E_{x,y~π}[ r(x,y) ] - β·KL( π(·|x) || π_ref(·|x) )

    backward for the reward yields

    1. r(x,y) = β · log( π*(y|x) / π_ref(y|x) ) + β·log Z(x)

    Substituting into the Bradley-Terry preference model, the partition function Z(x) cancels exactly, so the preference probability depends only on the difference of log-ratios between the policy and the reference policy:

    1. L_DPO = - E_{(x, y_w, y_l)} [ log σ( β·( log(π_θ(y_w|x)/π_ref(y_w|x)) - log(π_θ(y_l|x)/π_ref(y_l|x)) ) ) ]

    Its gradient is

    1. ∇_θ L_DPO = - β · E[ σ( r̂_l - r̂_w ) · ( ∇log π_θ(y_w|x) - ∇log π_θ(y_l|x) ) ]

    where r̂ = β·log(π_θ/π_ref) is the implicit reward, and σ(r̂_l - r̂_w) is a “mis-ranking weight”: the more reversely the current model ranks a preference pair, the larger that sample’s gradient weight.

    Advantages: no reward model, no critic, no online rollout needed; training is as stable as ordinary supervised fine-tuning, with low memory overhead.

    Limitations: it is an offline/semi-offline algorithm whose training distribution is fixed by the preference data, so it cannot obtain feedback from the policy’s own new samples. Distribution drift is pronounced across iterative rounds; it is sensitive to noise in the preference data; and length bias (longer answers are often judged as better) gets amplified by the log-ratio of the implicit reward. Subsequent variants such as IPO, KTO, ORPO, and SimPO basically address DPO’s overfitting tendency, the use of non-paired data, or the reliance on a reference model; these are not expanded on here.

    5. GRPO: replacing the critic with a group-relative advantage

    GRPO’s core idea is: since the critic’s role is to provide a baseline, sampling a group of answers for the same prompt and using an in-group statistic as the baseline is enough.

    For each question, sample G outputs o_1...o_G, obtaining rewards r_1...r_G, and use in-group standardization as the advantage:

    1. Â_i = ( r_i - mean(r_1..r_G) ) / ( std(r_1..r_G) + eps )

    This advantage is assigned to all tokens of output o_i (outcome supervision); if a process reward model is used, it can be distributed per step instead. The objective has the same form as the policy part of PPO but drops the value loss, while KL is usually retained:

    1. L_GRPO = - E[ (1/G) Σ_i (1/|o_i|) Σ_t min( ρ_{i,t}·Â_i , clip(ρ_{i,t}, 1-ε, 1+ε)·Â_i ) ] + β·L_KL
    2. ρ_{i,t} = π_θ(o_{i,t}|·) / π_θ_old(o_{i,t}|·)

    Trade-off analysis:

    • It removes the value network of the same size as the policy, significantly reducing memory and engineering complexity—this is the main reason it can be reproduced at scale;
    • In-group standardization makes the advantage naturally zero-mean, acting as an adaptive baseline and providing some normalization across questions of different difficulty;
    • But baseline quality depends on the group size G. If G is too small, the baseline is noisy; when a group’s answers are all correct or all wrong, the in-group reward variance is zero, the advantages are all 0, and these samples consume forward computation without contributing any gradient, wasting a great deal of compute—this is the problem DAPO tackles head-on;
    • If the reward scale drifts across batches, standardization introduces extra noise, so the stability of the reward function carries very high weight in the GRPO family of methods.

    6. DAPO: systematic fixes for long-horizon reasoning

    DAPO can be seen as a set of engineering patches on the GRPO framework, each corresponding to an observed failure mode.

    (1) Clip-Higher: decouple the upper and lower clipping bounds. The original PPO/GRPO uses a symmetric 1±ε. But the gradient behavior for positive and negative advantages is not symmetric: with a symmetric ε, a low-probability token that is assigned a positive advantage quickly hits the upper bound and its gradient goes to zero, locking in exploration prematurely and causing entropy to collapse rapidly. The fix is to split the bounds:

    1. clip(ρ, 1 - ε_low, 1 + ε_high), ε_high > ε_low

    A common magnitude in practice is ε_low around 0.2 and ε_high around 0.28 (tune per task). Giving more room on the positive-advantage side means giving low-probability but correct tokens more chances to be lifted.

    (2) Dynamic sampling: filter out zero-gradient groups. During sampling, discard prompts whose in-group rewards are all identical (all correct/all wrong), and top up the batch with samples that contribute gradients. This directly improves the effective gradient yield per unit of compute, at the cost of maintaining a sampling loop and throughput monitoring.

    (3) Token-level policy loss: eliminate length bias. With an implementation that “first averages within each sequence, then averages across sequences,” each token in a long answer gets compressed to a weight of 1/|o_i|, the model tends to produce shorter answers, and the most critical end tokens in long-horizon tasks do not get sufficient gradient. Changing to summing over tokens across the whole batch and dividing by the total token count in the batch:

    1. L = - Σ_i Σ_t ( ... ) / Σ_i |o_i|

    makes every token equally weighted, so long sequences are no longer structurally penalized.

    (4) Reward shaping for overlong responses. For samples that exceed the length limit, use a “soft penalty” (a penalty that decreases linearly within the truncation interval) instead of a hard truncation that gives zero or a negative reward, avoiding the introduction of a noisy signal unrelated to task quality.

    (5) Remove the KL term. In verifiable-reward settings, the benefit of the reference-model constraint declines, while the compute cost and gradient interference it brings remain, so it is often removed outright, relying on the clip range to control drift. This is a choice with a clear precondition: once the reward can be gamed by the policy, removing KL will accelerate reward hacking.

    7. GSPO: raising the importance ratio to the sequence level

    The problem GSPO addresses is the variance accumulation of the token-level importance ratio. For a sequence of length L, its sequence likelihood ratio equals the product of L token-level ratios. Even if every token’s ratio is within [1-ε, 1+ε], the effective sequence-level ratio after multiplication can deviate severely from 1. In long-sequence scenarios and mixture-of-experts (MoE) settings where routing switches between before and after an update, this phenomenon can directly cause training to collapse or amplify anomalous samples.

    GSPO’s approach is to define a length-normalized sequence-level importance ratio (essentially the geometric mean of the sequence likelihood ratio):

    1. s_i(θ) = ( π_θ(o_i|q) / π_θ_old(o_i|q) )^{1/|o_i|}
    2. L_GSPO = - E[ (1/G) Σ_i min( s_i(θ)·Â_i , clip(s_i(θ), 1-ε, 1+ε)·Â_i ) ]

    Its gradient form is

    1. ∇_θ L_GSPO = - E[ 𝟙{not clipped} · Â_i · (1/|o_i|) · Σ_t ∇_θ log π_θ(o_{i,t}|q, o_{i,<t}) ]

    Note that the weight each token receives is always Â_i / |o_i|, independent of the token’s own new-to-old probability ratio. This yields two properties:

    • The reward and the optimization unit are aligned. Since the reward is given for the whole answer, clipping and weighting should also be done per whole answer, rather than scattering the same sequence reward at the token level into mutually unrelated constraints;
    • Variance is significantly reduced and training is more stable. The 1/|o_i| exponential normalization compresses the product into an equivalent average log-ratio, avoiding scale explosion from length. Correspondingly, the pruning threshold needs recalibration, because it acts on the normalized ratio rather than a single-step ratio.

    One common engineering companion technique: in MoE, cache the expert routing from before the update and reuse it when recomputing the old-policy probabilities (routing replay), avoiding routing jitter being mistaken for a policy change. There is also a compromise form like GSPO-token, which keeps sequence-level weighting while clipping at the token level.

    Illustrative pseudocode (framework-agnostic):

    1. # Generic group-based RL training loop skeleton
    2. for batch in dataloader:
    3. with torch.no_grad(): # rollout phase
    4. outs = policy.generate(batch.prompts, n=G)
    5. logp_old = gather_logprob(policy, outs) # old-policy probabilities for training
    6. rewards = reward_fn(outs, batch) # rule check or reward model
    7. adv = group_normalize(rewards, group_size=G) # shared by GRPO/DAPO/GSPO
    8. keep = mask_zero_variance_groups(rewards) # DAPO dynamic sampling
    9. adv, logp_old, outs = adv[keep], logp_old[keep], outs[keep]
    10. # sequence-level ratio + decoupled clipping (GSPO style)
    11. seq_ratio = torch.exp((logp - logp_old).sum(-1) / outs.len) # length-normalized
    12. surr1 = seq_ratio * adv
    13. surr2 = seq_ratio.clamp(1 - eps_low, 1 + eps_high) * adv
    14. loss = -(torch.min(surr1, surr2) * mask).sum() / mask.sum() # equal token weight
    15. if use_kl:
    16. loss += beta * kl_estimator(logp, logp_ref) # prefer k3

    8. Why long-horizon tasks swing back toward PPO

    Stringing the mechanisms above together, the choice logic on long-horizon tasks becomes clear:

    • Rewards are sparse and delayed. In multi-turn tasks, whether it truly succeeded or failed often cannot be judged until a dozen or more steps later. Group-relative advantage (the GRPO family), with only a single terminal scalar, can offer signal granularity of just “the whole trajectory is good or bad,” and is nearly powerless for credit assignment to intermediate steps.
    • The critic can carry a process signal. PPO’s value network is naturally an estimator of “state value,” usable directly for step-level advantage computation, and also a carrier for a process reward model that provides dense feedback at every round of interaction.
    • Advantage variance is controllable. With a learned baseline, you no longer rely on in-group standardization, sidestepping both the “all-correct/all-wrong group zero-gradient” problem and reward-scale drift.
    • The cost must be acknowledged: larger memory and compute overhead; the value function needs a warm-up (a common approach is to first run several steps of supervised or behavior-cloning-style regression before joint optimization); the coupling of γ/λ with the critic’s learning rate makes tuning harder; and once the critic is poorly fit, the bias directly pollutes the policy gradient—harder to diagnose than GRPO’s variance problem.

    9. Selection decision table

    Scenario characteristics Recommended route Key rationale
    Paired preference data, no online sampling budget DPO and its variants No reward model or critic needed; training is equivalent to supervised fine-tuning
    Verifiable reward (math, code, format checks), single-turn or short-horizon GRPO / DAPO Drops the critic, high compute efficiency; DAPO fixes length bias and zero-gradient groups
    Very long sequences, MoE architecture, training instability or routing jitter GSPO The sequence-level importance ratio eliminates product-form variance
    Long-horizon agent tasks, sparse and delayed rewards, need step-level feedback PPO + critic (can include process reward) The value baseline provides dense credit assignment

    10. A monitoring checklist for deployment

    The correctness of an algorithm implementation often surfaces first in the metrics. Monitor at least the following:

    1. Reward mean and variance, and the distribution across different prompt subsets (to catch reward hacking);
    2. Clipping trigger rate: too high means the step size is too large or the new and old policies have diverged too far, while near zero means clipping is effectively doing nothing;
    3. Share of samples with zero in-group reward variance: to judge whether dynamic sampling is working;
    4. KL divergence from the reference model: whether or not KL goes into the loss, it should be recorded continuously as an observable;
    5. Response length distribution: to detect length bias and collapse-style shortening;
    6. Policy entropy: a too-rapid drop is an early signal of exploration drying up;
    7. Value function fit quality (when using PPO): regression error and explained variance—when the critic fails, the policy gradient fails along with it;
    8. Gradient norm and the likelihood-ratio distribution before and after an update: to judge whether individual samples are dominating the update.

    When training these algorithms at scale, rollout generation, weight synchronization, reward scoring, and log collection form a long pipeline. In practice this usually relies on general cloud capabilities such as containerized training clusters, distributed storage, logging services, and monitoring and alerting to keep throughput observable and failures diagnosable; this kind of infrastructure capability is orthogonal to the specific algorithm choice, but it directly determines the iteration speed of long experiments.

    11. Summary

    These five methods are not substitutes for one another, but a set of engineering trade-offs tailored to different reward structures and task lengths. PPO trades complexity for dense credit assignment via a critic; DPO trades the offline-distribution limitation for training simplicity via a closed-form solution; GRPO trades baseline noise and idle compute for removing the critic via a group baseline; DAPO makes targeted patches to GRPO’s specific failure modes; and GSPO mathematically corrects the variance accumulation of the token-level importance ratio over long sequences. The judgment criteria are always just three: where the reward is produced, how finely credit needs to be assigned, and how much compute and tuning cost you can bear.