LLM RL Alignment: From PPO to GSPO Algorithm Analysis

    Reinforcement learning (RL) alignment in large language model post-training balances generating preference-aligned outputs with constraining deviations from a reference model. This article explores the evolution of alignment algorithms—from PPO and DPO to GRPO, DAPO, and GSPO. By establishing a unified modeling framework, we analyze their core mechanisms in policy optimization, advantage estimation, sampling granularity, and clipping techniques, concluding with practical selection guidelines for different use cases.

    Reinforcement learning (RL) alignment in large language model post-training essentially balances making the model generate outputs that align with target preferences and preventing the policy from deviating too far from the reference model. PPO, DPO, GRPO, DAPO, and GSPO are not entirely isolated algorithms; rather, they represent evolving technical pathways centered around policy optimization objectives, advantage estimation methods, sampling granularity, and clipping mechanisms. This article starts from a unified modeling perspective, dissects the complete loss of PPO, and then discusses DPO’s offline preference optimization, GRPO’s group-relative advantage, as well as the improvements of DAPO and GSPO in long-horizon reasoning and sequence-level optimization.

    1. Unified Problem Modeling: Policy, Trajectory, and Advantage

    Let the language model policy be:

    [
    \pi\theta(y{t+1}\mid x,y_{\le t})
    ]

    Where (x) is the prompt, (y{\le t}) is the first (t) tokens generated, and (y{t+1}) is the next action. From an RL perspective, the state can be written as (st=(x,y{\le t})), and the action is the next token. A complete output (y=(y_1,\dots,y_T)) forms a trajectory. Rewards can come from a reward model, rule verifier, verifiable answer matching, or human preference data.

    The core of policy optimization is estimating the advantage function (At), which represents “how much better the current action is relative to the baseline.” If a value network (V\phi(s_t)) is used as the baseline:

    [
    At=R_t-V\phi(st),\quad
    R_t=\sum
    {k=t}^{T}\gamma^{k-t}r_k
    ]

    Where (\gamma) is the discount factor and (r_k) is the per-token or per-step reward. In policy loss, the advantage function is typically used only as a weight; gradients are not backpropagated through the advantage term to update the policy. The value network is trained separately using regression loss. This “stop-gradient” detail is crucial; otherwise, the policy loss will backpropagate the value network’s errors into the policy parameters, undermining the optimization objective.

    2. PPO: Complete Loss and Clipping Mechanism

    PPO is a common online policy optimization method in early RLHF. Its total loss usually consists of policy loss, value loss, entropy bonus, and KL penalty:

    [
    \mathcal L{\text{PPO}}(\theta)
    =
    \mathcal L
    {\text{policy}}(\theta)
    +cv\mathcal L{\text{value}}(\theta)
    -ce\mathcal L{\text{entropy}}(\theta)
    +\beta\mathcal L_{\text{KL}}(\theta)
    ]

    Where the policy loss is:

    [
    \mathcal L_{\text{policy}}(\theta)
    =
    -\mathbb E_t\left[
    \min\left(
    r_t(\theta)A_t,
    \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)A_t
    \right)
    \right]
    ]

    The importance ratio is:

    [
    rt(\theta)=
    \frac{\pi
    \theta(y{t+1}\mid s_t)}
    {\pi
    {\theta{\text{old}}}(y{t+1}\mid s_t)}
    ]

    The clipping limits the magnitude of policy change in a single update. When the advantage is positive, the policy increases the action’s probability, but the gradient is truncated once the ratio exceeds (1+\epsilon); when the advantage is negative, the policy decreases the action’s probability, but stops optimizing further once the ratio falls below (1-\epsilon). The gradient of the policy loss can be summarized as:

    [
    \nabla\theta \mathcal L{\text{policy}}
    =
    -\mathbb Et\left[
    A_t\nabla
    \theta
    \min\left(r_t(\theta),
    \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)
    \right)
    \right]
    ]

    In practice, this is handled by automatic differentiation, but understanding its piecewise nature is important: within the clipping range, it approximates a standard policy gradient, while outside the range, the gradient is suppressed, reducing training oscillations.

    The value loss is typically mean squared error:

    [
    \mathcal L{\text{value}}
    =
    \mathbb E_t\left[(V
    \phi(s_t)-R_t)^2\right]
    ]

    Its gradient is (2(V\phi(s_t)-R_t)\nabla\phi V_\phi(s_t)). The entropy bonus encourages exploration, and the KL penalty constrains the new policy from deviating too far from the reference policy:

    [
    \mathcal L{\text{KL}}
    =
    \mathbb E_t\left[
    \log\pi
    \theta(y{t+1}\mid s_t)
    -\log\pi
    {\text{ref}}(y_{t+1}\mid s_t)
    \right]
    ]

    PPO’s advantages are its versatility, stability, and online exploration capabilities; its disadvantages include maintaining policy, value, reward, and reference models simultaneously, resulting in high memory usage and engineering complexity. For long-horizon tasks, per-token advantage estimation and discounted reward design can also affect credit assignment.

    3. DPO: Offline Preference Optimization Without Explicit Reward Models

    DPO’s core idea is to transform “reward modeling + reinforcement learning” into direct classification-based optimization over preference pairs. Given a preferred output (yw) and a dispreferred output (y_l) for the same prompt, and a reference policy (\pi{\text{ref}}), the DPO loss is often written as:

    [
    \mathcal L{\text{DPO}}
    =
    -\mathbb E
    {(x,yw,y_l)}
    \log\sigma\left(
    \beta\left[
    \log\frac{\pi
    \theta(yw\mid x)}{\pi{\text{ref}}(yw\mid x)}
    -
    \log\frac{\pi
    \theta(yl\mid x)}{\pi{\text{ref}}(y_l\mid x)}
    \right]
    \right)
    ]

    Where (\sigma) is the sigmoid function and (\beta) controls the preference margin strength. Intuitively, DPO increases the probability of the preferred output relative to the reference model while suppressing the dispreferred output. It requires no explicit reward model or online sampling, making training simpler and more stable.

    However, DPO is an offline method, and its effectiveness is heavily influenced by the coverage of preference data. If there is a distribution shift between training data and target distribution, the model may only learn surface-level preferences from the data, lacking the corrective new samples generated by online exploration. Thus, DPO is suitable as a low-cost alignment baseline or to be combined with online policy optimization.

    4. GRPO: Replacing Value Networks with In-Group Relative Rewards

    GRPO’s core improvement is replacing the independent value network with “in-group relative advantage.” For each prompt, a group of outputs (y_1,\dots,y_G) is sampled, obtaining rewards (R_i) respectively, and then normalized within the group:

    [
    A_i=\frac{R_i-\operatorname{mean}(R_1,\dots,R_G)}
    {\operatorname{std}(R_1,\dots,R_G)+\delta}
    ]

    The policy loss can be written as:

    [
    \mathcal L{\text{GRPO}}
    =
    -\mathbb E\left[
    \frac{1}{G}\sum
    {i=1}^{G}
    \frac{1}{|yi|}\sum{t=1}^{|yi|}
    \min\left(
    r
    {i,t}Ai,
    \operatorname{clip}(r
    {i,t},1-\epsilon,1+\epsilon)Ai
    \right)
    \right]
    +\beta\mathcal L
    {\text{KL}}
    ]

    Where (r_{i,t}) is the importance ratio for the (t)-th token of the (i)-th output. GRPO eliminates the value network, reducing memory and training complexity; in-group normalization also provides a natural baseline. However, group sampling requires sufficient diversity. If rewards within a group are nearly identical, the advantage approaches zero, weakening the learning signal. Reward scale, group size, and normalization methods must be configured carefully.

    5. DAPO: Policy Optimization Enhancements for Long-Chain Reasoning

    DAPO can be understood as a policy optimization framework that enhances PPO/GRPO objectives, focusing on effective sample utilization, clipping ranges, and length bias in long-chain reasoning tasks. Common modifications include decoupled upper and lower clipping bounds, dynamic filtering of valid samples, token-level loss normalization, and length control.

    A common decoupled clipping form can be abstracted as:

    [
    \min\left(
    riA_i,
    \operatorname{clip}(r_i,1-\epsilon
    {\text{low}},1+\epsilon_{\text{high}})A_i
    \right)
    ]

    Compared to symmetric clipping, decoupling the bounds allows separate control over the magnitude of “increasing probability” and “decreasing probability,” reducing premature convergence in long sequence training. Dynamic sampling tends to filter out groups that are entirely correct or entirely incorrect, retaining samples with discriminative power and increasing the information density per batch. Token-level normalization can mitigate the issue of long outputs being overly averaged in the loss.

    DAPO’s advantage lies in its suitability for long-horizon reasoning scenarios, but it introduces more hyperparameters, and dynamic sampling strategies may introduce distribution bias. In practice, it is necessary to monitor the proportion of valid samples, average length, KL distance, and validation set performance to avoid sacrificing output diversity for training signals.

    6. GSPO: Sequence-Level Ratio and Group Sequence Optimization

    GSPO’s typical approach is to elevate the importance ratio from the token level to the sequence level, using the likelihood ratio of the entire output for clipping, combined with in-group relative advantage. The sequence-level ratio can be written as:

    [
    ri(\theta)=
    \exp\left(
    \log\pi
    \theta(yi\mid x)
    -\log\pi
    {\text{old}}(y_i\mid x)
    \right)
    ]

    The corresponding sequence-level policy loss can be abstracted as:

    [
    \mathcal L{\text{GSPO}}
    =
    -\mathbb E\left[
    \frac{1}{G}\sum
    {i=1}^{G}
    \min\left(
    riA_i,
    \operatorname{clip}(r_i,1-\epsilon,1+\epsilon)A_i
    \right)
    \right]
    +\beta\mathcal L
    {\text{KL}}
    ]

    Compared to per-token ratios, the sequence-level ratio focuses more on the overall probability change of the entire response, reducing the instability caused by the accumulation of variance in token-level ratios in long sequences. However, sequence ratios can easily become numerically too large or too small in long texts, often requiring computation in log space and length normalization. GSPO is suitable for scenarios where controlling the magnitude of policy updates at the “entire output” level is desired, but attention must be paid to length bias and in-group reward scales.

    7. Algorithm Comparison and Selection Guidelines

    Algorithm Explicit Reward Model Value Network Sampling Method Core Advantage Primary Risk
    PPO Usually required Required Online Versatile, stable, explorable Complex engineering, high memory
    DPO Not required Not required Offline preference Simple, stable, low cost Distribution shift, lack of exploration
    GRPO Can be shared Not required Group sampling Reduced memory, in-group baseline In-group reward homogenization
    DAPO Can be shared Not required Dynamic group sampling Long-chain reasoning enhancement Complex hyperparameters, sampling bias
    GSPO Can be shared Not required Sequence-level group sampling Reduced token ratio variance Sequence ratio numerical stability and length bias

    Practical selection can follow these guidelines:

    1. If high-quality preference pairs are available and compute is limited, start with DPO to establish an offline alignment baseline to verify data quality and reference policy settings.
    2. If online exploration and verifiable rewards are needed, prioritize PPO or GRPO-like methods; GRPO is more lightweight when memory is tight.
    3. In long-chain reasoning tasks, pay attention to the differences between token-level and sequence-level normalization. DAPO and GSPO can serve as enhancement solutions but should undergo small-scale ablation studies first.
    4. Regardless of the algorithm chosen, monitor KL distance, output length, reward distribution, valid sample ratio, and validation set metrics to prevent reward hacking and loss of diversity.
    5. Evaluation should not only look at average rewards but also cover factuality, safety, format adherence, length bias, and robustness.

    Overall, PPO provides a complete online policy optimization framework, DPO simplifies the process with offline preference optimization, GRPO reduces value network costs with group-relative advantages, while DAPO and GSPO enhance stability from the perspectives of long-chain sample utilization and sequence-level ratio control. Understanding the unified objectives and differences behind these algorithms is more important than merely memorizing formulas.