Technical Turning Point: GRPO’s Retirement and PPO’s Resurgence

The GLM-5.2 technical report reveals a pivotal strategy shift: abandoning the long-standing GRPO method in favor of PPO for long-sequence reinforcement learning. This transition reflects broader trends in AI training efficiency and model scalability.

GRPO’s Limitations in Large-Scale Training

While GRPO’s group-based relative advantage evaluation demonstrated value in image captioning and combinatorial optimization since 2017, three critical weaknesses emerged during GLM-5.2 development:

1. Long-sequence degradation: Gradient propagation between policy groups衰减 (attenuated) by over 60% beyond 2048 tokens
2. Baseline complexity: Dynamic baseline network maintenance increased training resource consumption by 40%
3. Sampling inefficiency: Grouping mechanism reduced effective sample utilization by 30%

PPO’s Performance Advantages

PPO’s clipped probability ratio and generalized advantage estimation (GAE) delivered measurable improvements:

• 42% faster convergence in mathematical reasoning tasks
• Elimination of baseline network maintenance requirements
• 55% lower variance in advantage estimation

Evolutionary Path: From Self-Critical to GRPO to PPO

Understanding this transition requires examining reinforcement learning’s progression in NLP:

Self-Critical Baseline (2017)

Rennie’s method introduced sampling-based baseline construction using greedy policy trajectories:

# Pseudocode: Self-Critical advantage calculation
def compute_self_critical_advantage(trajectory, greedy_trajectory):
    return trajectory['reward'] - greedy_trajectory['reward']

This reduced variance but suffered from single-trajectory representation bias.

GRPO’s Group-Based Improvement (2019)

Kool’s team enhanced stability through multi-trajectory averaging:

# GRPO baseline calculation (simplified)
def compute_grpo_baseline(trajectories):
    baseline = sum(t['reward'] for t in trajectories)/len(trajectories)
    return [r - baseline for r in [t['reward'] for t in trajectories]]

While improving baseline stability by 60%, the grouping mechanism created exponential computational overhead at scale.

PPO’s Technical Implementation

PPO achieves stability through two key innovations:

Mathematical Foundation

The advantage estimation formula balances temporal difference learning:

A_t = δ_t + (γλ)δ_{t+1} + ... + (γλ)^{T-t+1}δ_{T-1}
where δ_t = r_t + γV(s_{t+1}) - V(s_t)

GLM’s Engineering Optimizations

The implementation incorporated three critical enhancements:

1. Dynamic clipping: Automated ε adjustment from 0.2 (early training) to 0.05 (late training)
2. Hybrid advantage estimation: Combined n-step returns with GAE for robust temporal credit assignment
3. Distributed synchronization: Parameter server architecture enabled efficient training across 1,000+ GPUs

Strategic Considerations Behind Algorithm Selection

GLM-5.2’s choice reflects three emerging trends in large model training:

Performance Metrics Comparison

Under identical compute resources:
| Metric | GRPO | PPO | Improvement |
|————————-|——————|——————|——————-|
| Throughput (2048 tokens) | 12K samples/sec | 18K samples/sec | +50% |
| Advantage variance | 0.32 | 0.14 | -55% |

Model Performance Impact

PPO-trained models demonstrated:

• 3.7% higher accuracy on MATH dataset
• Particularly strong performance in geometric proof tasks requiring long-context reasoning

Future Evolution Directions

The GLM team’s experience suggests three research trajectories:

1. Baseline estimation: Transition from explicit networks to implicit advantage calculation
2. Update mechanisms: Shift from relative to absolute advantage optimization
3. Distributed training: Develop more efficient communication protocols and gradient compression

Current industry research focuses on:

• Transformer-specific advantage functions
• Adaptive clipping coefficient mechanisms
• Attention-based advantage estimation

This technical evolution marks an important milestone in reinforcement learning’s application to large models, with GLM-5.2’s implementation providing valuable insights for the broader AI community. The complete methodology and experimental data are available in the official technical report.