Evolution of Long-Horizon Reinforcement Learning: From GRPO to PPO

In reinforcement learning, algorithm selection directly impacts model training efficiency and final performance. A major language model recently abandoned the long-established GRPO (Group Relative Policy Optimization) algorithm for PPO (Proximal Policy Optimization) with critic networks in long-horizon task training, sparking widespread discussion among developers. This article provides an in-depth analysis of this technical shift from three perspectives: algorithmic principles, task adaptability, and engineering implementation.

1. GRPO’s Rise: Lightweight Optimization for Short Tasks

Traditional PPO requires maintaining a critic network matching the policy network’s scale, predicting future cumulative rewards (Value Function) to guide policy optimization. This architecture presents two key challenges:

• High memory consumption: Dual-network structure doubles GPU memory usage, limiting model scale and batch size
• Training instability: Critic network prediction errors directly propagate to policy updates, causing oscillations

GRPO addressed these issues through innovative design:

• Decentralized evaluation: Eliminated critic networks by using group-averaged rewards as benchmarks
• Relative advantage calculation: Determined policy update direction based on reward deviations from group means
• Parallel-friendly architecture: Enabled complete parallelization of sampling and evaluation processes

For short-duration tasks like code generation and mathematical reasoning, GRPO demonstrated significant advantages. One open-source project reduced training memory usage by 47% while maintaining 92% task success rates using GRPO, making it the standard solution for short-sequence training.

2. Long-Horizon Challenges: GRPO’s Limitations

When tasks transition from single-step decisions to complex workflows spanning hours, GRPO reveals critical weaknesses:

Trajectory Length Heterogeneity

Long-horizon tasks generate execution trajectories with vast length variations (e.g., 3-step simple queries vs. 50-step complex reasoning). When trajectory length standard deviation exceeds 30% of the mean, GRPO’s reward estimation error grows exponentially.

Temporal Credit Assignment

In long-horizon tasks, single-step decision impacts often manifest only after dozens of steps. GRPO’s group comparison mechanism fails to capture these delayed feedback signals, with research showing 62% lower sensitivity to early decisions compared to PPO in tool-calling tasks.

Sample Alignment Difficulties

Compressed sub-trajectories vary significantly in quantity (e.g., 15 sub-trajectories from one prompt vs. 3 from another), preventing fair comparison group construction. This sample misalignment problem becomes particularly pronounced in multi-stage tasks, causing systematic bias in policy updates.

3. PPO’s Return: Robust Solution for Long-Horizon Training

PPO with critic networks demonstrates unique strengths in addressing these challenges:

Temporal Difference Learning

Through critic networks, PPO implements temporal difference (TD) learning for accurate intermediate state valuation. In document summarization tasks, PPO achieves 41% higher credit assignment accuracy than GRPO, particularly effective for cross-paragraph information integration.

Dynamic Trajectory Handling

With Generalized Advantage Estimation (GAE), PPO adaptively adjusts trajectory length weights, maintaining 89% training efficiency even when trajectory length standard deviation reaches 50% of the mean.

Stability Guarantees

PPO’s truncation update strategy and importance sampling effectively constrain policy update magnitudes, preventing policy collapse from sparse rewards in long-horizon tasks. During 72-hour continuous agent training, PPO showed 73% lower policy variance than GRPO.

4. Engineering Implementation Considerations

Technical selection requires balancing theoretical advantages with engineering constraints in long-horizon training scenarios:

Memory Optimization Strategies

• Gradient checkpointing: Offloads critic network intermediate activations to CPU memory
• Mixed-precision training: Uses FP16 computation to reduce critic network weight storage
• Parameter sharing: Shares layers between policy and critic networks

Training Acceleration Techniques

• Asynchronous sampling: Decouples sampling from policy updates for improved hardware utilization
• Experience replay: Implements prioritized replay buffers to address sample sparsity
• Distributed training: Adopts parameter server architecture for synchronized critic network updates

Debugging and Monitoring Framework

• Reward curve analysis: Tracks convergence trends of immediate vs. long-term rewards
• Gradient norm monitoring: Detects update magnitude matching between critic and policy networks
• State coverage statistics: Ensures training trajectories cover critical state spaces

5. Technical Evolution Insights

The transition from GRPO to PPO reveals core principles for reinforcement learning algorithm selection:

1. Task-specific adaptation: Algorithm characteristics must closely match task requirements
2. Engineering constraint awareness: Memory and computational resources significantly influence technical decisions
3. Long-term benefit evaluation: Requires balancing short-term training efficiency with final model performance

In long-horizon agent training, PPO and its variants (IMPALA, A2C) are becoming the new standard. Cloud providers’ distributed training platforms have optimized communication protocols for these algorithms, achieving 3× efficiency improvements for models with hundreds of billions of parameters. As multimodal large models evolve, designing more efficient long-horizon training algorithms will represent the next major breakthrough.