Technical Evolution Context: Dual Challenges in Long-Horizon RL

Long-horizon reinforcement learning faces two fundamental challenges during GLM model iterations: sequence length scalability and training stability. Traditional Group Rollout Policy Optimization (GRPO) employs parallel trajectory sampling, performing well in short sequences but revealing critical limitations at scale:

1. Off-policy deviation accumulation: Multi-trajectory sampling creates distribution mismatch, with exponential bias growth in long sequences
2. Training variance explosion: Group sampling amplifies advantage estimation randomness, necessitating conservative learning rates
3. Resource utilization imbalance: GPU compute units remain idle during parallel training, achieving <40% of theoretical throughput

Benchmark tests from a major cloud provider show GRPO’s accuracy drops sharply (“cliff effect”) beyond 800 steps, with performance collapsing in 1024-step sequences. This severely limits applications requiring long-term memory, such as code generation and mathematical reasoning.

SAO Architecture: Three Innovations Breaking Training Barriers

1. Single-Trajectory Sampling Mechanism

SAO introduces real-time single-trajectory sampling with three key technologies:

• Dynamic trajectory truncation: Automatic termination when value function advantage signals decay below threshold
• Asynchronous data buffering: Three-stage queue system (input/sampling/training) decouples sampling from training
• Gradient pre-computation: Initial gradient calculation during trajectory generation reduces training latency to <50ms

# Pseudocode: SAO Sampling Process
def sao_sampling(policy_model, value_model, env, max_steps=1024):
    trajectory = []
    state = env.reset()
    for step in range(max_steps):
        action = policy_model.sample(state)
        next_state, reward = env.step(action)
        trajectory.append((state, action, reward))
        # Dynamic truncation check
        if value_model.estimate(next_state) < threshold:
            break
        state = next_state
    # Asynchronous training queue submission
    training_queue.put((trajectory, compute_initial_gradient(trajectory)))

2. Dual-Sided Importance Sampling (DIS)

SAO’s Direct Dual-Sided Importance Sampling corrects asynchronous training bias through:

• Forward correction: Policy updates incorporate importance weights ( wt = \frac{\pi{\theta}(at|s_t)}{\pi{\theta_{old}}(a_t|s_t)} )
• Backward correction: Value function updates use inverse probability weights ( \tilde{w}_t = 1/w_t )
• Dynamic weight clipping: Truncation at threshold ( w_{max} ) maintains gradient stability

Experiments show DIS reduces 1000-step training bias from 23.7% (GRPO) to 4.2% while maintaining >98% gradient effectiveness.

3. Hierarchical Value Model Architecture

SAO implements a three-tier value model with differentiated update strategies:

1. Base value network: Synchronized updates per policy cycle
2. Dynamic advantage network: Independent step-wise updates with TD(λ) error correction
3. Meta-value supervisor: Global calibration every 10 training cycles

The dual-frequency update mechanism (1:2 policy-to-value update ratio) and parameter freezing strategy reduce training variance by 62%. In SWE-Bench Verified tests, SAO achieves 41% lower value function prediction error and 58% reduced advantage estimation standard deviation compared to GRPO.

Engineering Implementation: Stability in Kilostep Training

1. Mixture-of-Experts Optimization

SAO addresses value model volatility through MoE projection layer optimization:

• Dynamic expert routing: Automatic adjustment of activated experts based on trajectory length
• Gradient分流 mechanism: Separation of value gradients into expert and gating components
• Warm-start training: Initialization with pre-trained 10B-parameter expert library

This reduces value model parameter fluctuation from ±12% to ±3.5% in 1000-step training.

2. Asynchronous Scheduling System

SAO’s intelligent routing framework optimizes resource allocation through:

• Dynamic load balancing: Real-time adjustment of task distribution ratios
• Priority queue mechanism: Dedicated compute units for long-sequence tasks
• Automatic fault recovery: Checkpoint-based training continuation

In 750B-parameter clusters, the system improves GPU utilization from 58% to 89% with 2.4TFLOPS per-node throughput.

Performance Validation: Breakthrough Results

SAO demonstrates superior performance in AIME2025 mathematical reasoning benchmarks:
| Metric | SAO Accuracy | GRPO Accuracy | Improvement |
|—————————|———————|———————-|——————-|
| 1024-step reasoning | 97.3% | 84.2% | +15.6% |
| Complex theorem proving | 89.7% | 71.4% | +25.6% |
| Cross-domain transfer | 82.1% | 65.3% | +25.7% |

In SWE-Bench Verified code generation tests, SAO achieves 29.8% accuracy (+37% vs GRPO), with 81.2% code structure integrity in long functions (>500 lines) compared to GRPO’s 54.7%.

Technical Outlook: A New Era for Long-Horizon RL

SAO establishes a new paradigm for long-sequence training with three core benefits:

1. Training efficiency revolution: Reduces kilostep training time from 12 hours to 3.5 hours
2. Resource utilization breakthrough: Cuts GPU idle time from 42% to 9%, improving energy efficiency by 3.8×
3. Application expansion: Enables stable 2048-step training, unlocking complex decision-making scenarios

The technology now powers post-training systems for intelligent programming tools at a major cloud provider, delivering significant improvements in code completion and error repair. With ongoing value model optimizations and scheduling system upgrades, long-horizon reinforcement learning is poised to transform autonomous driving, financial risk management, and other ultra-long-sequence applications.