1. Technical Evolution and Core Challenges

During the GLM-5.2 phase, long-horizon RL strategies demonstrated superiority but faced two fundamental limitations:

1. Off-Policy Deviation: Traditional Group Rollout Policy Optimization (GRPO) suffered exponential deviation growth with trajectory length
2. Value Model Instability: Infrequent updates caused training variance fluctuations, limiting generalization in complex reasoning tasks

Benchmark data reveals critical performance drops:

• AIME2025 mathematical reasoning: Accuracy fell from 85% to 62% beyond 20-step reasoning chains
• SWE-Bench Verified code generation: Memory usage surged 300% with 500+ step trajectories, while training stability decreased by 40%

SAO addresses these challenges through three design objectives:

1. Maintain off-policy deviation below 5%
2. Enable coordinated optimization between value and policy models
3. Double training throughput under equivalent computational resources

2. SAO Architecture Breakdown

2.1 Single-Trajectory Sampling Innovation

SAO replaces GRPO’s batch sampling with single-rollout generation per task, delivering three key advantages:

• Memory Efficiency: 65% reduction at 750B parameters, enabling context window expansion from 2,048 to 4,096 tokens
• Deviation Control: Direct Importance Sampling (DIS) reduces bias from 18% (GRPO) to 7.2%
• Real-Time Feedback: Second-level policy updates replace minute-level delays through immediate trajectory training

# SAO Sampling Workflow
def sao_sampling(policy_model, value_model, task):
    context = initialize_context(task)
    trajectory = []
    while not terminal_condition(context):
        action = policy_model.sample(context)
        next_context, reward = environment_step(context, action)
        trajectory.append((context, action, reward))
        context = next_context
    train_async(policy_model, value_model, trajectory)

2.2 Value Model Enhancement System

SAO implements a three-tier optimization mechanism:

1. High-Frequency Updates: Value model updates twice per policy update, reducing estimation variance by 42%
2. Parameter Freezing: Attention modules remain fixed while optimizing MoE projection layers, decreasing training fluctuation by 31%
3. Advantage Signal Optimization: Token-level GAE calculations are skipped, focusing advantage propagation between action nodes for 55% faster computation

At 750B parameters, this system achieves:

• 2.8× faster value model convergence
• Policy gradient estimation error reduction from 0.32 to 0.11
• 92% value model accuracy in mathematical reasoning (vs. 78% for GRPO)

2.3 System-Level Coordination

Integration with slime framework delivers three system breakthroughs:

• Dynamic Load Balancing: Router algorithm improvements boost node utilization from 68% to 92%
• Asynchronous Communication: Dual-buffering reduces end-to-end latency by 57%
• Mixed-Precision Training: FP16/FP32 hybrid precision maintains numerical stability while increasing throughput by 1.8×

System benchmarks show:

• Throughput increase from 120 to 280 samples/sec in 1,000-step training
• Memory bandwidth utilization optimized to 89%
• 29.8% code generation accuracy in SWE-Bench Verified (vs. 22% for GRPO)

3. Performance Validation and Industry Applications

3.1 Benchmark Results

AIME2025 mathematical reasoning improvements:
| Trajectory Length | GRPO Accuracy | SAO Accuracy | Improvement |
|—————————|———————|——————-|——————|
| 50 steps | 82.1% | 94.7% | +15.4% |
| 200 steps | 68.3% | 91.2% | +33.5% |
| 500 steps | 52.7% | 85.6% | +62.4% |

SWE-Bench Verified code generation shows:

• 41% accuracy improvement in complex functions (recursive algorithms, multithreading)
• Error repair rate increase from 18% to 34%

3.2 Practical Applications

1. Intelligent Coding Assistants: Handle 1,000+ line context windows with 27% higher completion accuracy
2. Legal Document Analysis: Process 20-page documents with 91% key clause identification accuracy
3. UI Test Automation: Generate cross-page test scripts with 35% higher coverage

4. Evolution and Future Directions

SAO development follows three key trajectories:

1. Multimodal Expansion: Integrating visual-language joint modeling (validated in document understanding tasks)
2. Dynamic Trajectory Adjustment: Attention-based adaptive truncation algorithms (projected 40% efficiency gain)
3. Distributed Optimization: Cross-node value model sharing for linear scaling on 10,000+ GPU clusters

The SAO-Pro roadmap introduces neurosymbolic integration targeting 99%+ accuracy in mathematical reasoning, while lightweight versions aim for real-time inference of 750B models on consumer-grade GPUs.

Conclusion: SAO establishes a reproducible technical paradigm for long-horizon RL training through its innovative sampling mechanism and value model optimization system. The verified performance improvements and system-level optimizations position it as a benchmark solution for AI agent training, with continued breakthroughs expected in complex decision-making scenarios through multimodal expansion and dynamic trajectory adjustment technologies.