Technical Evolution Background and Core Challenges

Long-horizon reinforcement learning (LHRL) faces two fundamental challenges during GLM model iterations: off-policy bias accumulation and training variance explosion. Traditional group-based sampling strategies like GRPO suffer from distribution drift when handling extended execution trajectories, causing model updates to deviate from optimal solutions. The synchronous update mechanism between value and policy models further amplifies training instability.
The SAO framework, developed by leading academic laboratories, has achieved breakthrough stability in training 750B-parameter language model agents for 1,000 consecutive steps. Its core innovation lies in replacing group sampling with single-trajectory generation, where each task produces one complete training trajectory, fundamentally reducing off-policy impact.

Three Innovation Dimensions of SAO Architecture

1. Sampling Strategy Revolution: Single-Trajectory Asynchronous Mechanism

Traditional GRPO employs multi-trajectory parallel sampling with intra-group averaging to reduce variance, but sacrifices sample relevance to current policies. SAO introduces three breakthrough designs:

• Just-in-time sampling: Immediately trains on single trajectories generated per task, maintaining strong data-policy correlation
• Dynamic trajectory truncation: Adaptively adjusts trajectory length based on task complexity to eliminate redundant computation
• Asynchronous buffer management: Uses dual buffers to decouple sampling and training, improving system throughput
  Experimental results show SAO achieving 97.3% accuracy on AIME2025 mathematical reasoning (13.1% improvement over GRPO) and 29.8% accuracy on SWE-Bench code generation (up from 24.5%).

2. Value Model Optimization: Bilateral Importance Sampling and High-Frequency Updates

SAO reconstructs value model training through three key technologies:

• Direct Bilateral Importance Sampling (DIS): Reduces off-policy error by 62% using dual-weight estimators that correct both policy distribution shifts and state transition biases
• Asymmetric update frequency: Updates value models twice for each policy model update, enhancing value estimation timeliness
• Hybrid expert layer freezing: Maintains attention module parameters while optimizing MoE projection layers, reducing training fluctuation by 41%
  For system implementation, SAO adopts token-level GAE (Generalized Advantage Estimation) that skips observation tokens, lowering value computation complexity by 35% while preserving estimation accuracy.

3. System-Level Optimization: Asynchronous Training Load Balancing

Addressing distributed training challenges for超大模型 (ultra-large models), SAO integrates three system optimizations:

• Dynamic routing scheduling: Automatically allocates compute resources based on task characteristics, improving long-tail task processing efficiency by 2.8×
• Gradient compression communication: Uses 4-bit quantization to reduce parameter synchronization bandwidth by 75%
• Fault-tolerant training mechanism: Enables second-level recovery from interruptions through checkpoint snapshots and gradient checksums
  In 750B-parameter training clusters, these optimizations increase end-to-end throughput by over 2.3× while reducing costs by 58% for long-horizon coding RL.

Key Implementation Pathways

1. Decoupled Sampler-Trainer Design

SAO implements an asynchronous training pipeline using producer-consumer pattern:

class SAOPipeline:
    def __init__(self):
        self.sampler = SingleRolloutSampler()  # Single-trajectory sampler
        self.trainer = AsynchronousTrainer()    # Asynchronous trainer
        self.buffer = CircularBuffer(size=1024) # Circular buffer
    def run(self):
        while True:
            rollout = self.sampler.generate()  # Generate single trajectory
            self.buffer.put(rollout)          # Store in buffer
            if self.buffer.ready():
                batch = self.buffer.get_batch() # Retrieve training batch
                self.trainer.update(batch)     # Asynchronously update models

This design decouples sampling and training processes, allowing the sampler to operate at maximum throughput while the trainer dynamically adjusts batch sizes based on system load.

2. Dual-Stream Value Model Updates

SAO implements asymmetric updates between value and policy models:

def update_models(policy, value, batch):
    # Policy model update (per step)
    policy_loss = compute_policy_loss(policy, batch)
    policy.update(policy_loss)
    # Value model double update
    for _ in range(2):
        value_loss = compute_value_loss(value, batch)
        # Freeze attention module
        with freeze_layers(value.attention):
            value.update(value_loss)  # Only update MoE projection layers

This mechanism enables faster value model adaptation to policy changes while maintaining training stability.

3. Distributed Training Load Balancing

For thousand-GPU training clusters, SAO employs dynamic task allocation:

Initialization: Assign base load weights based on GPU performance
Each iteration:
    1. Monitor gradient computation time per node
    2. Calculate load deviation index = (actual time - theoretical time) / theoretical time
    3. Adjust next task allocation ratio:
       New weight = current weight * (1 - α * deviation index)
    Where α is decay coefficient (experimentally set to 0.3)

This algorithm improves overall cluster utilization from 68% to 92% while reducing long-tail task waiting time by 87%.

Application Scenarios and Performance Validation

In agent training for ultra-large language models, SAO demonstrates significant advantages:

• Long-horizon reasoning tasks: Achieves 41% higher task completion rate than GRPO for trajectories exceeding 2,000 steps
• Sparse reward scenarios: Triples reward signal propagation efficiency through DIS sampling
• Resource-sensitive tasks: Delivers 2.3× training throughput compared to industry averages under identical hardware conditions
  Particularly in code generation, SAO-trained models improve first-attempt correctness from 31% to 57% for complex algorithm problems, showcasing superior long-term planning capabilities.

Technology Evolution Directions

Current SAO development focuses on three key areas:

1. Multimodal trajectory fusion: Supporting cross-modal sampling across text, images, and audio
2. Adaptive trajectory length: Dynamically adjusting sampling depth based on task complexity
3. Hardware-aware optimization: Kernel-level optimization for emerging accelerator architectures
   As long-horizon reinforcement learning advances in autonomous driving and industrial control, SAO’s stable training capabilities and efficient resource utilization position it as critical infrastructure for ultra-large intelligent systems. Its innovative single-trajectory sampling and asynchronous training architecture are redefining engineering paradigms for reinforcement learning implementation.