Single-Trajectory Asynchronous Optimization: A Breakthrough in Long-Horizon Reinforcement Learning Training

1. Technical Evolution Context: Core Challenges in Long-Horizon RL

During the GLM-5.2 development phase, long-horizon RL emerged as a critical path for enhancing model decision-making capabilities by extending agent execution trajectories. However, traditional group rollout sampling methods revealed three fundamental bottlenecks:

1.1 Off-Policy Deviation Accumulation
Group sampling requires simultaneous generation of multiple trajectories for joint optimization. State distribution discrepancies across trajectories cause policy updates to deviate from true objectives.

1.2 Memory and Computational Waste
Parallel sampling of multiple trajectories demands substantial buffer space for intermediate states, frequently triggering OOM errors in models with 100B+ parameters.

1.3 Limited Generalization
Group sampling tends to optimize local trajectory segments, struggling to capture global dependencies in long-term tasks.

To overcome these limitations, Tsinghua University’s KEG Lab proposed the SAO framework, achieving breakthroughs in stable training beyond 1000 steps within GLM-5.2’s agent RL pipeline through single-trajectory sampling and asynchronous policy updates.

2. SAO Core Mechanism: Single-Trajectory Sampling Innovation

2.1 Sampling Architecture Redesign
Unlike traditional group sampling (e.g., GRPO) that generates N≥4 trajectories per task, SAO implements single-trajectory instant sampling:

# Pseudocode: SAO single-trajectory sampling
def sao_rollout_sampling(env, policy, max_steps):
    trajectory = []
    state = env.reset()
    for _ in range(max_steps):
        action = policy.select_action(state)  # Single-step decision
        next_state, reward, done = env.step(action)
        trajectory.append((state, action, reward))
        state = next_state
        if done: break
    return trajectory  # Returns single complete trajectory

This design decouples trajectory generation from policy updates, eliminating synchronization delays caused by inconsistent trajectory lengths in group sampling.

2.2 Asynchronous Optimization Engine
SAO employs a producer-consumer model for parallel sampling and optimization:

• Sampling Thread Pool: Continuously generates single-trajectory data into shared buffers
• Optimization Thread: Asynchronously retrieves trajectories for policy gradient computation
• Dynamic Batching: Automatically adjusts optimization batch sizes based on GPU resources

Experimental results in GLM-5.3 training show 2.3× throughput improvement over group sampling through:

• Zero-Copy Data Transfer: Shared memory reduces CPU-GPU data movement
• Gradient Accumulation: Combines small-batch gradients into equivalent large-batch updates
• Mixed Precision Training: FP16/FP32 hybrid computation accelerates matrix operations

2.3 Context Compression Technology
To address state explosion in long trajectories, SAO introduces hierarchical state encoding:

1. Local State Encoding: Transformer encodes current observations
2. Global Context Compression: Sparse attention maintains key states in sliding windows
3. Dynamic Memory Units: Create retrievable indexes for critical historical events

This design maintains training stability beyond 1000 steps while reducing memory usage by 40% compared to traditional methods.

3. Technical Iteration Path: GLM-5.2 to GLM-5.3 Evolution

3.1 GLM-5.2 Validation
Initial SAO implementation on 750B-parameter GLM-5.2 demonstrated:

• Training Stability: 1000-step continuous training without divergence through gradient clipping and adaptive learning rates
• Sampling Efficiency: 1200 effective trajectories/hour per GPU (vs. 300 in group sampling)
• Policy Quality: 18% higher pass rate in code generation tasks compared to baselines

3.2 GLM-5.3 Full-Stack Optimization
Building on SAO’s core framework, GLM-5.3 achieved performance leaps through three upgrades:

1) Slime Framework Synergy

• Algorithmic Layer: Introduced prioritized experience replay to improve critical sample utilization
• System Layer: Optimized router-node load balancing, boosting cluster resource utilization from 65% to 88%

2) End-to-End Throughput Breakthrough
Achieved 2.3× throughput improvement through:

Original Architecture: Group sampling → Synchronous optimization → 120 samples/sec per GPU
SAO Architecture: Single-trajectory sampling → Asynchronous pipeline → 280 samples/sec per GPU

3) Long-Horizon Capability Enhancement

• Trajectory Extension: Increased from 512 steps (GLM-5.2) to 1024 steps
• Reward Shaping Optimization: Introduced intermediate reward functions for long-term goal alignment
• Exploration Strategy Improvement: Combined ε-greedy with Upper Confidence Bound (UCB) for exploration-exploitation balance

4. Industry Applications and Current Limitations

4.1 Typical Use Cases

• Complex Decision Systems: Autonomous driving trajectory planning, financial trading strategy generation
• Multi-Turn Dialogue Systems: Context-consistent interaction through long-term memory
• Code Automation: Cross-file/function program repair and optimization

4.2 Technical Limitations

• Ultra-Long Trajectory Challenge: Information loss occurs when trajectory length exceeds 2048 steps due to state compression
• Latency Requirements: Asynchronous optimization introduces 50-100ms delays unsuitable for high-frequency trading
• Sparse Reward Problem: Policy update efficiency remains suboptimal in tasks with extremely sparse reward signals

5. Future Development Directions

1. Neural-Symbolic Integration: Combine symbolic reasoning to enhance long-term planning capabilities
2. Distributed SAO Scaling: Research cross-node trajectory sampling and global policy synchronization
3. Adaptive Trajectory Truncation: Dynamically identify critical states for variable-length trajectory optimization

The evolution of SAO framework marks a critical transition of long-horizon RL from laboratory research to industrial applications. Its innovative combination of single-trajectory sampling and asynchronous optimization provides a replicable technical path for efficient training of models with 100B+ parameters. With continued breakthroughs in context compression and distributed collaboration, this technology is poised to drive paradigm shifts in autonomous driving, intelligent healthcare, and other complex decision-making domains.