From 22 to 86 Pages: A Paradigm Shift in Technical Transparency

The opacity of mainstream large language models (LLMs) has long hindered academic research and industrial deployment. The second edition of DeepSeek-R1’s research paper achieves three breakthroughs through systematic disclosure:

1. Algorithmic Transparency: Full mathematical derivation and experimental comparison data for the GRPO algorithm
2. System Decoupling: First public disclosure of modular RL training infrastructure architecture
3. Reproducibility: Intermediate checkpoint data and failure case analysis to lower technical replication barriers

This transparency provides dual value: standardized benchmarks for reinforcement learning research academically, and reference engineering solutions for model optimization industrially. The disclosed dynamic memory unloading strategy, for instance, directly applies to training optimization of models with 100B+ parameters.

GRPO Algorithm: Breaking PPO’s Three Bottlenecks

PPO’s Fundamental Limitations

Traditional PPO algorithms face two core challenges in long-sequence reasoning:

• Value Model Dependency: Requires training value networks comparable in scale to policy models, doubling GPU memory consumption
• Cross-Timestep Prediction: Continuous value estimation correction during generation leads to exponential decay in correlation between early tokens and final rewards

Experimental data shows PPO requires λ=0.95 tuning for GAE on the MATH dataset to achieve baseline performance, with misconfiguration causing >40% performance degradation.

GRPO’s Innovative Design

GRPO introduces group-relative advantage estimation through three improvements:

# GRPO advantage estimation pseudocode
def grpo_advantage(rewards, group_size=16):
    mean_reward = np.mean(rewards)
    std_reward = np.std(rewards)
    normalized_rewards = (rewards - mean_reward) / (std_reward + 1e-8)
    return normalized_rewards - np.mean(normalized_rewards)  # Group-relative advantage

• Value-Free Estimation: Constructs relative advantage benchmarks through group sampling (default 16 outputs)
• Normalization Processing: Uses mean-variance normalization to eliminate absolute reward scale effects
• Long-Sequence Adaptation: Advantage estimation relies solely on intra-batch comparisons, avoiding cross-timestep prediction

On the CodeGen benchmark, GRPO reduces memory usage by 62% compared to PPO for code generation exceeding 2048 tokens, achieving 92% of PPO’s optimal performance without parameter tuning.

Engineering Optimizations

The paper reveals three critical implementation details:

• Sampling Strategy: Balances exploration-exploitation using Top-k sampling (k=4) with temperature coefficient (τ=0.7)
• Parallel Acceleration: Achieves 8-way parallel sampling via vLLM workers, reducing single-batch generation time from 3.2s to 0.8s
• Reward Calculation: Combines rule-based rewards (code execution success rate) with learned rewards (BERTScore similarity) for composite evaluation

RL Training Infrastructure: Modular Architecture Design

System Decomposition

The paper discloses a four-layer modular architecture for the first time:

┌───────────────┐    ┌───────────────┐    ┌───────────────┐    ┌───────────────┐
│   Rollout     │───▶│   Inference   │───▶│ Rule-based    │───▶│   Training    │
│   Module      │    │   Module      │    │   Reward      │    │   Module      │
└───────────────┘    └───────────────┘    └───────────────┘    └───────────────┘
       ▲                                                                  │
       └──────────────────────────────────────────────────────────────────┘

• Rollout Module: Handles batch candidate generation with multi-token prediction (MTP) acceleration
• Inference Module: Performs forward reasoning with reward/reference models using FP16 mixed precision
• Rule-based Reward Module: Calculates hard metrics like code execution results and answer matching
• Training Module: Executes policy gradient updates with gradient checkpointing and memory optimization

Asynchronous Scheduling

The system achieves module decoupling through triple buffering:

1. Input Queue: Stores candidate responses generated by Rollout in shared memory
2. Processing Queue: Inference module asynchronously pulls data for reasoning
3. Output Queue: Reward calculation results write to Training module’s circular buffer

This design boosts system throughput by 3.8x, enabling 128 candidate responses per second on A100 clusters.

Memory Optimization Strategies

For training models with 100B+ parameters, the paper proposes three key optimizations:

• Gradient Partitioning: Stores optimizer states in CPU memory with ZeRO-3 technology to reduce GPU memory usage
• Activation Recomputation: Selectively recomputes activations for Transformer attention and feedforward layers
• Dynamic Offloading: Temporarily unloads non-critical layer parameters during backpropagation, reducing peak memory by 57%

Engineering Challenges and Solutions for Long-Sequence Reasoning

Generation Time Optimization

Multi-Token Prediction (MTP) technology achieves:

• Speculative Decoding: Predicts multiple tokens per timestep, selecting the highest confidence output
• Dynamic Length Adjustment: Adjusts predicted token count (2-5) based on historical generation speed
• Batch Optimization: Splits longest samples into sub-batches for parallel processing

Experiments show MTP reduces single-sample generation time from 12.4s to 4.7s for 2048-token texts.

Stability Enhancement Measures

The paper discloses three critical stability designs:

• Gradient Clipping: Applies global norm clipping (threshold=1.0) to policy gradients
• Reward Smoothing: Processes reward signal fluctuations using exponential moving average (α=0.9)
• Early Stopping: Triggers termination when reward improvement <0.1% over 5 consecutive training steps

These measures increase training convergence rate from 68% to 92%, significantly reducing model collapse risk.

Industrial Impact of Technical Transparency

DeepSeek’s practice is reshaping AI development paradigms:

1. Academic Research: Provides standardized RL benchmarks for reproducible research
2. Model Optimization: Public ablation experiment data offers parameter tuning references
3. System Design: Modular architecture becomes universal paradigm for training system design
4. Ethical Governance: Intermediate checkpoint disclosure enhances model interpretability

A cloud provider survey shows transparent schemes reduce model debugging cycles by 40% on average while improving hardware utilization by 25%. This open practice is driving the industry toward more open, efficient development models.

Conclusion: The leap from 22 to 86 pages represents more than expanded documentation - it marks a fundamental breakthrough in technical sharing philosophy. When training pipelines, infrastructure designs, and failure cases become fully transparent, AI development transforms from “black-box operations” to “transparent engineering,” laying the foundation for trustworthy AI ecosystems. This open practice suggests future LLM competition will shift from pure performance metrics to depth of technical transparency and reproducibility.