Introduction: New Opportunities for China-Developed Computing Power in Ultra-Large Model Training

As AI model parameters scale from 100 billion to 10 trillion, traditional training architectures face triple challenges: computing utilization efficiency, communication bandwidth, and software ecosystem compatibility. A China-developed computing platform recently validated the feasibility of training 10-trillion-parameter models through hardware architecture innovation and self-developed communication protocol optimization, marking a critical advancement in ultra-large-scale training capabilities. This article provides an in-depth technical analysis across three dimensions: hardware architecture, communication optimization, and software ecosystem development.

Hardware Architecture Innovation: Super-Node and Networking Breakthroughs

1.1 Ascend 950 Super-Node Architecture

The Ascend 950 super-node employs a 3D stacking design integrating 128 AI processors per node with PCIe 5.0 interconnects. Benchmark tests on ResNet-50 training achieved 92% computing utilization efficiency (+18% vs. previous generation). Key innovations include:

• Dynamic Power Management: Real-time temperature monitoring via hardware sensors with adaptive core frequency adjustment
• Hierarchical Memory Optimization: 640GB HBM3 memory + 8TB DDR5 cache supporting intelligent parameter sharding
• Self-Healing Mechanism: Redundant computing units enabling automatic failover during single-card failures

1.2 UBL128 Networking Architecture

For 10-trillion-parameter training requirements, the UBL128 architecture (128-card 3.2Tbps single-layer switching network) implements a two-stage fat-tree topology:

Level 1: 8x 400G core switches
Level 2: 16x 200G aggregation switches

Using RDMA over Converged Ethernet (RoCE), the architecture achieves 94.7% cluster bandwidth utilization (+35% vs. traditional solutions). Key optimizations include:

• Adaptive Congestion Control: Dynamic window adjustment based on ECN marking
• Topology-Aware Routing: Dynamic path selection according to network load
• Hardware Offloading Engine: Communication protocol processing on NIC FPGA

Communication Protocol Optimization: Breaking Hardware Bandwidth Limits

2.1 DeepEP Communication Stack Architecture

The self-developed DeepEP communication stack adopts a three-layer design:

User Interface Layer → Protocol Processing Layer → Hardware Abstraction Layer

Core innovations include:

• Hybrid Scheduling: Combining static task allocation with dynamic load balancing
• Zero-Copy Optimization: Memory mapping to reduce data copying operations
• Pipeline Overlap: Concurrent execution of communication and computation

2.2 Performance Benchmarks

In 128-card training clusters, DeepEP achieves:

• All-to-All Communication: 375GB/s bandwidth (93.75% of theoretical maximum)
• Gradient Aggregation: 127μs latency (-40% vs. industry solutions)
• Parameter Synchronization: <8% overhead in 10-trillion-parameter training

Key performance drivers:

1. Topology-Aware Algorithms: Automatic network topology recognition and path optimization
2. Hierarchical Compression: Hybrid precision quantization and gradient sparsification
3. Asynchronous Communication: Parallel operation of computing and communication units

Full-Stack Software Ecosystem: From Frameworks to Infrastructure

3.1 Open-Source Component Stack

The team has open-sourced a complete technology stack for China-developed computing power:

• Programming Language: TileLang supporting hybrid dynamic/static graph programming
• Compute Library: 200+ optimized operators covering mainstream model architectures
• Communication Library: DeepEP with NCCL protocol compatibility mode
• Distributed Framework: Automatic parallel strategy search and elastic training

3.2 Operator Optimization Methodology

Three-stage optimization achieves operator performance breakthroughs:

1. Algorithmic Optimization: Winograd algorithm for convolution computation reduction
2. Memory Optimization: Computation graph-level memory reuse
3. Hardware Mapping: Custom instruction scheduling for China-developed architectures

Performance improvement examples (matrix multiplication):
| Matrix Dimensions | Baseline (TFLOPS) | Optimized | Improvement |
|—————————|—————————|—————-|——————-|
| 8192×8192 | 12.4 | 45.7 | 268% |
| 4096×16384 | 8.9 | 32.1 | 261% |

3.3 Framework Adaptation Layer

Three-layer adaptation architecture addresses heterogeneous computing challenges:

Application Layer → Framework Adaptation Layer → Hardware Abstraction Layer

Key technologies:

• Automatic Operator Fusion: Combining multiple small operators into single kernels
• Dynamic Batching: Automatic batch size adjustment based on memory capacity
• Gradient Checkpointing: Recomputation techniques to reduce memory usage

Engineering Implementation: 10-Trillion-Parameter Training Deployment

4.1 Model Partitioning Strategy

3D parallel strategy for 10-trillion-parameter models:

• Data Parallelism: 128-way data sharding
• Pipeline Parallelism: 8 model stages
• Tensor Parallelism: 1/16 matrix operations per card

Dynamic load balancing maintains <5% utilization variance across computing units.

4.2 Training Process Optimization

Three key optimizations:

1. Mixed Precision Training: FP16+FP32 reducing memory usage by 40%
2. Gradient Accumulation: Simulating large batches through small-batch accumulation
3. Checkpoint Optimization: Hourly model state saving to minimize I/O overhead

Benchmark data shows 12 million card-hours required for complete training (-18% vs. theoretical minimum).

4.3 Fault Recovery Mechanism

Three-tier fault tolerance system:

• Hardware Level: ECC memory protection and link redundancy
• System Level: Process-level heartbeat detection and automatic restart
• Algorithmic Level: Gradient verification and parameter rollback

Simulated fault tests demonstrate <30-second recovery time with <0.001% data loss rate.

Challenges and Future Directions

Three challenges remain:

1. Ecosystem Compatibility: Enhancing mainstream framework support
2. Engineering Capability: Improving large-scale cluster operational efficiency
3. Energy Efficiency: Reducing power consumption per unit computing capacity

Future development directions:

• Compute-in-Memory Architecture: Exploring integration of novel storage media with computing units
• Optical Interconnect Technology: Researching silicon photonics for cluster communication
• Auto-Tuning Systems: Developing reinforcement learning-based parameter optimization tools

Conclusion: Strategic Value of China-Developed Computing Power

This technical breakthrough validates the feasibility of collaborative hardware-software innovation for China-developed computing power. As 10-trillion-parameter model training matures, China-developed computing solutions will play increasingly important roles in scientific research and industrial upgrading. Developers should monitor three key trends:

1. Deep optimization of heterogeneous computing architectures
2. Increased overlap between communication and computation
3. Continuous improvement of full-stack software ecosystems

Through sustained technical iteration, China-developed computing power has the potential to establish global competitive advantages in ultra-large-scale AI training, providing a solid foundation for AI industry development.