Technical Analysis: Chinese Full-Featured GPU Completes Training Adaptation for a Nearly 3-Trillion-Parameter Model
A Chinese GPU vendor recently reported that its flagship training-and-inference accelerator and self-developed software stack completed training adaptation for an open-source model with nearly 3 trillion parameters, running 400 consecutive optimizer updates stably on 16 cards across two nodes. This article explains what training adaptation means versus pretraining, examines the FSDP2, cross-node communication, and numerical-stability requirements behind the result, reviews the vendor’s broader large-scale training experience, and outlines remaining gaps between “can train” and “out-of-the-box training” for domestic training stacks.
Technical Analysis: Chinese Full-Featured GPU Completes Training Adaptation for a Nearly 3-Trillion-Parameter Model
A Chinese GPU vendor recently announced that its flagship training-and-inference accelerator card and self-developed software stack have completed training adaptation for an open-source model with nearly 3 trillion parameters. The system ran 400 consecutive optimizer updates stably on 16 cards across two machines. The announcement has once again brought a core industry question to the forefront: can domestic compute support frontier large-model training? This article interprets the meaning, key challenges, and industry value of this event from a technical perspective.
1. Clarifying the Concept: Training Adaptation Is Not Pretraining
One easily misunderstood concept should be clarified first. “Training adaptation” means enabling a given chip and software stack to run the complete training process of an open-source model correctly and stably. It does not mean that the model’s base pretraining was completed on this chip. This is at the same level as what the industry often calls “Day 0 access to the latest open-source model”: it validates hardware and software stack compatibility, stability, and engineering maturity, not the source of the original compute.
There are precedents in the industry. One large model’s technical report validated two hardware paths—NPU and general-purpose GPU—in parallel, and the industry generally understands this as validation at the inference-operator level; the source of its base pretraining compute was not publicly disclosed. Therefore, when reading such news, readers should distinguish three progressive capability levels:
- Inference access: the ability to load model weights and complete inference services, with the lowest barrier;
- Training adaptation: the ability to run the complete loop of forward pass, backward pass, and optimizer update, validating numerical correctness;
- Full pretraining: the ability to complete large-scale pretraining from scratch or from a checkpoint, with the highest requirements for cluster scale, interconnect bandwidth, and fault recovery.
This adaptation event corresponds to the second level. However, achieving continuous stability on a model with nearly 3 trillion parameters is already a relatively representative advance among domestic chips.
2. Key Technical Path for Cross-Node Training
According to the technical report disclosed by the vendor, this adaptation built a native FSDP2 multimodal training path and connected the complete链路 from model loading and sharding scheduling to gradient synchronization. We break down the technical points involved one by one.
1. Fully Sharded Data Parallel (FSDP2)
FSDP (Fully Sharded Data Parallel) is one of the mainstream parallel strategies for current large-model training. Its core idea is to shard model parameters, gradients, and optimizer states across multiple devices, temporarily all-gather the required shards on demand during forward and backward computation, and release them after computation. This greatly reduces per-card memory usage.
Compared with the first-generation implementation, FSDP2 uses a lower-level tensor sharding approach based on DTensor, supporting finer sharding granularity control and higher communication efficiency. Porting it “natively” to a domestic GPU software stack means reimplementing the underlying collective communication operators—all-gather, reduce-scatter, and all-reduce—and ensuring behavior consistent with mainstream frameworks. This is a nontrivial systems engineering task.
2. Communication Challenges in Cross-Node Joint Training
Although the two-machine, 16-card experimental configuration is not large, it already involves cross-node communication. Across nodes, collective communication must go through the network, typically RDMA or high-speed Ethernet, where latency and bandwidth become bottlenecks. Stable training requires:
- Correctness and fault tolerance of the communication library under different topologies;
- Numerical consistency of gradient synchronization—different floating-point reduction orders may introduce small differences, and these must not be amplified into training divergence;
- Sufficient validation of replacement implementations of NCCL-like communication libraries on domestic interconnect hardware.
3. Numerical Stability: What 400 Steps Mean
The technical report specifically emphasizes that 16 training processes completed 400 consecutive optimizer updates, with loss and gradient norm remaining finite throughout, and no numerical overflow or abnormal interruption occurred.
For very large model training, numerical overflow is a common source of failure. In mixed-precision training, the limited dynamic range of FP16/BF16, gradient explosion, and improper loss scaling can all lead to NaN. 400 consecutive stable steps indicate that:
- The numerical implementations of operators—especially matmul, softmax, layer norm, and optimizer kernels—have passed correctness validation;
- The mixed-precision strategy and loss scaling mechanism are working properly;
- A gradient-norm monitoring system has been established, which is a basic requirement for observability in large-scale training.
A simplified gradient-monitoring logic example is as follows:
for step in range(max_steps):loss = model.forward(batch)loss.backward()total_norm = clip_grad_norm_(model.parameters(), max_norm)if not torch.isfinite(total_norm):raise TrainingAnomaly(f"step {step}: grad norm overflow")optimizer.step()log(loss.item(), total_norm)
This is exactly the engineering practice corresponding to “loss and gradient norm remain finite throughout.”
3. Accumulated Results of “Domestic Chips Training Domestic Models”
This adaptation is not an isolated breakthrough. It is understood that the vendor has already achieved several practical results in large-scale training:
- Completed the full from-scratch pretraining of a scientific foundation model with hundreds of billions of parameters—direct proof of third-level capability;
- Supported full-stack native training of the world’s first 5D world model;
- Together with research institutions, completed full-pipeline training of an embodied brain model;
- Is currently advancing trillion-parameter large-model training.
From scientific models and world models to embodied intelligence, the diversity of training scenarios is precisely the best test of the software stack’s generality. Scientific computing emphasizes numerical precision, multimodal world models emphasize heterogeneous data processing, and embodied models emphasize multimodal inputs and long sequences. Covering these scenarios shows that the training framework’s abstraction capabilities and operator ecosystem have reached a relatively high level of maturity.
4. How Far from “Can Train” to “Out-of-the-Box Training”?
Despite notable results, the industry’s expectation for domestic training stacks is “out-of-the-box training”—that is, after users receive the chips, they can train mainstream open-source models without extensive manual adaptation. The remaining gaps include:
- Operator coverage: new model architectures, such as new attention variants and MoE routing, continue to emerge, and the operator library must keep up;
- Automated toolchain: capabilities such as automatic operator migration, automatic model conversion, and automatic training-task diagnosis determine adaptation cost;
- Large-scale cluster stability: fault detection, fast checkpoint recovery, and elastic scheduling for training on more than 1,000 cards remain hard battles;
- Ecosystem compatibility: the speed of version alignment with mainstream training frameworks and parallel libraries determines developer experience.
The vendor says it will continue to refine complete training capabilities from operator adaptation and multi-card coordination to an automated toolchain. The direction is consistent with this.
Conclusion
The training adaptation of a nearly 3-trillion-parameter model on a Chinese GPU validates the staged capability of the domestic software and hardware stack moving from inference to training and from single-card to cross-node operation. For developers, when judging the maturity of domestic compute, they should focus on the completeness of the training pipeline, numerical-stability validation, and the scale of real training cases, rather than paper specifications alone. As “domestic chips training domestic models” cases continue to accumulate, Chinese full-featured GPUs are moving from “usable” to “easy to use” in the field of large-model training.
