What a Domestic Chinese GPU Passing a Trillion-Parameter MoE Multimodal Training Test Means
A Chinese GPU vendor reported 400 training steps on a 2.8-trillion-parameter Mixture-of-Experts (MoE) multimodal model, run on 16 GPUs across two machines with no numerical overflow or interruptions. This article explains why a test like this is hard to pass, what the reported metrics show and do not show, and what remains before full-scale pretraining.
1. Background: A Rare Validation of a Trillion-Parameter Training Pipeline
On November 17, a domestic Chinese GPU vendor’s lab announced that it had adapted the training pipeline for a native multimodal large language model with 2.8 trillion parameters and an MoE architecture. Running on two machines with 16 GPUs in total, the team completed 400 consecutive training steps. The run reported no numerical overflow and no unexpected interruptions.
Four hundred steps is a very small slice of a pretraining run, which often spans hundreds of thousands of steps. Still, the test matters for engineering reasons. A training pipeline that handles the forward pass, gradient backpropagation, optimizer updates, and cross-node communication without failure shows that the training framework, operator library, and hardware stack work together. For a trillion-parameter MoE multimodal model, that check carries more weight than the short step count suggests.
2. Why Trillion-Parameter MoE Multimodal Models Are Demanding
To understand the difficulty, start with how the model’s architecture stresses the hardware and software stack.
Dynamic sparse attention creates memory and scheduling pressure. Compared with standard dense attention, dynamic sparse attention changes its computation pattern with each input. Token-to-token attention links do not follow a fixed topology. For GPUs, this means kernel execution paths are hard to predict, and workload is uneven across both time and space. The stack must handle kernel fusion, memory access locality, and adaptive scheduling well.
Expert routing amplifies multi-GPU communication. In an MoE model, each token activates only a few experts. This validation used Top-8 activation over 16 routed experts. When expert parameters are spread across many GPUs, each routing decision triggers large-scale All-to-All communication. The more widely experts are distributed and the more GPUs are involved, the more data must move across GPUs and nodes. Communication often becomes the bottleneck for training throughput. MoE training therefore places heavy demands on the bandwidth efficiency of the collective communication library and on its ability to overlap computation with communication.
The multimodal structure increases memory pressure. The model includes a 27-layer vision tower, a cross-modal projection module, and a language backbone. These branches differ noticeably in parameter size and activation shape. With limited memory, the system must hold parameters, gradients, optimizer states, and activations at once. That requires careful sharding and memory management.
3. Validation with a Reduced Configuration
The test did not use the full model. It used a reduced configuration:
- The language backbone was cut to 13 layers. The full model is deeper.
- The model kept 16 routed experts with Top-8 activation.
- The 27-layer vision tower and the cross-modal projection module were kept intact.
- Training started from random initialization on a public image-text question-answering dataset.
This design keeps the full multimodal data path while controlling model size. Trimming the language backbone lets the model fit in memory on 16 GPUs. Keeping the complete vision branch and projection module ensures that image encoding, cross-modal alignment, and text generation are all exercised. The goal was to check functional correctness and the trend of the loss curve, not to train a high-quality model.
The publicly disclosed figures are as follows. Average training loss fell from 8.9365 early in the run to 2.1802 in the final segment. Validation loss dropped from 6.5297 to 4.6873. Peak memory usage was 44.80 GiB. The vision branch’s parameters were confirmed to receive updates. Three signals support the claim that the pipeline is correct: the loss curve converged smoothly, no NaN or Inf values appeared, and validation loss fell alongside training loss.
The sharding setup can be summarized as follows (illustrative pseudocode):
# Illustrative full-sharding data parallel strategy (pseudocode)fsdp_config = {"shard_strategy": "full_shard", # Shard parameters, gradients, and optimizer states"mixed_precision": "bf16", # Lower memory footprint and bandwidth pressure"activation_checkpointing": True, # Recompute activations to save memory"comm_backend": "collective_lib", # Cross-node collective communication"moe_all_to_all": "overlap", # Overlap MoE routing communication with computation}
4. Where the Engineering Work Sits
The adaptation work falls into three layers.
Operator layer. The team implemented forward and gradient computation for the hybrid attention and MoE operators. MoE routing, expert computation, and result aggregation involve many irregular operations, and their backward passes are especially complex. A single error in gradient derivation would stop the loss from converging. A normally decreasing loss curve is therefore indirect evidence that the operators are correct.
Parallelism layer. The setup uses fully sharded data parallelism (FSDP-style) to shard parameters, gradients, and optimizer states. Even with the reduced configuration, a trillion-parameter model would barely fit on one GPU without state sharding. Combining sharding with activation recomputation is a common memory strategy for large models across the industry.
Communication layer. A collective communication library handles cross-node traffic. On the two-machine, 16-GPU topology, it must reliably support gradient synchronization and MoE All-to-All routing. Communication stability determines whether training can run for long periods without interruption.
5. Industry Context: Training-Side Adaptation Is Accelerating
Adaptation of domestic AI chips for large model training is moving quickly in China. Public disclosures include the following claims:
- One chip vendor said its thousand-GPU clusters are used by several LLM companies for multimodal pretraining and reinforcement learning, and that its precision is aligned with leading international products.
- Another vendor said in a half-year earnings briefing that its training adaptation covers several mainstream open-source model families and is extending to thousand-GPU and ten-thousand-GPU clusters.
- A cloud provider said an all-domestic cluster trained an important version of its in-house model, with an effective training rate of 97%.
- A further vendor said its general-purpose compute acceleration product has been adapted to hundreds of mainstream models, covering most non-closed-source models.
A third-party research firm forecasts that China’s high-end AI chip shipments will reach about 5 million units in 2026, up 83% year over year, with domestic solutions holding a share close to 90%.
What stands out in this case is the level of detail disclosed. The announcement did not stop at saying the adaptation was complete. It also reported continuous step counts, loss values, peak memory, and gradient checks that outside parties can verify. Public cases at this level of detail for trillion-parameter MoE multimodal models are still rare. The disclosure gives the domestic GPU training ecosystem a concrete reference point for reproducibility.
6. What the Test Does Not Prove
A reduced-configuration pipeline check does not equal a full-scale pretraining run. Moving from 16 GPUs to clusters of thousands or tens of thousands of GPUs requires clearing at least three hurdles:
- Cluster scaling. All-to-All efficiency drops as nodes are added. Fault domains grow, and checkpointing costs rise. Each of these needs a solution.
- Long-duration stability. Real pretraining runs continue for weeks or months. Hardware failures, thermal stability, and communication jitter all become more visible over that time.
- Effective training rate. The industry typically measures cluster quality by the share of time spent in effective training. Leading clusters reportedly reach above 95%. Reaching that level requires coordinated fault tolerance and fast recovery across software and hardware.
Conclusion
The value of this test lies less in the number of steps than in what it shows: the domestic GPU software stack can correctly handle the hardest operator, parallelism, and communication problems in a trillion-parameter MoE multimodal model. It also provides verifiable data points for engineering comparison. Developers and companies evaluating domestic compute should focus on loss curve quality, memory management, communication efficiency, and long-duration stability, rather than peak single-GPU performance. The path from pipeline validation to full-scale pretraining is still long, but the direction is clear.
