Training a 2.8T-Parameter MoE Multimodal Model on Domestic GPUs: Engineering Practice and Validation
A domestic GPU vendor recently validated a full training pipeline for a 2.8-trillion-parameter MoE-native multimodal model, completing 400 consecutive training steps on a 2-node, 16-GPU setup without numerical overflow or abnormal interruptions. This article examines the technical challenges of trillion-parameter MoE multimodal training, the trimmed-down validation strategy used, the underlying operator implementations and sharded parallelism, and the gap that remains between pipeline validation and full-scale pretraining.
As large model parameter counts push toward the trillion scale, Mixture-of-Experts (MoE) architectures and multimodal capabilities have become central directions of underlying technology evolution. Recently, a domestic GPU vendor completed training pipeline adaptation for a 2.8-trillion-parameter MoE-native multimodal model in a lab environment, running 400 consecutive training steps on a two-node, 16-GPU configuration without numerical overflow or abnormal interruption. This milestone not only demonstrates the viability of domestic hardware in complex large-model scenarios, but also provides a valuable engineering reference for systematically bringing up a distributed training pipeline.
Technical Barriers of Trillion-Parameter MoE Multimodal Architectures
With sparse activation, MoE architectures dramatically expand model capacity without proportionally increasing inference compute, making them a preferred choice for trillion-parameter models. However, a natively multimodal model at the 2.8-trillion-parameter scale, with its complex dynamic sparse attention mechanisms and expert routing, imposes severe demands on the underlying hardware system.
- Communication bottlenecks: Expert routing in MoE requires high-frequency, imbalanced All-to-All communication between GPUs. When input tokens are dispatched to different expert nodes, cross-GPU and cross-node data movement surges, placing extreme demands on network bandwidth and the scheduling capability of the communication library.
- Memory management challenges: Multimodality introduces a vision tower and cross-modal projection modules. Interwoven with the massive language backbone, these components intensify volatility and fragmentation in memory usage. The intermediate states produced by dynamic sparse attention during computation can also easily trigger memory peak overflows.
A Trimmed-Down Validation Plan for the Distributed Training Pipeline
In this engineering validation, the team adopted a scientifically trimmed configuration to focus on bringing up the training pipeline and verifying core functionality. This approach controls validation cost while still effectively exposing potential operator-level and communication-level defects.
The model structure was trimmed as follows:
- Language backbone: 13 layers were retained for training.
- Expert configuration: 16 routed experts were kept, with a Top-8 activation strategy.
- Multimodal modules: The full 27-layer vision tower and cross-modal projection modules were preserved.
- Data: Training was performed with random initialization on a publicly available image-text VQA dataset.
On the two-node, 16-GPU setup, training showed a healthy convergence trend. Training loss converged from an average of 8.9365 over steps 21–70 to an average of 2.1802 over the final 50 steps, while validation loss dropped from 6.5297 to 4.6873. Notably, with multimodal parameters participating in updates, peak memory usage was held strictly to 44.80 GiB. This shows that the underlying memory allocation and reclamation mechanisms remained robust under the pressure of multimodal data flows. This engineering approach—focused on functional and trend validation—provided reliable data for verifying the correctness of underlying operators and parallelism strategies.
Operator Implementation and Sharded Parallelism
For distributed large-model training, systematically bringing up the end-to-end pipeline matters far more than chasing peak single-GPU compute. In this adaptation, the GPU team optimized everything from low-level operators to the upper-level parallelism framework.
- Core operator implementation: The team focused on implementing forward propagation and gradient computation for hybrid attention mechanisms and MoE-related operators. This requires the hardware instruction set to efficiently support complex tensor operations while guaranteeing the mathematical precision of gradients during backpropagation, avoiding numerical overflow.
- Sharded parallelism management: To handle the memory pressure of extremely large parameter counts, the system relies on mainstream Fully Sharded Data Parallel (FSDP) mechanisms for fine-grained sharding of model parameters, gradients, and optimizer states. Each GPU holds only a portion of the state, aggregating it via communication when computation requires it, thereby reducing per-GPU memory usage by a large factor.
- Cross-node communication scheduling: Cross-node data exchange is fully scheduled by a high-performance collective communication library. Through topology-aware optimization and pipelining, the library effectively hides compute-communication latency, ensuring low latency and high throughput for All-Reduce and All-to-All operations across nodes.
Accelerating Evolution of the Domestic AI Compute Ecosystem
From an industry-wide perspective, the pace of adaptation between domestic GPUs and large models continues to accelerate, with the ecosystem maturing rapidly. Multiple domestic compute vendors have disclosed recent progress:
- One leading compute vendor completed multimodal pretraining and reinforcement learning on a thousand-GPU cluster, with training accuracy successfully aligned to mainstream international products.
- An AI chip company continues to advance training adaptation for mainstream open-source large models, extending toward thousand- and ten-thousand-GPU cluster scenarios.
- Baidu AI Cloud announced at a technology conference that its self-developed compute cluster completed training of a major large-model version, achieving an effective training rate of 97%.
- Another vendor disclosed that its heterogeneous computing platform has adapted over 400 mainstream large models, covering the vast majority of non-closed-source models.
Together, these developments show that domestic compute is moving rapidly from “basically compatible” to “highly efficient.” The depth and breadth of hardware-software co-optimization is becoming a core driver in breaking the compute monopoly.
From Pipeline Validation to Full-Scale Pretraining: The Remaining Gap
Although the two-node, 16-GPU pipeline validation provides a reproducible engineering reference point for domestic GPUs on the training side, a significant gap remains before full-scale pretraining of trillion-parameter models becomes reality.
Pipeline validation only proves the mathematical correctness of a single forward and backward pass along with basic communication capability. Full-scale training presents far more complex engineering challenges:
- Cluster scaling: When node counts grow from dozens to thousands, communication topology complexity rises exponentially. Network congestion control, routing optimization, and inter-node load balancing become decisive for overall throughput.
- Long-term stability: During continuous pretraining spanning weeks or months, long-tail issues such as hardware aging, microsecond-level network jitter, and operator precision drift will inevitably surface. Efficient fault tolerance, checkpoint-based resumption, and elastic scheduling are the foundation for maintaining a high effective training rate.
- Global memory and communication optimization: With the full-scale model, expert parallelism makes cross-machine communication the norm. How to overlap computation with communication and optimize memory swap-in/swap-out strategies to improve MFU (Model FLOPs Utilization) remains a hard problem to solve.
According to industry consulting forecasts, high-end AI chip shipments in the Chinese market are expected to reach roughly 5 million units in 2026, an 83% year-over-year increase, with domestic solutions potentially approaching a 90% share (including cloud vendors’ self-developed ASICs). Facing such broad market prospects, domestic GPU vendors must bridge the gap from trimmed configurations to full-scale training, continuing to invest in cluster-level system optimization and high-availability scheduling to secure a position in the competition for the compute foundation of the large-model era. This pipeline validation is undoubtedly a solid starting point, signaling the full awakening of domestic compute in the trillion-parameter era.
