Domestic GPU Validates Training Pipeline for Trillion-Parameter MoE Multimodal Model

    A domestic GPU vendor recently validated the end-to-end training pipeline for a 2.8-trillion-parameter MoE-based natively multimodal model on its own hardware, completing 400 consecutive training steps on a 16-GPU, two-node setup without numerical overflow or abnormal interruption. This article explains why such models are hard to train, the trimmed-configuration methodology used for validation, the distributed training stack involved (operators, sharded memory management, collective communication), the disclosed loss and memory metrics, and the boundaries between pipeline validation and full-scale pretraining—offering practical lessons for teams planning large-model training on domestic compute platforms.

    1. Background: Domestic Compute Enters an Acceleration Phase for LLM Training

    Over the past two years, domestic AI chips in China have visibly accelerated their evolution from “capable of inference” to “capable of training.” Based on publicly disclosed information, compute solutions from multiple domestic chip vendors have successively supported high-load training tasks such as multimodal pretraining and reinforcement learning, with some solutions extending to thousand-GPU and even ten-thousand-GPU cluster scenarios. Cloud providers have also trained major versions of flagship large models on fully domestic chip clusters while maintaining high effective training rates. Third-party market research firms predict that shipments of high-end AI chips in the Chinese market will grow significantly year over year in 2026, with domestic solutions already accounting for an absolute majority of overall shipments.

    Against this backdrop, what truly differentiates platforms is no longer “whether the model runs,” but the systemic maturity of the training pipeline: whether operator coverage is complete, whether distributed strategies are reliable, whether long-duration training is stable, and whether loss curves behave as expected. Recently, the lab of a domestic GPU vendor disclosed a representative validation effort—adapting the training pipeline of a 2.8-trillion-parameter MoE-based natively multimodal model on its self-developed GPUs, and completing 400 consecutive training steps on a two-node, 16-GPU configuration without numerical overflow or abnormal interruption. This work provides a concrete sample for observing the real capability level of domestic compute in ultra-large-scale model training.

    2. Why Trillion-Parameter MoE Multimodal Models Are Hard to Train

    To appreciate the weight of this validation, one must first understand the architectural complexity of the target model. This class of natively multimodal MoE models typically contains three components that place extreme demands on training systems:

    1. Dynamic sparse attention. Compared with standard full attention, dynamic sparse attention dynamically selects the key-value pairs participating in computation based on input content. This causes per-GPU workload to fluctuate with data distribution, increasing the difficulty of load balancing and memory management.

    2. Expert routing. The MoE architecture dispatches tokens to different experts via a router. The Top-K activation strategy makes the number of tokens each expert actually receives highly imbalanced, easily causing the devices hosting “hot experts” to bear far greater memory and communication pressure than others—the classic expert load skew problem.

    3. Cross-modal structure. Beyond the language backbone, there is an independent vision tower and cross-modal projection modules. Whether the vision branch’s parameters effectively participate in updates, and whether gradients propagate back correctly, are aspects that pipeline validation must cover.

    The combination of these three means that multi-GPU communication volume (All-to-All dispatch, gradient synchronization) and peak memory usage (expert parameters, activation caches) are significantly higher than in a dense model of the same scale. For any GPU hardware and software stack, this is a comprehensive stress test of the operator library, communication library, and memory management capabilities.

    3. Trimmed-Configuration Validation: A Pragmatic Methodology

    This validation did not directly launch full-scale pretraining. Instead, it adopted a trimmed configuration:

    • The language backbone was truncated to 13 layers, with routed experts reduced to 16 and Top-8 activation;
    • The complete 27-layer vision tower and cross-modal projection modules were retained;
    • Training ran for 400 steps with random initialization on a public image-text QA dataset.

    This approach is engineering-sound. Full-scale pretraining is extremely costly, and if the pipeline has defects, problems usually surface early in training—numerical overflow, abnormal gradients, communication deadlocks, non-converging loss. Using a small-scale configuration to quickly close the full loop of “forward pass — gradients — optimizer update — cross-node communication,” then observing whether the loss trend is healthy, is a standard industry method for validating training functionality.

    Judging from the disclosed data, the validation results are convincing:

    Metric Early stage (avg. of ~steps 21–70) Final stage (avg. of last 50 steps)
    Training loss 8.9365 2.1802
    Validation loss 6.5297 4.6873
    Peak memory — 44.80 GiB

    Training loss and validation loss decreased in sync, with no rebound in validation loss, indicating the model was genuinely learning rather than overfitting. Confirmation that the vision branch participated in parameter updates shows the multimodal pipeline was fully connected. And 400 consecutive steps without abnormal interruption preliminarily demonstrates system stability.

    It is worth noting that, based on public disclosures, cases where vendors proactively reveal engineering details such as loss curves, gradient checks, and peak memory usage for trillion-parameter-class MoE multimodal models remain rare. Compared with conclusory statements like “adaptation successful,” the transparency of process data better reflects technical maturity.

    4. Systematic Enablement of the Distributed Training Pipeline

    From an engineering perspective, the focus of this adaptation lies not in single-GPU peak compute, but in the completeness of the distributed training stack. Public information indicates the validation covered three key layers:

    1. Operator layer. The forward computation and gradient implementations of hybrid-attention and MoE-related operators were completed. This is the most fundamental layer—and the one most prone to “hidden gaps.” Many chips perform well in inference scenarios precisely because the backward operators required for training are not yet complete.

    2. Memory management layer. A fully sharded data parallel approach was adopted, sharding parameters, gradients, and optimizer states. For trillion-parameter-class MoE models, optimizer states are typically several times the parameter count, making sharding a necessary means of fitting large models into limited memory.

    This strategy has mature reference implementations in the industry. An illustrative configuration is shown below:

    1. # Illustrative configuration for fully sharded data parallel (pseudocode)
    2. fsdp_config = {
    3. "sharding_strategy": "FULL_SHARD", # Full sharding of params/grads/optimizer states
    4. "offload_params": False, # CPU offloading not enabled in this case
    5. "mixed_precision": "bf16", # Mixed precision, balancing stability and efficiency
    6. "backward_prefetch": "BACKWARD_PRE", # Backward prefetch to reduce communication wait
    7. }

    3. Communication layer. Cross-node communication is handled by the vendor’s self-developed collective communication library, which must simultaneously support two types of workloads: gradient synchronization (All-Reduce) and expert token dispatch (All-to-All). The library’s stability and bandwidth utilization directly determine whether multi-node training can run for extended periods without interruption.

    5. Boundaries That Must Be Recognized: Pipeline Validation ≠ Full-Scale Training

    An objective reading of such disclosures is equally important. The vendor explicitly stated that this was only a pipeline validation under a trimmed configuration, intended to verify training functionality and loss trends; full-scale pretraining will be conducted separately. There are significant gaps between the two:

    • Scale gap: 400 steps on a trimmed configuration versus full pretraining that often runs for weeks on thousands of GPUs—these are not comparable in terms of communication pressure, fault-tolerance requirements, or long-duration stability;
    • Data gap: The validation used a single image-text dataset, whereas real pretraining involves massive heterogeneous data, data cleaning, and curriculum scheduling;
    • Efficiency gap: After the pipeline is connected, whether MFU (model FLOPs utilization) can reach a usable level is key to the actual economics of training—and this disclosure did not address it.

    Therefore, a reasonable conclusion is: domestic GPUs have demonstrated an embryonic capability to support the training pipeline of trillion-parameter-class MoE multimodal models, but more validation and iteration are needed before scaled, production-grade training.

    6. Engineering Takeaways

    For teams planning to train large models on domestic compute, this practice offers a referenceable methodology:

    1. Validate with a trimmed configuration first, then scale up: Use a small-scale configuration to close the functional loop first, focusing on loss curve health and peak memory, to avoid debugging on expensive configurations;
    2. Treat MoE load balancing as a first-class concern: When evaluating a platform, focus on All-to-All communication performance and the slowest GPU’s behavior under expert skew, rather than only average metrics;
    3. Pay attention to software stack completeness: Backward operator coverage, sharding strategy support, and communication library stability matter more to training success than on-paper single-GPU specs;
    4. Establish an observability baseline: Metrics such as loss, gradient norm, peak memory, and communication latency should be recorded throughout, accumulating data for subsequent scaled training.

    7. Conclusion

    From operator completion to sharded management, from cross-node communication to loss curve convergence, what this validation demonstrates is not the peak performance of a single chip, but the systemic maturity of a training pipeline. As the bidirectional adaptation between domestic compute and the large-model ecosystem continues to deepen, training capability for trillion-parameter MoE multimodal models is moving from “conceptually feasible” to “engineering-verifiable.” The next things to watch will be the progress of full-scale pretraining and actual training efficiency.