Trillion-Parameter Multimodal MoE Training on Domestic AI Accelerators: Adaptation and Validation
This article examines how to adapt trillion-parameter native multimodal Mixture-of-Experts models for training on domestic AI accelerators and an in-house software stack. It addresses memory, communication, and numerical stability challenges, unifies attention, expert routing, vision encoding, and cross-modal projection in a single training pipeline, and outlines a distributed design based on Fully Sharded Data Parallel and expert parallelism. It also presents a two-node, 16-accelerator validation and clarifies that functional validation is not the same as full-scale pretraining.
1. Why Training Trillion-Parameter Multimodal MoE Models Is Harder
Over the past year, mainstream large models have moved from text-only systems to native multimodal systems, while total parameter counts have been pushed into the trillion-parameter range. For example, one open-weight native multimodal model has approximately 2.8 trillion parameters, supports contexts of up to one million tokens, and combines a language backbone, a vision tower, and a cross-modal projection structure.
This type of model introduces engineering challenges at three levels:
- Memory wall: Even after partitioning, trillion-scale parameters, gradients, and optimizer states put heavy pressure on a single accelerator. Memory sharding and recomputation are therefore required.
- Communication wall: MoE architectures rely on dynamic expert routing. Tokens must be efficiently rearranged across devices according to routing results. When cross-modal image and text tensors are added, inter-device and inter-node communication becomes a throughput bottleneck.
- Numerical stability wall: Mixed-precision training, sparse attention, and large-scale MoE can easily trigger numerical overflow or abnormal gradients, causing training to stop midway.
Running such a model on domestic AI training/inference accelerators and an in-house software stack is therefore not just operator porting. It is a system-level adaptation problem.
2. Core Adaptation: Unifying Four Key Modules in One Training Pipeline
One of the hardest parts of native multimodal training is that the vision branch often “runs forward but does not learn.” In practice, this means the image-side gradient becomes zero, and training effectively degrades into text-only training.
To avoid this, four core modules must share the same forward pass, backward pass, and parameter update process:
Attention computation: The model uses a hybrid mechanism of linear attention and gated sparse attention. During adaptation, the mixed attention operators must remain numerically correct in backpropagation, while keeping memory usage controllable for long sequences.
Dynamic expert routing: Top-K routing determines which experts are activated for each token. Routing dispatch, expert computation, and result aggregation must work seamlessly together, with balanced load across experts.
Vision encoding: The full vision tower, which may contain dozens of layers, must be trained jointly with the language backbone so that image embeddings can enter the cross-modal fusion layers.
Cross-modal projection: The projection module aligns visual features with the language space. It is critical for image-text alignment and must produce nonzero gradients and participate in parameter updates.
Only when all four modules are included in one unified training pipeline can “multimodal” training be genuine, rather than text-only training with a visual input attached.
3. Distributed Design: Combining Fully Sharded Data Parallel with Expert Parallelism
For the training paradigm, the approach retains the automatic model-building workflow of the native training framework. Its core is Fully Sharded Data Parallel (FSDP): model parameters, gradients, and optimizer states are distributed across different GPUs, and parameters are gathered on demand through All-Gather operations. This significantly reduces per-GPU memory usage.
In more advanced implementations, FSDP is usually combined with the following mechanisms:
Expert parallelism (EP): Different experts are placed on different devices. Combined with FSDP, this helps spread the parameter pressure of extremely large MoE models.
Parameter prefetching: The next layer’s parameters or the next group of expert parameters are fetched in advance to hide communication latency.
Gradient communication scheduling: Gradient reduction is overlapped with forward and backward computation to reduce idle communication time.
Cross-device and cross-node collective communication: An in-house collective communication library handles operations such as All-Reduce, All-Gather, and Reduce-Scatter, providing the underlying guarantee for throughput.
The following pseudocode illustrates how sharding, expert parallelism, and chunked loss computation can be organized:
# Illustrative code: it conveys the parallelism and loss organization, not a specific framework APIfor step in range(num_steps):# 1) Split the batch globally across ranksbatch = next(dataloader)# 2) Forward: vision encoding -> cross-modal projection -> language backbone with MoE routingvision_feat = vision_tower(batch.images) # vision branchfused_feat = projector(vision_feat, batch.text)logits = language_backbone(fused_feat) # expert routing# 3) Chunked loss computation to reduce the memory peak with a large vocabularyloss = chunked_cross_entropy(logits, batch.labels)# 4) Backward and parameter update; FSDP handles gradient reduction and parameter updatesloss.backward()optimizer.step(); optimizer.zero_grad()
4. Validation Test Design: Running the Full Pipeline Across Two Nodes and Multiple Accelerators
To verify correctness and stability, a typical test used the following configuration: two servers, 16 accelerators in total, and 16 training processes, with each process handling one rank.
The model used a “trimmed configuration” to focus on functional validation:
- The language backbone used 13 layers.
- There were 16 routing experts with Top-8 routing.
- The full multi-layer vision tower and cross-modal projection were retained.
- The training dataset was an image-text visual question answering (VQA) dataset.
- The global batch size was 16, and the sequence length was 1024.
This configuration intentionally narrows the language backbone while preserving the full vision side. The purpose is to verify whether the vision branch truly participates in training, rather than merely serving as an input.
5. Key Optimization Points
To improve large-scale training efficiency, four optimizations are typically added:
Chunked loss computation: For large vocabularies, logits are processed in chunks when computing cross-entropy. This significantly reduces the memory peak during loss computation and is especially important for long-sequence, large-vocabulary scenarios.
FSDP and expert parallelism: Sharding and expert placement need to be carefully arranged to avoid both communication and memory bottlenecks.
Parameter prefetching and gradient communication scheduling: These techniques allow communication to overlap with computation, improving overall throughput.
High-performance attention operators: Operator implementations should be optimized for the hybrid attention mechanism to improve computational density and maintain numerical stability.
In a 400-step test, chunked loss computation was enabled. The other three optimizations were also adapted as capability reserves for larger-scale configurations.
6. Results and Scope
The two-node, 16-accelerator test led to three main conclusions:
Training remained stable: The run completed 400 consecutive optimizer updates without numerical overflow or abnormal interruption.
Convergence behaved normally: Training loss decreased from an early-stage average of about 8.94 to about 2.18 in the later stage. Validation loss also continued to decrease. Sampled device peak memory was about 44.8 GiB, which was within a controllable range.
The vision branch participated effectively: The input layer, output layer, and cross-modal projection of the vision tower all produced nonzero gradients and parameter updates. This shows that vision encoding truly entered training.
It is important to clarify the scope: this validation used a randomly initialized, trimmed configuration. It verifies functional correctness and stability, but it is not equivalent to full-scale pretraining of a complete model. Full-parameter pretraining still requires longer and larger-scale validation across convergence quality, throughput efficiency, and ultra-large-scale stability. Clearly distinguishing between “the pipeline runs” and “the model trains well” is essential when evaluating the maturity of domestic AI accelerator adaptation.
7. Practical Recommendations for Engineering Teams
Validate the pipeline first, then scale: Use a small configuration to confirm the closed loop of forward pass, backward pass, and parameter update, including nonzero vision gradients. Then increase the number of layers, experts, and nodes step by step.
Use peak memory as a core monitoring metric: Combine chunked loss computation, parameter sharding, and recomputation to keep memory peaks below hardware limits.
Treat collective communication as a critical path: The efficiency of cross-device and cross-node All-Reduce and All-Gather operations directly affects scalability. Communication-computation overlap should be continuously optimized.
Add numerical monitoring and checkpoint recovery: Set alerts for overflow, sudden loss changes, and uneven expert routing load to support stable long-duration training.
Overall, adapting trillion-parameter multimodal MoE models to domestic AI accelerators is a systems-engineering problem that combines distributed training strategy, operator quality, communication scheduling, and numerical stability. Only when every module genuinely participates in learning within a unified training pipeline, and when conclusions are constrained by rigorous and reproducible validation, can large-model training capability be reliably migrated to a self-controlled compute foundation.
