Validating Multimodal MoE Training on 16 Domestic GPUs Across Two Nodes
This article examines a training-pipeline validation rather than full-scale training of a 2.8-trillion-parameter model. It explains which components were reduced, why the complete vision tower was retained, how FSDP2 and MoE communication were used, and which metrics and environment details should be recorded when reproducing a similar validation.
Scope of the Validation
This was a link and pipeline validation, not a full training run. The original title’s reference to “running 2.8 trillion parameters” could be misleading: 2.8 trillion is the total parameter count of the target model. The validation did not train a model at that scale. Instead, selected structures were retained to verify whether the training pipeline worked correctly.
Why MoE Training Places Heavy Demands on Infrastructure
Mixture-of-Experts (MoE) models use sparse activation to separate the total parameter count from the amount of computation performed in a single forward pass. As the model grows, the training system must address three major challenges.
Communication
Expert parallelism routes tokens to experts distributed across different devices. As a result, communication changes from relatively regular All-Reduce operations to irregular All-to-All transfers. As the number of GPUs increases, both communication volume and traffic patterns become less predictable, which can reduce overall throughput.
Memory
Even when weights are stored in BF16, the weights of a 2.8-trillion-parameter model far exceed the memory capacity of a single GPU. Training also requires gradients and optimizer states such as the AdamW momentum and variance terms, causing memory consumption to grow substantially.
Stability
In large-scale parallel training, small changes in gradient norms can be amplified, eventually causing loss divergence or producing NaN and Inf values. Adding a vision tower introduces further uncertainty because the gradient magnitudes of the visual and language components can differ significantly.
How the Validation Was Reduced
The validation preserved key architectural structures while reducing the computational scale. The hardware configuration consisted of two nodes with 16 GPUs in total.
- Language backbone: 13 Transformer layers were retained, along with 16 routed experts. Each token activated the top eight experts. This configuration allowed the team to test dynamic routing and the computation path under imbalanced expert loads.
- Vision tower: The complete 27-layer vision Transformer was retained. According to the original rationale, the vision tower and the high-dimensional tensors produced by the downsampled feature pyramid interact with the language backbone through the cross-modal projection module. The complete gradient path therefore had to remain intact to verify multimodal training behavior.
The validation did not focus on the behavior of positional encoding under extremely long sequences. Instead, it prioritized gradient synchronization between the visual and language components, which the original analysis considered more likely to fail.
Distributed Strategy: FSDP2 and Communication
The validation used FSDP2, or Fully Sharded Data Parallel V2. Unlike DDP, which stores complete parameters on every GPU, FSDP2 partitions parameters, gradients, and optimizer states into shards distributed across devices.
The weights of each Transformer layer were evenly divided among 16 GPUs. During the forward pass, the GPUs used All-Gather to collect the complete weights for the current layer. After computation, each GPU released the shards that did not belong to it. The original analysis stated that this approach reduced the per-GPU memory peak from the hundreds of gigabytes required for full loading to the tens-of-gigabytes range.
For communication, the validation used an MCCL-type communication library. For MoE sparse routing, tokens were pre-aggregated by expert before cross-node communication. The original analysis stated that this helped balance traffic between the two nodes and avoid making a single network interface the bottleneck. However, it did not provide the MCCL version or specify the scope of its topology awareness.
Dynamic Sparse Attention Kernels
The original analysis noted that the model’s dynamic sparse attention could not rely entirely on general-purpose operators and automatic compilation. The team therefore implemented dedicated forward and backward kernels.
These kernels were described as hand-written for the target hardware instruction set to reduce the additional memory overhead caused by sparse-mask computation. Implementation details were not disclosed, and no standalone performance benchmark was provided. The available information is therefore insufficient to determine their efficiency from the original report alone.
Training Data and Results
The experiment trained from random initialization on the MedPix-VQA image-text dataset. Because pretrained weights could not be reused, all representations had to be learned from scratch, creating higher requirements for optimizer stability.
The reported logs included the following results:
- The average training loss from steps 21 to 70 was 8.9365, with a relatively rapid decline.
- The average training loss from steps 350 to 400 was 2.1802.
- Validation loss decreased from 6.5297 to 4.6873.
- Peak GPU memory usage was 44.80 GiB.
- Gradient clipping was not used.
These figures have important limitations. The original report did not disclose the batch size, sequence length, learning rate, mixed-precision configuration, whether activation recomputation was enabled, the statistical windows used for reporting, or the exact software versions.
The numbers should therefore be treated as logs from this validation rather than performance benchmarks. They cannot be directly generalized to other GPU types or configurations.
The original analysis reported no numerical overflow and considered the model capable of continuing to learn. However, this was only a short training run. It cannot establish the model’s final accuracy or generalization ability.
Engineering Lessons
Three aspects of the validation are particularly relevant to engineering teams.
Validate the Pipeline Before Scaling Up
When compute resources are limited, reducing the number of layers and experts while preserving the complete training workflow can expose data-loading timeouts, communication failures, and gradient-synchronization issues at a lower cost.
Overlap Communication with Computation
All-to-All communication in MoE systems is difficult to eliminate completely. However, operator fusion can allow communication to proceed during matrix multiplication and hide part of its latency. The original analysis attributed the memory-control effect to the combined use of communication compression and recomputation.
Treat Convergence as One Signal, Not the Only Standard
A smoothly declining loss generally indicates that training is functioning at a basic level, but silent data errors may not immediately appear in the loss curve. Teams should also use checksums, hardware monitoring, and metric monitoring, and compare results across multiple random seeds.
Industry Context
The original analysis stated that some vendors had already completed multimodal pretraining workflows on clusters with thousands of GPUs and were working toward adaptations for clusters with tens of thousands of GPUs. It also suggested that domestic software stacks for large-model operators were moving from merely being able to run models toward supporting actual training.
Because the original report did not provide verifiable market data, this article does not include specific shipment volumes or market-share figures.
Frequently Asked Questions
Is this validation equivalent to full-scale training?
No. The validation used a reduced model with fewer layers and experts than the complete target model. It can indicate whether parts of the training pipeline function correctly, but it cannot demonstrate the final performance of a 2.8-trillion-parameter model.
What environment and versions are required to reproduce a similar validation?
At a minimum, teams should record the GPU type, driver and framework versions, FSDP2 and MCCL versions, precision settings, batch size, sequence length, learning rate, and dataset size.
The original report did not disclose these details. As a result, anyone reproducing the validation must document and verify the configuration in their own environment.
