Training Trillion-Parameter Multimodal Models on Domestic GPUs: A Practical Adaptation Path
How can domestic GPUs support training of trillion-parameter multimodal models? This article breaks down a real adaptation effort: implementing mixed-attention and MoE routing operators, combining FSDP2 sharding with a custom collective communication library, applying chunked loss computation, and validating correctness over 400 continuous training steps on a 16-card, two-node setup—including gradient-level verification that the vision branch was genuinely learning.
Background: Infrastructure Challenges of Ultra-Large Multimodal Models
Open-weight, natively multimodal large models continue to grow in scale. Models with a trillion total parameters and million-token context windows typically share several technical characteristics:
- Mixture-of-Experts (MoE) architecture: a routing mechanism dynamically activates a subset of parameters across many experts, expanding model capacity while keeping compute costs bounded;
- Hybrid attention mechanisms: linear attention combined with dynamic sparse retrieval reduces computational complexity in long-context scenarios;
- Visual understanding and tool-calling capabilities: beyond text, the model must handle image encoding and joint image-text cross-modal training.
These features pose three challenges for training infrastructure: compute density (hybrid attention demands high-performance custom operators), memory management (even with sparse activation, parameters and optimizer states of a trillion-parameter model consume enormous VRAM), and multi-GPU communication (expert routing introduces All-to-All traffic and cross-node synchronization overhead).
Recently, a domestic GPU vendor announced training adaptation progress showing that its flagship AI training-and-inference accelerator card, together with its software stack, has completed adaptation of the training pipeline for models of this scale. In a two-node, 16-GPU environment, the system ran 400 consecutive training steps with no numerical overflow, no abnormal interruptions, and genuine gradient updates in the vision branch. This article examines the key elements of that adaptation work from a technical perspective.
Four Core Modules of Training Adaptation
Adapting a large-scale multimodal model for training is far more than “getting the forward pass to run”—every module must be integrated into a unified forward, backward, and parameter-update pipeline. This adaptation covered four areas:
- Attention computation: high-performance operators for the hybrid attention mechanism, ensuring correct forward computation and precise backward gradients;
- Dynamic routing: MoE expert routing involves Top-K selection and load balancing, requiring correct updates of routing gradients and expert parameters in a distributed environment;
- Vision encoding: the vision tower’s image embeddings and multi-layer feature extraction must join the training pipeline, rather than serving as frozen input data;
- Cross-modal projection: the projection module that aligns images and text is critical to multimodal training, and its gradient flow must be fully connected.
Together, these four modules determine the functional correctness of the training pipeline—any broken gradient or numerical anomaly gets amplified over long training runs.
Distributed Training Strategy: Layered Parallelism Within a Unified Pipeline
The adaptation preserves the native workflows of mainstream training frameworks without invasive modifications to the model architecture. Its core consists of the following layers of parallelism and management mechanisms:
FSDP2 Fully Sharded Data Parallelism
FSDP2 shards model parameters, gradients, and optimizer states across GPUs, dramatically reducing per-card memory pressure. Compared with traditional replicated data parallelism, fully sharded approaches are essential for trillion-parameter models. A conceptual illustration of the sharding logic:
# Conceptual illustration: FSDP2 shards states across GPUs# param_shard / grad_shard / optimizer_state_shard# all-gather on demand during forward; reduce-scatter gradients after backward
Unified Communication Scheduling
Cross-GPU and cross-node collective communication is handled by a self-developed communication library, filling the role commonly played by NCCL. In MoE scenarios, expert parallelism introduces additional All-to-All traffic, so the communication library must coordinate with FSDP2 to prevent communication and computation from competing for bandwidth. This work also included parameter prefetching and gradient-communication scheduling optimizations, overlapping communication with computation as much as possible to improve overall throughput.
Chunked Loss Computation
For long sequences and large vocabularies, computing the full loss in one pass creates memory spikes. Chunking the loss computation significantly reduces activation memory usage. The 400-step training run enabled this optimization—one of the key reasons the sampled peak device memory stayed around 44.8 GiB.
Two-Node, 16-GPU Test: Correctness Validation on a Trimmed Configuration
It is important to emphasize that this test used a randomly initialized trimmed configuration, not full-scale pretraining of the complete model. The goal was to validate the functional correctness and stability of the training pipeline. The configuration and results:
| Item | Configuration / Result |
|---|---|
| Hardware | Two servers, 16 accelerator cards |
| Language backbone | 13 layers, 16 routed experts (Top-8 routing) |
| Vision component | Full 27-layer vision tower and cross-modal projection retained |
| Dataset | Image-text VQA-style multimodal dataset |
| Global batch size / sequence length | 16 / 1024 |
| Training steps | 400 consecutive optimizer updates, no interruptions |
| Training loss | Steps 21–70 average 8.9365 → final 50 steps 2.1802 |
| Validation loss | Step 100: 6.5297 → Step 400: 4.6873 |
| Peak memory | 44.80 GiB |
Steady loss reduction is the first line of evidence for pipeline correctness. More critical, however, is verifying actual participation of the vision branch: the team confirmed that the vision tower’s image embeddings, its first and last layers, and the cross-modal projection module all produced non-zero gradients and parameter updates—proving the vision encoder genuinely participated in training rather than idling as a passive data input. Gradient-level checks like this are an easily overlooked yet extremely important part of multimodal training validation.
Lessons for the Domestic Compute Ecosystem
Several generalizable lessons for large-model training adaptation can be drawn from this work:
- Prioritize functional correctness: validating gradient flow and numerical stability on a trimmed configuration before scaling up is an effective way to control trial-and-error costs;
- Co-evolve the software stack with the hardware: high-performance custom operators, communication-library scheduling, and memory-chunking optimizations are all indispensable—raw compute alone cannot support trillion-parameter models;
- Ecosystem compatibility beats proprietary modifications: preserving native workflows of mainstream training frameworks (such as FSDP2 and AutoModel) substantially lowers the barrier for migrating community models to domestic hardware;
- Push validation down to the gradient level: declining loss only shows overall convergence; module-level non-zero gradient checks confirm that each multimodal branch is actually “learning.”
As the adaptation deepens, optimizations such as high-performance KDA operators, expert-parallelism coordination, parameter prefetching, and gradient-communication scheduling are now in place, providing a foundation for larger training configurations. For developers following the domestic GPU ecosystem, this practical path demonstrates a clear progression from “it runs” to “it trains” to “it scales.”
