Adapting Large Multimodal Model Training to Domestically Developed AI Compute: 400-Step Stability Validation on 16 Cards Across Two Nodes

    This article describes how a leading Chinese cloud provider adapted a large multimodal model to domestically developed AI compute. It covers the distributed training design, including FSDP2 and expert parallelism, and verifies that the vision branch receives gradient updates. It also reports a 400-step training run on a reduced-size model across two nodes with 16 cards, along with loss, memory, and optimization results.

    Adapting Large-Scale Multimodal Model Training to Domestically Developed AI Compute

    As AI model parameter counts keep growing exponentially, training multimodal large models places unprecedented demands on the underlying compute infrastructure. When a model reaches the trillion-parameter scale and combines visual understanding, dynamic sparse retrieval, and tool calling, adding more hardware of one type no longer solves the core problem. Key open questions in AI infrastructure include how to train efficiently across a distributed cluster, how to keep numerical computation stable during long-sequence training, and how to make the vision encoder take part in end-to-end gradient updates rather than serve only as a data pipeline.

    Recently, a leading Chinese cloud provider reported that it had completed training adaptation for a well-known very large multimodal model. In a joint test on two nodes with 16 cards, the team ran a complete training workflow continuously for 400 steps. The work offers an engineering reference for applying domestically developed AI compute to complex large-model development. This article covers the technical background, training architecture design, key optimization paths, and validation results.

    Three Layers of Compute Challenges in Large Multimodal Models

    The model has about 2.8 trillion total parameters and supports context windows of up to one million tokens. Its architecture combines hybrid attention with a Mixture-of-Experts (MoE) design, and adds a vision encoder module and a cross-modal projection module. This layered, multimodal structure creates three main challenges.

    First, dynamic sparse retrieval and expert routing make computation irregular. The experts activated for each token vary during both inference and training. This dynamic behavior causes uneven load across compute nodes, and static graph partitioning struggles to use all available compute.

    Second, joint multimodal training puts much heavier pressure on memory management. Interleaving image and text data along the sequence dimension makes peak activation memory hard to predict. If optimizer states and gradient synchronization are not handled carefully, memory overflow becomes likely.

    Third, cross-node communication is highly sensitive to bandwidth. In a distributed setup that mixes data parallelism and model parallelism, the efficiency of gradient synchronization and parameter prefetch scheduling determines the overall scaling efficiency.

    Restructuring the Distributed Training Pipeline: Parallelism for Heterogeneous Routing

    To address these challenges, the provider did not limit itself to the standard patterns of conventional LLM training frameworks. Instead, it reworked the distributed training pipeline based on the model’s architecture. One core step was to keep the native runtime of a mainstream pretraining workflow framework and focus changes on how the underlying parallel strategies are attached.

    The training design includes a native FSDP2 (Fully Sharded Data Parallel) path. This path fully shards model parameters, gradients, and optimizer states, and distributes them across compute cards. Compared with the coarser sharding of traditional ZeRO stages, FSDP2 registers a separate communication handle for each layer’s parameters, so computation and communication can overlap at a finer granularity. In a multimodal model, the vision tower and the language backbone need different parallel strategies. The design lets the language backbone use sharded data parallelism, while the vision tower uses layer-wise pipeline parallelism. This balances memory use and compute efficiency.

    Data exchange across cards and nodes is handled by a unified low-level communication library. The library needs topology-aware communication primitives, tuned in depth, especially for the All-to-All communication produced by expert parallelism, so that no severe bandwidth bottleneck appears. In this validation, the communication mechanism supported multi-node RDMA networking. The two-node, 16-card setup ran without training interruptions caused by communication blocking during frequent gradient synchronization.

    Making the Vision Branch Trainable: From “Data Input” to “Gradient Update”

    Many current multimodal training efforts share a common misconception: the vision encoder is treated only as a feature extractor, frozen during training or used only in the forward pass. However, if image embeddings and cross-modal projections receive no real gradient feedback, the model’s understanding of visual semantics stays shallow, and true multimodal alignment cannot be achieved.

    A key result of this adaptation was confirming that the vision branch takes part in parameter updates within the end-to-end training pipeline. The team inspected gradient tensors during training. The modules checked included the vision tower’s image embedding layer, the first and last layers of the vision tower, the position encoding layer of the temporal block, and the cross-modal projection matrix. All of them produced non-zero gradients and parameter updates, so the vision weights changed in every optimizer step. By fully connecting the vision branch to the unified forward pass, backward pass, and parameter update flow, the model can dynamically calibrate semantic alignment between the two modalities during joint image-text training.

    400-Step Continuous Run