Domestic GPU Achieves Day-0 Adaptation for 3-Trillion-Parameter Open-Source Large Model

Background: New Compute Challenges Posed by 3-Trillion-Parameter Models

The leap in large model parameter counts from the hundreds of billions to the trillions—and now to 3 trillion—is no longer merely a matter of stacking more GPUs. Consider a recently open-sourced 3-trillion-parameter multimodal model: it includes not only a text encoder and a vision encoder but also MoE (Mixture of Experts) structures with sparse activation and dynamic routing. As a result, the model weight files reach tens of terabytes in size. This means that even the most advanced AI accelerator cards cannot hold the full weights in a single card’s memory—multi-card distributed inference frameworks are mandatory for coordination.

At the same time, Day-0 adaptation for open-source models—running on specific hardware on the very day the model is officially released—has become a key benchmark for AI infrastructure maturity. For any compute provider, failing to bring up model loading, service startup, and basic inference pathways at the earliest opportunity can leave them at a disadvantage in the developer ecosystem competition. This explains why AI chip vendors have been investing in Day-0 adaptation as a strategic capability.

Breaking Down the Adaptation Pipeline: From Model Loading to Multimodal Inference

Model Loading: Sharding and Pipelining of Large Weight Files

Weight files for 3-trillion-parameter models are typically stored in sharded formats (such as safetensors shards), with individual shard sizes usually ranging from 2 GB to 8 GB. During adaptation, the first challenge is high-performance file I/O. Traditional disk reads clearly cannot meet the requirements—a distributed file system or a high-throughput storage pool composed of local NVMe SSDs is necessary. In its adaptation process, one domestic GPU vendor adopted a multi-threaded pipeline reading strategy, distributing weight shards evenly across compute nodes according to tensor parallelism degree, while using a CPU memory pre-loading mechanism to avoid idle GPU windows waiting on I/O.

Notably, the vendor also tuned the memory management module of the MUSA software stack. Specifically, during the weight loading phase, an “allocate-on-demand” strategy was adopted rather than “full residency”—loading model structure metadata and critical weights first, then dynamically swapping in the parameters of the corresponding experts when inference requests arrive. This strategy is particularly well-suited to MoE-structured large models, because sparse activation means each inference only requires the weights of a small number of experts, so there is no need to keep all parameters resident in memory.

Service Startup: Orchestration Optimization for Distributed Inference Engines

Model service startup involves coordination across multiple layers: inference engine initialization, establishment of distributed communication groups, KV Cache pre-allocation, and health check mechanisms. For 3-trillion-parameter models, service startup can take several minutes, and if any node fails midway, the entire startup process must restart. During adaptation, the vendor optimized distributed communication primitives—changing the originally globally-synchronized initialization flow to staged asynchronous initialization, allowing each node to report its status after completing local resource preparation rather than waiting for all nodes to be ready before proceeding.

Additionally, given the input differences between text and image modalities, the adaptation team designed independent tensor shape preprocessing modules. Text inputs, after tokenizer processing, can be unified into fixed-shape token sequences, but image inputs involve variable sizes, variable resolutions, and EXIF orientation information, requiring dynamic adjustment of the ViT (Vision Transformer) encoder’s patch embedding parameters. The vendor provided dynamic shape support through the MUSA runtime, registering multiple specialized kernels in the inference engine and selecting the optimal implementation at runtime based on actual image dimensions, thereby avoiding repeated compilation overhead caused by shape changes.

Inference Pipeline: Coordinated Scheduling of Text and Images

Models like Kimi-K3.1 Pro currently use a cascaded architecture of “vision encoder + language model.” In the inference pipeline, images are first converted by the vision encoder into visual token sequences, then concatenated with text tokens and fed into the language model. On the MTT S5000, the vendor implemented joint graph optimization of the vision encoder and language model—passing the vision encoder’s output tensor directly to the language model’s embedding layer through high-speed on-chip interconnect, eliminating the host memory copy step and reducing end-to-end latency by approximately 30%.

At the multi-card inference level, the team adopted a hybrid strategy of tensor parallelism and data parallelism. Since the vision encoder has relatively few parameters (approximately 300 million), it can reside entirely in a single card’s memory; the language model’s MoE layers are distributed across multiple cards by expert, with routing decisions made centrally on the CPU side to avoid frequent inter-card communication. Measurements show that with an 8-card configuration, this approach keeps first-token latency under 1.2 seconds (with a single image plus 128 text tokens as input), meeting the service-level agreement requirements of cloud production environments.

Software Stack Evolution: MUSA and CUDA Ecosystem Compatibility Design

Any domestic GPU adaptation effort must address the software ecosystem question. The vendor’s MUSA software stack is designed with a CUDA-like programming model, enabling large model inference frameworks developed on CUDA (such as vLLM and TensorRT-LLM) to complete adaptation through low-cost source-level migration. Specifically, MUSA provides function signatures and memory management models that are highly aligned with NVIDIA CUDA APIs—developers only need to switch the compilation target to the MUSA platform and handle a small amount of architecture-related inline assembly or PTX instruction replacement.

However, this compatibility is not without cost. Due to underlying hardware architecture differences, certain CUDA optimization techniques (such as warp shuffle instructions and explicit shared memory management) may not achieve the same efficiency on MUSA. The adaptation team therefore developed a kernel template library specifically for the MUSA architecture, including commonly used operators such as FlashAttention, RoPE positional encoding, and RMSNorm, and used an auto-tuning framework to scan performance across different batch size and sequence length combinations to select optimal tile sizes and thread organization.

Industry Perspective: The Coupling Point of Compute Self-Sufficiency and Open-Source Models

The industrial significance of this adaptation event goes beyond a single technical validation. On one hand, the open-sourcing of a 3-trillion-parameter model means global developers can access a capability foundation on par with commercial closed-source models, and the fact that a domestic GPU completed Day-0 adaptation demonstrates that domestic AI compute can already handle frontier international models. On the other hand, it reflects the key to compute ecosystem building—not simply running models on hardware, but providing a complete toolchain for development, tuning, and deployment so that developers can iterate efficiently on domestic platforms.

From a cost perspective, sparse MoE models like Kimi-K3.1 Pro activate only about 10%–15% of parameters during inference, making it feasible to build large-scale inference clusters on domestic GPUs. Taking the MTT S5000 as an example, while its memory capacity and bandwidth are not industry-leading, sparse computation and model parallelism strategies enable it to deliver acceptable inference throughput at a lower total cost of ownership (TCO). For budget-constrained research institutions and small-to-medium enterprises, this is undoubtedly attractive in practical terms.

Practical Recommendations: Deploying Large Open-Source Models in Heterogeneous Environments

For teams planning large model inference infrastructure, decisions across the following three dimensions are critical. First, assess the match between model structural sparsity and memory requirements. If the MoE layer proportion is high, prioritize multi-card interconnect solutions with high tensor parallelism; if the model is dense, consider increasing per-card memory or introducing KV Cache quantization (such as FP8) to compress memory usage. Second, focus on the progress of inference framework support for domestic hardware rather than simply comparing per-card floating-point throughput. A mature framework that fully utilizes memory bandwidth is often more practically valuable than hardware with higher peak compute but a weak software ecosystem. Finally, establish a top-down staged acceptance process—first validate the pipeline with single-request latency as the target, then gradually increase concurrency pressure to expose distributed communication bottlenecks, and finally run long-duration stability tests to check for memory leaks and GPU temperature drift and other operational issues.

It is foreseeable that as open-source model scales continue to grow and domestic GPU software stacks undergo deeper iteration, more enterprises will gravitate toward the combination of “open-source models + domestic compute.” This not only reduces dependence on a single commercial cloud service but also provides greater flexibility for customized AI application deployment. In this process, technical adaptation practices like those described in this article will become important knowledge accumulation for the industry.