Domestic GPU Adapts Trillion-Parameter MoE Model: Enabling Full-Chain Multimodal Inference

Background: Deployment Challenges of Ultra-Large Sparse Models

Recently, a frontier large language model series officially open-sourced its first Pro version. This model features up to 3 trillion sparse parameters and supports an ultra-long context window of 1 million tokens. Models of this scale impose extremely stringent requirements on underlying computing hardware and software stacks.

From a technical perspective, a 3-trillion-parameter model typically adopts a Mixture-of-Experts (MoE) architecture. During inference, it only activates a subset of experts for each token, thereby controlling computational costs while maintaining model capacity. However, the massive parameter count means the model weight files can reach the terabyte (TB) level, presenting systemic challenges for VRAM capacity, memory bandwidth, cross-card communication, and task scheduling within the inference engine. Traditional single-card or even single-node deployment solutions are almost impossible to support such models; relying on multi-card collaboration and refined memory management strategies is mandatory.

1. Hardware Foundation: Architecture of the AI Training and Inference Accelerator

The core hardware foundation for this adaptation work is a specific computing accelerator card designed for integrated training and inference. Based on a domestically produced full-function GPU architecture, its key features include:

• Unified Architecture Support: The same hardware platform accommodates both training and inference workloads, reducing the cost for users to migrate models across heterogeneous hardware.
• High VRAM Capacity and Bandwidth: Provides foundational guarantees for accommodating large model weights and intermediate activation values, supporting larger batch sizes and longer sequences.
• MUSA Software Stack: Provides a CUDA-like programming model and a rich set of operator libraries, allowing models from mainstream deep learning frameworks (like PyTorch) to run with low migration costs.

In actual deployment, multiple accelerator cards form a cluster via high-speed interconnects, creating a unified VRAM pool. Taking the 3-trillion-parameter model as an example, even if stored in mixed precision (like BF16), the model weights require at least about 6TB of VRAM. This dictates the necessity of combining multi-card tensor parallelism with pipeline parallelism strategies.

2. Software Stack Synergy: The Key Role of the MUSA Platform

The MUSA software stack played a bridging role in the Day-0 adaptation. The core problems it solved include:

1. Operator Mapping and Optimization: Operators in the model such as MoE routing, matrix multiplication, and attention calculations need to be efficiently mapped to the underlying instruction set. The MUSA compiler and runtime libraries reduce kernel launch overhead and increase computational density through operator fusion and memory access optimization.
2. Memory Management Strategy: For ultra-large models, MUSA provides virtual VRAM pools and unified addressing capabilities. This allows developers to bypass physical VRAM limits, combining CPU memory and disk caching to achieve “over-VRAM” loading.
3. Collective Communication Library: Multi-card parallel computing relies heavily on efficient communication primitives like All-to-All and AllReduce. The MUSA communication library is tuned for specific interconnect topologies, reducing the impact of communication latency on overall throughput.

A notable detail is that during the adaptation process, the team connected the complete pipeline of model loading, service startup, and inference requests. This means not only can model weights be correctly loaded onto the device, but the service framework can also stably receive requests, schedule computations, and return results.

3. Implementation Points for Text and Image Inference Pipelines

The model supports not only pure text input but also mixed-modal inference with images and text. At the implementation level, this involves two key subtasks:

3.1 Text Inference: KV Cache Management under Long Contexts

For a 1-million-token context window, the VRAM consumption of the KV Cache is a massive engineering challenge. Assuming a hidden layer dimension of 8192 and 80 layers, a single token’s KV Cache in BF16 precision consumes about 2.6MB. One million tokens would require about 2.6TB of VRAM. This requires the inference framework to feature:

• PagedAttention Mechanism: Manages KV Cache in blocks, allocating on demand to reduce fragmentation waste.
• Prefix Sharing: For identical prefix parts in multi-turn conversations or batched requests, the KV Cache is reused to significantly reduce redundant computation.
• Sliding Window and Summary Compression: For ultra-long contexts, combining sliding window attention with key information summary extraction may be necessary to control computational complexity.

3.2 Image Inference: Integration of the Visual Encoder

Image inputs are typically first processed by a visual encoder (such as ViT) to convert into a visual token sequence, which is then concatenated with the text token sequence and fed into the main large model network. During this process, attention must be paid to:

• Variable Resolution Support: Image scaling strategies affect the number of encoded tokens, requiring dynamic adjustment during the preprocessing stage.
• Positional Encoding Alignment: Visual tokens and text tokens have different positional encoding rules and must be correctly fused to avoid information confusion.
• Multi-Image and Interleaved Input: When the input contains multiple images interleaved with text, the sequence’s order identifiers must be maintained correctly.

4. Deployment Practice: The Complete Flow from Model Loading to Service Startup

In practice, the deployment team followed this workflow to complete the adaptation:

1. Model Format Conversion: Converted the open-source model weights into a format loadable by the MUSA platform (such as MUSA SafeTensors), handling partitioning and quantization options during the process.
2. Parallel Strategy Configuration: Determined the split dimensions for tensor and pipeline parallelism based on the available number of cards. For example, in an 8-card environment, combinations like TP=8 or TP=4+PP=2 can be applied.
3. Inference Engine Integration: Utilized a service framework compatible with the OpenAI API protocol (such as the MUSA version of vLLM or TGI), injecting model paths and parallel parameters via configuration files.
4. Startup Verification: Executed load testing scripts to verify basic capabilities like text generation and image understanding, while monitoring VRAM usage and request latency.

Pseudocode example is as follows:

# Conceptual example, not actual API
from musa_llm import MusaEngine, MusaConfig
config = MusaConfig(
    model_path="/mnt/models/kimi_k3.1_pro",
    tensor_parallel_size=4,
    pipeline_parallel_size=2,
    dtype="bfloat16",
    max_model_len=1000000,
    gpu_memory_utilization=0.9
)
engine = MusaEngine(config)
engine.start()
result = engine.chat(
    messages=[{"role": "user", "content": "Describe this image", "image": "path/to/img.jpg"}]
)

In actual operation, stability is a key metric. Due to the massive scale of the model, any improper parameter configuration can lead to Out-Of-Memory (OOM) errors or communication deadlocks. The adaptation team ultimately achieved a stable service state by repeatedly adjusting parameters like the KV Cache ratio and communication timeout limits.

5. Implications for AI Infrastructure Selection

The completion of this Day-0 adaptation carries multiple industry significances:

• Ecosystem Maturity: Domestic full-function GPUs are no longer limited to small and medium models; they are now capable of supporting frontier ultra-large models.
• Necessity of Software-Hardware Co-design: Hardware is merely the carrier; deep optimization within the software stack (compilation, communication, scheduling) is the key to unlocking computing power.
• Long Context and Multimodality Becoming Standard: Future AI applications will gradually evolve towards agentic and multimodal interactions. Infrastructure must be adapted in advance to handle processing demands at the million-token level.

For enterprises and developers, when selecting an AI computing platform, in addition to focusing on peak hardware computing power, they need to evaluate more closely:

• The compatibility degree of the software stack with mainstream frameworks and model architectures.
• The completeness of the full-chain toolchain from model download to service deployment.
• The vendor’s response speed to frontier models (Day-0 adaptation capability).

6. Future Outlook and Technology Trends

As model parameter scales continue to grow, the following technological directions will be the focus of the industry:

1. Sparsification and Quantization: More aggressive quantization schemes (like FP8, INT4) will reduce VRAM pressure, but require a tradeoff with accuracy loss.
2. Heterogeneous Computing: Mixed scheduling across CPU + GPU + NPU, assigning different operators to the most suitable computing units.
3. Distributed Inference Engine Optimization: Smarter request scheduling strategies, such as Continuous Batching based on load, to increase throughput.

The rapid adaptation of this domestic GPU to the 3-trillion-parameter model marks a transition for domestic AI computing power from “usable” to “highly effective.” As the ecosystem further matures, more ultra-large models will run efficiently on domestic hardware, providing AI services across various industries.

Conclusion

Completing full-chain adaptation just days from model open-sourcing is the result of the synergistic effort of multiple technology layers: hardware architecture, compilers, runtimes, communication libraries, and inference frameworks. In the future, as model parameters move toward the 10-trillion level, this deep software-hardware integrated optimization capability will become even more indispensable. For technical decision-makers, understanding the key links in this chain will help make more forward-looking judgments in AI infrastructure investment and construction.