Domestic GPU Completes Day-0 Adaptation of 3-Trillion-Parameter Model, Pioneering a New Path for Multimodal Inference

Background: Infrastructure Challenges of Scaling to 3 Trillion Parameters

As large language models scale toward extreme parameter counts, the emergence of 3-trillion-parameter models marks a new stage in AI development. Compared with conventional 10-billion- or 100-billion-parameter models, these models are not only orders of magnitude larger; they also adopt frontier techniques such as sparse Mixture-of-Experts (MoE), shifting inference-time computation from pure dense computing to a hybrid of sparse dispatch and dense computation. This shift imposes entirely new requirements on underlying AI acceleration hardware: enough memory bandwidth and capacity to hold massive parameters, plus flexible scheduling mechanisms to keep up with the dynamic routing of sparse experts.

Multimodal capabilities add another layer of complexity. The model must process text token sequences and visual feature vectors within a single inference framework, even though the two differ sharply in data format, compute characteristics, and memory access patterns. Efficiently supporting such heterogeneous workloads on one unified acceleration platform has become a key technical challenge for GPU vendors to solve.

Technical Breakthrough: Training-Inference Integrated Architecture and Full-Stack Software Coordination

Design Philosophy of the Training-Inference Integrated AI Card

For ultra-large model inference, the traditional GPU design approach of chasing peak compute alone is no longer sufficient. Leading solutions instead embrace a “training-inference integrated” architecture that optimizes for the characteristics of both workloads at the hardware level. Such AI accelerator cards are typically engineered along the following dimensions:

• High-bandwidth memory system: HBM-class high-bandwidth memory with large capacity ensures that 3 trillion parameters (stored at appropriate precision) can reside on a single card or a small group of cards, avoiding frequent model sharding and data movement overhead.
• Flexible compute unit scheduling: Support for fine-grained, operator-level concurrency allows different expert networks to be dispatched efficiently to different compute cores under expert parallelism, easing the communication bottlenecks of MoE inference.
• Heterogeneous compute fusion: Matrix computation units optimized for vision models (such as ViT architectures) are integrated in hardware, so image feature extraction and autoregressive text generation can switch seamlessly within the same execution flow.

These hardware features lay the physical foundation for “memory-resident” inference of ultra-large models: parameters no longer need to be fetched from external storage on every request, which dramatically reduces first-token latency.

The MUSA Software Stack: Closing the “Last Mile” Between Hardware and Frameworks

Whether hardware capabilities can be fully unleashed depends on the maturity of the software stack. A complete GPU software stack typically spans the driver layer, runtime libraries, the compilation toolchain, and upper-level compute libraries. In this case, the domestic GPU achieved model adaptation through its self-developed MUSA software stack, following a key path that includes:

1. Model format conversion and optimization: Open-source models are usually released in PyTorch format and must be converted into the target GPU’s intermediate representation (IR) by the compiler. This step involves operator fusion, precision calibration (such as FP16/BF16 conversion), and memory layout optimization.
2. Inference engine adaptation: Registering the underlying operators of mainstream inference frameworks with the MUSA runtime enables hardware acceleration for advanced features such as KV Cache management, sampling strategies, and dynamic batching.
3. Multimodal encoder-decoder coordination: Custom cross-module data channels are built for how the visual encoder interacts with the language decoder, reducing the number of host-device copies.

During Day-0 adaptation, the software team must analyze the model structure in advance and identify compute hotspots. For MoE models, the sparse invocation pattern of the experts’ feed-forward networks deserves particular attention; for multimodal models, the fused computation of image patch embedding and positional encoding needs to be optimized.

Core Adaptation Workflow: From Model Loading to a Fully Connected Inference Pipeline

Model Loading: Efficient Loading Strategies for 3 Trillion Parameters

Given the sheer volume of 3 trillion parameters, the loading stage must solve two problems: loading time and memory peaks. Common strategies include:

• Sharded pipeline loading: Weights are split into shards by expert or layer, read from storage in parallel by multiple threads, and injected into GPU memory in dependency order, avoiding a single peak that would overflow capacity.
• Memory mapping: Using the operating system’s virtual memory mechanism, model weight files on disk are mapped into the GPU’s virtual address space and paged in on demand, bypassing the full-copy overhead of a traditional load() call.
• Weight deduplication and sparse storage: For attention parameters shared across experts in a MoE model, deduplicated storage keeps a single physical copy with logical sharing implemented through index matrices, reducing both storage and loading burdens.

Service Startup: Building a Highly Available Inference Gateway

The core task during service startup is to quickly stand up an inference service ready to respond to user requests. This stage involves:

• GPU memory pre-allocation and KV Cache reservation: The KV Cache space required by concurrent requests is estimated up front and a memory pool is created in advance, avoiding the memory fragmentation and unstable latency caused by runtime dynamic allocation.
• Model replica initialization: To support high concurrency, multiple model replicas (instances) are launched with a load balancing policy configured. A 3-trillion-parameter model is typically deployed across multiple GPUs using a combination of tensor parallelism and expert parallelism; at startup, model state must be synchronized across cards to guarantee parameter consistency.
• Health checks and warm-up: Dummy inference requests are sent to trigger the model’s forward pass, completing dynamic compilation of all operators and kernel loading, so the first real request does not suffer excessive cold-start latency.

Text Inference Pipeline: Optimizing the Decoding Process

Text generation runs autoregressively: every time step performs attention computation, invokes expert feed-forward networks, and samples the next token. Specific optimizations for a 3-trillion-parameter MoE model include:

• Dynamic expert routing cache: In batched inference, similar requests often activate the same subset of experts. By maintaining an expert activation frequency table, hot experts can be kept resident in high-bandwidth cache regions, reducing memory access latency.
• Speculative sampling and parallel decoding: A small draft model predicts candidate tokens and the large model verifies them, so a single forward pass yields multiple tokens and decoding throughput improves significantly.
• Continuous batching: Unlike static request batching, continuous batching inserts new requests as soon as any token generation completes, maximizing GPU utilization and minimizing idle bubbles.

The following pseudocode illustrates the core expert dispatch logic in MoE inference:

# Pseudocode: expert parallelism dispatch
def moe_forward(hidden_states, router_logits, top_k=2):
    # Compute routing weights and select the top_k experts
    routing_weights = softmax(router_logits, dim=-1)
    top_k_weights, top_k_indices = topk(routing_weights, k=top_k)
    final_output = zeros_like(hidden_states)
    for expert_id in unique(top_k_indices.flatten()):
        # Fetch the corresponding expert network from the expert table
        expert_network = expert_table[expert_id]
        # Batch the tokens routed to this expert and compute them together
        expert_mask = (top_k_indices == expert_id)
        token_indices = nonzero(expert_mask)
        expert_input = hidden_states[token_indices]
        expert_output = expert_network(expert_input)
        final_output[token_indices] += top_k_weights[token_indices] * expert_output
    return final_output

Image Inference Pipeline: Visual Feature Extraction and Cross-Modal Fusion

For image inputs, the model first runs a visual encoder (such as ViT) to perform patch embedding and positional encoding, producing a sequence of visual features. That sequence is then concatenated with text token embeddings and fed into the Transformer decoder. In GPU adaptation, the priorities for the image inference pipeline are:

• GPU-accelerated image preprocessing: Operations such as resize, normalization, and data augmentation can be implemented as custom kernels (MUSA kernels within the MUSA stack), avoiding frequent CPU-GPU data transfers.
• Vision-language feature alignment: Because visual sequences are far longer than text sequences, attention matrix computation grows substantially. IO-aware attention optimizations such as FlashAttention reduce HBM accesses and improve computational efficiency.
• Mixed multimodal batching: Pure-text and image-text requests are processed within the same batch, with attention masks isolating requests of different modalities—preserving output correctness while raising hardware utilization.

Ecosystem Synergy and Outlook: AI Infrastructure Comes of Age

The success of this Day-0 adaptation demonstrates that domestic full-function GPUs can support the latest frontier models on par with mainstream international accelerator cards. The achievement rests on the coordinated evolution of three ecosystem elements:

1. An open model ecosystem: The open-source LLM community continuously releases high-quality pretrained weights, giving hardware adaptation clear targets. Hardware vendors can iterate quickly on public weights instead of waiting for proprietary model releases.
2. Software stack standardization: Building a self-developed software stack on widely adopted compilation frameworks (such as MLIR and TVM) sharply reduces model conversion costs. Developers can reuse extensive open-source operator libraries and concentrate on optimizing hardware-specific operators.
3. Developer community co-building: Opening the toolchain and providing debugging tools and profilers draws more developers into the GPU ecosystem, creating a positive hardware-software-application flywheel.

Looking ahead, as inference costs for 3-trillion-parameter models gradually fall, more real-world scenarios—intelligent customer service, document understanding, multimodal search—will move into production. Sustained investment in ultra-large model inference will give the domestic large model industry a solid, independently controlled compute foundation. The next round of technical competition is likely to center on lower-bit precision support (such as INT4/FP8), inference optimization for longer context windows, and elastic scaling of cross-node distributed inference.

For AI infrastructure developers, the takeaway from this case is clear: when evaluating an AI accelerator chip, look beyond peak compute figures. Examine the completeness of the software stack, the efficiency of the model adaptation toolchain, and the flexibility of multimodal extensions. Only by advancing hardware, software, and ecosystem together can the full potential of ultra-large models be unlocked.