Domestic Full-Feature GPU Completes Adaptation for 3-Trillion-Parameter Sparse Expert Model: Technical Path and Key Challenges

Background: New Compute Challenges from the Leap in Model Parameter Scale

As large language models continue to evolve toward multimodality and long-context capabilities, parameter counts have rapidly escalated from the hundreds-of-billions range to trillions, even exceeding 3 trillion. This trend imposes near-stringent requirements on underlying AI compute infrastructure: not only strong floating-point performance, but also comprehensive capability in memory capacity, memory bandwidth, cross-device communication, and software ecosystem maturity.

Recently, a sparse Mixture-of-Experts (MoE) model with 3 trillion parameters, support for a 1-million-token context window, and multimodal text-and-image reasoning capability was officially open-sourced. As the first model in its series, its architectural hallmark is a sparse activation mechanism—during inference, not all parameters are activated; instead, tokens are dynamically routed to a subset of expert networks based on their input. While this design substantially increases model capacity and knowledge density, it shifts engineering challenges to the system level: How can hundreds of gigabytes or even terabytes of weights be loaded and scheduled within limited memory? How can the dynamic memory access patterns introduced by expert routing be efficiently managed during inference? How can KV cache stability be guaranteed in long-context scenarios?

A domestic full-feature GPU, based on a training-inference unified compute card and a self-developed software stack, achieved Day-0 adaptation on the day the model was released, enabling the full pipeline from model loading and service startup to basic text and image inference. This result is not merely a “it runs” milestone, but a systematic validation of the domestic AI compute stack’s capability to support ultra-large-scale models.

1. Hardware Foundation: Architectural Advantages of the Full-Feature GPU

The primary prerequisite for adapting an ultra-large-scale MoE model is that the hardware possesses sufficient ability to break through the “memory wall.” Traditional GPUs are constrained by memory capacity; facing a 3-trillion-parameter model, they typically require multi-card pipeline parallelism or tensor parallelism, and cross-card communication overhead grows non-linearly with model scale.

The training-inference unified compute card used in this adaptation emphasizes the following core capabilities in its design:

1. Large-Capacity, High-Bandwidth Memory: Single-card memory capacity must satisfy the basic requirement of hosting expert weights for a single MoE layer. A 3-trillion-parameter model at FP16 precision theoretically requires approximately 6 TB for weight storage; even with INT8 quantization, roughly 3 TB is still needed. A single card cannot hold the full weight set, but through a reasonable expert parallelism strategy, different experts can be distributed across different compute nodes, so that each card only needs to host a subset of experts and the shared attention layer weights.

2. High-Throughput Data Path: Memory bandwidth determines the efficiency of streaming weight loading during inference. Sparse activation models must dynamically gather routed expert weights during each forward pass, imposing higher memory bandwidth requirements than dense models. This compute card reduces memory access latency during expert switching by optimizing the data movement path between on-chip SRAM and HBM.

3. Multimodal Compute Unit Fusion: Support for image input relies on coordination between the visual encoder and the language decoder. At the hardware level, compute requirements for both CNN-style visual feature extraction and Transformer-style sequence modeling must be met simultaneously. The full-feature GPU, through unified MUSA compute cores, balances efficiency for both dense matrix multiplication and convolution operations.

2. Software Stack Coordination: Full-Pipeline Support from MUSA

Unlocking hardware performance depends on deep software stack optimization. This Day-0 adaptation demonstrates the complete coordination capability of the MUSA software stack from low-level drivers to upper-level inference frameworks. Core efforts focused on the following four areas:

2.1 Model Loading and Memory Planning

Facing terabyte-scale weights, a naive one-shot loading approach is clearly infeasible. The adaptation team adopted a sharded loading + lazy initialization strategy:

• First, parse metadata such as the number of experts, per-layer expert dimensions, and attention head counts from the model configuration file;
• Based on the number of available devices and memory capacity, compute the expert parallelism partitioning scheme;
• Load only the expert shards required by the current node into memory, keeping remaining weights in host memory or a distributed file system, pulling them on demand when routing requests occur.

Memory planning must also account for KV cache pre-allocation. A 1-million-token context means that in extreme scenarios, the KV cache per layer may occupy several gigabytes of memory. The MUSA software stack provides a dynamic memory pool mechanism that allows KV cache and expert weights to share memory space and dynamically adjust the allocation ratio based on actual request lengths.

2.2 Runtime Optimization for Sparse Expert Routing

The core overhead in MoE model inference lies in expert routing. The MUSA runtime system designed a two-level scheduling mechanism for sparse activation:

• Level 1: The gating network outputs Top-K expert selections for each token, forming a routing mapping table;
• Level 2: The runtime dispatches intermediate representations of different tokens to the corresponding expert compute queues based on the mapping table.

To avoid GPU idling caused by uneven load across experts, the runtime introduces dynamic batching: tokens routed to the same expert across all requests are aggregated into a single batch for matrix multiplication, maximizing compute throughput. Empirical results show that with hundreds of experts and Top-K of 8–16, dynamic batching can improve GPU utilization by over 40%.

2.3 Memory Optimization for Long-Context Inference

A 1-million-token context window is a tremendous challenge for memory. With traditional MHA (multi-head attention), the KV cache size per layer grows linearly with sequence length, and the total KV cache for a 3-trillion-parameter model could exceed the weights themselves.

The MUSA software stack introduces a paged KV cache management mechanism, borrowing the paging concept from operating system virtual memory. KV cache is allocated in fixed-size pages, allowing physically non-contiguous memory blocks to form a logically contiguous sequence space through page table mapping. This design yields two direct benefits:

• Eliminates memory fragmentation and improves memory utilization;
• Supports dynamic scaling of sequence length without pre-allocating full memory for the maximum length.

2.4 Multimodal Input Processing Pipeline

Support for image input requires the inference service to possess visual encoding capability. During adaptation, the MUSA software stack seamlessly integrated the forward computation of the visual Transformer. Images are first split into fixed-size patches, linearly projected, and concatenated with text token embeddings before being fed into shared Transformer layers. In terms of memory allocation, visual features only need to reside during the first forward pass; subsequent layers share a unified sequence representation of text and vision.

3. Key Technical Challenges and Solutions for Inference Performance

After establishing the basic pipeline, further optimizing inference performance requires tackling the following core technical challenges:

3.1 Trade-offs Between Expert Parallelism and Communication Topology

Experts of an MoE model are distributed across multiple GPUs, and the forward computation of each token may involve cross-card expert access. Communication volume is strongly correlated with routing decisions, which are dynamic and cannot be pre-compiled.

An effective optimization path is hierarchical routing awareness: at a high level, semantically similar experts are placed on physically proximate GPUs as much as possible, reducing the probability of cross-rack communication. Additionally, using All-to-All communication primitives combined with RDMA networking can keep single expert access latency in the microsecond range.

3.2 Quantization Compression and Precision Preservation

To reduce memory bandwidth pressure, a mixed-precision strategy was applied to model weights during adaptation:

• QKV projection and O projection in attention layers retain FP16 precision, ensuring precision-sensitive parts of attention computation are unaffected;
• Feed-forward layer weights in expert networks can be compressed to INT8, leveraging hardware integer compute units for acceleration;
• MoE router weights remain FP32 to ensure routing decision accuracy.

Through this strategy, memory bandwidth requirements were reduced by approximately 30% while maintaining end-to-end task precision loss below 0.5%.

3.3 High Concurrency and Elastic Scaling for Inference Services

For production environments, inference services must support high-concurrency requests and elastic scaling. The MUSA software stack provides continuous batching capability: when a new request arrives, it is dynamically inserted into the currently executing batch without waiting for the current batch to fully complete. For MoE models, continuous batching must coordinate with dynamic expert scheduling—each token still routes independently, but shares the same batch’s GPU compute resources.

4. Engineering Significance of Day-0 Adaptation

Day-0 adaptation means completing the full pipeline on the same day the model is open-sourced. This capability reflects the following engineering maturity:

• Automated model conversion toolchain: Supports automated conversion from general model formats to MUSA runtime formats without manually writing operators;
• Operator library coverage: Mainstream large model operators (such as FlashAttention and MoE-related sparse operators) are pre-built and optimized in the operator library, eliminating the need for ad-hoc development;
• Distributed runtime stability: In thousand-card cluster environments, the stability of hybrid model-parallel and data-parallel deployment has passed large-scale stress testing.

This “support at launch” capability significantly shortens the cycle from model release to business deployment, enabling enterprise users to deploy the latest frontier models on domestic compute without waiting for delayed chip vendor adaptation.

5. Summary and Outlook

From this Day-0 adaptation of a 3-trillion-parameter MoE model, it is evident that domestic full-feature GPUs have established a foundation for engaging with mainstream ecosystems in ultra-large-scale model support. End-to-end capability validation covering model loading, memory management, sparse routing, and multimodal inference demonstrates the engineering depth of the MUSA software stack and the soundness of the hardware design.

Looking ahead, the following directions merit continued investment:

1. Larger context windows: Moving from 1 million tokens toward 2 million or higher requires architectural breakthroughs in sparse attention mechanisms and KV cache compression;
2. Online incremental learning: Experts in MoE models can be incrementally added or updated; how to enable hot-swapping of experts without interrupting service;
3. Cross-cluster distributed inference: Optimization of network topology and scheduling strategies for terabyte-scale models across geographically distributed inference.

The evolution of large model technology will not stop at 3 trillion parameters. Each generational upgrade of compute infrastructure must be predicated on a synchronized leap in the software ecosystem. This adaptation validated not just a single model or a single GPU, but a complete technical path from chip to cluster, from framework to application—this is precisely the hallmark of domestic AI compute maturing.