Domestic GPU Completes Day-0 Adaptation for a Trillion-Parameter-Scale MoE Model, Unblocking the Full AI Inference Path

Background: From Open Model Release to the Last Mile of Compute Adaptation

As large model parameter counts exceed the trillion level, model architecture complexity and inference-stage resource overhead grow exponentially. For AI infrastructure teams, a pressing engineering challenge is how to adapt an existing GPU compute cluster to a newly released ultra-large open-source model in the shortest possible time. This process is often called “Day-0 adaptation.” The goal is to have usable inference and training service capabilities on the same day model weights are officially published. It tests both chip hardware design and the maturity of the software stack.

Recently, an industry sparse large model with roughly 3 trillion parameters was released as open source. It uses a Mixture-of-Experts (MoE) architecture and reduces actual computation substantially through sparse activation. However, the memory footprint, communication overhead, and scheduling complexity introduced by ultra-large-scale parameters still place extremely demanding requirements on the underlying compute platform. Against this background, a domestic GPU vendor completed Day-0 adaptation of the model using its self-developed training-inference compute card and accompanying software stack, unblocking a complete business path from model loading and service startup to basic text and image inference. This article uses that work as a starting point to systematically examine the key technical steps, engineering challenges, and general solutions involved in adapting ultra-large-scale models.

Hardware Foundation: Design Logic of a Training-Inference Compute Card

For models with trillions of parameters, the traditional adaptation approach centered on per-card memory capacity is no longer sufficient. Mainstream solutions now rely on the coordination of multi-card parallelism, memory pooling, and high-speed interconnect. The training-inference compute card used by the domestic GPU in this adaptation reflects several specific design considerations for large-model scenarios:

1. Large memory capacity and high bandwidth: A single card needs to hold as many model shards as possible to reduce cross-card communication frequency. For expert-parallel strategies in MoE models, memory bandwidth directly determines memory access efficiency during token generation. High-speed interconnect buses form a unified memory address space across cards, allowing a single logical device to access hundreds of gigabytes of memory in total. This provides the physical foundation for holding trillion-scale parameters.
2. Flexible compute unit scheduling: In a 3-trillion-parameter sparse model, only part of the expert network is activated during a single forward pass. Hardware must support fine-grained task scheduling to avoid idle compute caused by uneven expert load. The compute card optimizes compute core scheduling for sparse activation patterns so that compute resources dynamically follow expert routing results.
3. Compute-communication coordination: The natural All-to-All communication pattern of MoE models places extremely high demands on inter-card interconnect bandwidth. Hardware needs an independent communication engine that can process cross-card data exchange in parallel with tensor computation, thereby hiding communication latency.

Software Stack Core: An Adaptation Path Centered on the MUSA Architecture

Complete hardware capability can only be converted into real productivity through a mature software stack. The MUSA software stack used in this adaptation provides a unified programming model and runtime environment, allowing developers to migrate models without needing to perceive underlying hardware differences.

In practice, MUSA adaptation can be divided into three layers:

1. Framework integration: Mainstream deep learning frameworks such as PyTorch connect to the MUSA runtime through backend extension mechanisms. Specifically, a custom device backend plugin translates tensor operations, operator scheduling, and memory management requests from the framework into MUSA instruction streams. The key at this layer is operator coverage: whether commonly used framework operators have efficient MUSA implementations, or whether an automatic fallback mechanism can be triggered when they are missing.
2. Operator-level optimization: For special operators in large models, such as the TopK gating in MoE routing and FlashAttention variants in attention mechanisms, the software stack needs highly hand-optimized kernel implementations. In this adaptation, operator-level tuning focused on various computation patterns ranging from 7B to 3 trillion parameters, balancing compute-intensive matrix multiplications against memory-bound normalization operations.
3. Graph compilation and runtime: Graph capture converts a dynamic model graph into a static execution graph for global optimizations such as operator fusion and memory reuse. The runtime system handles device management, stream scheduling, memory pooling, and related functions. During service startup, strategies such as parallel loading of model shards and deferred initialization of CUDA contexts, with zero-copy memory allocated in advance on the CPU side, significantly shorten the window from process launch to service readiness.

Key Engineering Steps: Model Loading, Service Startup, and Inference Path Validation

Model Loading: Parallel Sharding and Memory Fragmentation Control

Loading the weight files of a 3-trillion-parameter model is itself an I/O-intensive engineering challenge. Stored in FP16 precision, the model weights total about 6 TB. In practice, the following engineering strategies were adopted:

• Multi-threaded pipeline loading: Weight files are split into multiple shards along tensor dimensions. Different CPU threads read the shards in parallel and write them directly to GPU memory through PCIe/DMA channels, avoiding redundant copies in host memory.
• Memory fragmentation management: Frequent memory allocation and release during large-model loading can easily cause fragmentation. By calculating tensor lifecycles across layers before loading and using a pooling allocator to manage memory blocks uniformly, the process ensures that enough reserved space remains for KV Cache during inference after the full model is loaded.

Service Startup: Readiness Process of a Distributed Inference Engine

After model loading, service startup involves initializing the distributed inference engine. The core metric at this stage is the waiting time before “first-token latency.” Optimization measures include:

• Lazy initialization strategy: Defer non-critical-path operator compilation and memory pre-allocation until the first inference request arrives. Warm-up inference requests trigger rapid initialization of the critical path.
• Pre-built distributed communication groups: In multi-card and multi-node environments, establish NCCL-style communication groups and complete topology discovery in advance to avoid latency jitter from dynamic link creation during inference.
• Version compatibility checks: At startup, validate consistency among model file format versions, operator versions, and MUSA runtime versions to prevent runtime crashes caused by version mismatch.

Inference Path Validation: Text and Image Bimodal Testing

Unblocking the path is not the endpoint. Verifying the correctness of basic inference capability is the core goal. The adaptation validation phase designed a complete test matrix:

• Text generation path: Cover input length gradients from short text of a few dozen tokens to long context of thousands of tokens. Validate correctness in content generation quality, repetition penalty control, and stopping condition logic. Monitor peak memory usage and throughput changes across different batch sizes to establish a performance baseline curve.
• Image understanding path: For multimodal image-text inputs, validate tensor dimension matching between the visual encoder and language decoder, numerical precision of the modality alignment projection layer, and correctness of cross-modal attention mask generation.
• Long-duration stability testing: Run sustained constant-pressure inference for more than 48 hours while monitoring memory leaks, communication link degradation, and operator numerical drift.

Technical Implications: Adaptation Methodology and Future Directions

Several broadly applicable engineering lessons can be drawn from this successful adaptation of a 3-trillion-parameter model:

1. Hardware-software co-design determines adaptation efficiency. Memory pooling and communication engine capabilities reserved in hardware can only be converted into actual performance through fine-grained software stack scheduling. The “fully managed runtime” model embodied by the MUSA stack significantly reduces the adaptation burden on algorithm engineers.
2. Sparsity is a practical path for deploying ultra-large models. Although the MoE architecture increases system complexity, its characteristic that activated parameters are far fewer than total parameters allows per-card compute to focus on critical computation paths. Future hardware and operator library support for finer-grained sparsity, such as token-level pruning and dynamic depth, will become a competitive differentiator.
3. Ecosystem compatibility matters more than any single performance metric. The prerequisite for quickly adapting new models is standardized and stable framework-level interfaces. Only by maintaining synchronized evolution with mainstream deep learning frameworks can each model innovation be extended to new compute resources through the shortest path.

Conclusion

Moore Threads completed Day-0 adaptation of a 3-trillion-parameter large model using its training-inference compute card and MUSA software stack. This result shows that domestic AI compute infrastructure now has complete engineering capability to handle frontier model challenges. From hardware scheduling to software stack optimization, and from model loading to multimodal inference path validation, every breakthrough is built on solid systems engineering practice. For developers and enterprise users, this case provides a reusable reference pattern: when facing trillion-parameter-scale large models, compute adaptation is not one-dimensional hardware accumulation. It is a comprehensive systems engineering effort in which hardware architecture, software runtime, distributed strategies, and operational systems are tightly coupled. As model parameter counts move toward the ten-trillion level, coordinated innovation between algorithms and hardware will become even more deeply intertwined, and a compute foundation with full-stack self-development capability will play an increasingly critical role in this technological race.