Day-0 Adaptation of a Domestic AI Large Model: An Efficient Deployment Path from Hundreds of Billions to Trillions of Parameters

Background: Inference Challenges in the Trillion-Parameter Era

As large language model parameter counts climb from hundreds of billions to trillions, model architectures are undergoing profound shifts—sparse Mixture-of-Experts (MoE) has become the mainstream choice. But the surge in parameter scale brings three challenges to inference systems:

• VRAM capacity bottleneck: Even at BF16 precision, the weights of a 3-trillion-parameter model require roughly 60 TB of VRAM, far beyond the physical limits of a single GPU or even a single node. Quantization compression and distributed parallelism must therefore be pursued in tandem.
• Heterogeneous compute patterns: MoE introduces sparse routing, so the model contains both dense Attention computation and sparse Expert computation, with different compute paths placing different demands on hardware utilization.
• New operator adaptation: Each architecture update may bring new operators such as sparse attention (DSA) and mixed-precision weights (MXFP4). Getting a new model running on existing hardware in the shortest possible time is key to determining product launch speed.

Facing these challenges, a domestic GPU acceleration platform based on its self-developed AI training-and-inference integrated compute card and unified software stack completed adaptation on the very day the latest open-source model was released. This article reviews that process and breaks down its technical path.

1. Software and Hardware Foundation: The Basis for Integrated Training and Inference

The hardware platform used in this adaptation carries a self-developed AI inference-and-training integrated compute card, deeply customized and optimized for the matrix operations of Transformer architectures, and equipped with high-bandwidth memory. But hardware performance is only the foundation—the real adaptation efficiency depends on the maturity of the software ecosystem.

The MUSA software stack on top of the platform is the core of the entire ecosystem, covering from bottom to top:

1. Low-level drivers and runtime: responsible for device management, memory management, and command dispatch;
2. Programming model and compiler: provides a CUDA-like parallel programming interface and uses the MTCC compiler to translate high-level languages into optimized device code;
3. Operator libraries and acceleration tools: provides the MATX high-performance operator library covering common neural network layers, and supports custom operators written in DSLs such as Triton and TileLang;
4. AI framework adaptation layer: ensures that models from mainstream frameworks such as PyTorch can switch seamlessly to this hardware platform.

This layered software design means new model adaptation does not need to start from scratch. Development teams can prioritize reusing mature components of the existing ecosystem and focus targeted development only on newly added compute paths.

2. Migration and Adaptation of Key Inference Paths

The open-source model in this practice includes a 1-million-token context window. In long-sequence scenarios, its Attention compute complexity grows linearly with sequence length, thanks to the sparse attention mechanism (DSA) the model adopts. Meanwhile, the Expert layer weights use MXFP4 format storage, further compressing static VRAM usage. Below are the three steps of adaptation work we carried out around the model’s architectural characteristics.

1. Engineering Migration Based on the SGLang Framework

The first step was to build the inference serving framework. Because the software stack maintains good compatibility with the open-source inference framework SGLang, the development team quickly took over the upstream model’s original implementation through the SGLang-MUSA interface, connecting the complete basic链路 from model weight loading to service startup.

During this process, the team focused on several engineering issues:

• Tensor parallelism strategy adaptation: Trillion-scale models must use multi-card distributed inference, splitting weights of different layers or within the same layer across multiple compute cards via tensor parallelism. The software stack’s multi-card communication library provides NCCL-like AllReduce and AllGather capabilities, keeping communication overhead controllable after partitioning.
• Dynamic shape handling: In long-context inference, input and output sequence lengths change dynamically, requiring the inference framework to support dynamic-shape kernel scheduling to avoid VRAM waste or recompilation overhead caused by static shape constraints.

2. On-Board Integration of Sparse Attention and Vision Compute Operators

The most challenging part of the model architecture is sparse attention DSA. Unlike traditional dense attention, DSA focuses only on a selected subset of important tokens in the sequence, reducing compute at long sequence lengths. Its implementation difficulties include:

• Complex index selection logic: similarity scores between Query and Key must be used to dynamically determine which token interactions need computation;
• Non-contiguous KV cache layout: In sparse mode, the KV Cache is no longer stored in sequence order but in a compact format based on routing indices, placing high demands on memory access efficiency.

The team implemented a sparse attention kernel in the MATE operator library and coordinated KV cache reads and writes through the KDA state manager. In addition, for the vision encoder introduced in the model, its internal Vision Attention computation was integrated into the overall inference path, ensuring end-to-end inference stability for image-text multimodal input scenarios.

Notably, the team cleverly reused the basic operators for cache management and cross-layer residual connections rather than designing a full set of custom kernels for each new structure, greatly shortening the development cycle.

3. Online Quantization Execution of MXFP4 Expert Weights

For the MXFP4 format used by the Expert layers (a mixed-precision storage format combining scaling factors with 4-bit mantissas), the team adopted an online conversion and cross-layer reuse strategy rather than traditional offline full conversion:

1. Online conversion: Model weights are dequantized in real time to FP8 or BF16 as they are loaded from VRAM to the compute unit. This eliminates the need to pre-prepare a complete FP8 weight copy and directly saves about 30% of VRAM usage.
2. Workspace reuse: Because different Expert layers share the same physical compute core, the platform reuses a fixed-size FP8 workspace across layers, effectively avoiding fragmentation overhead from frequent VRAM allocation and release.

This strategy is significant for deploying ultra-large-scale models: even when physical VRAM cannot hold all dequantized weights, the system can still complete inference through on-demand computation, offering a viable path for deploying even larger models.

3. Methodology Takeaways: From “Day-0” to “Continuous Evolution”

Completing Day-0 adaptation is only the first step; what truly tests a platform’s capabilities is the performance optimization and feature iteration that follow. We distilled three core lessons:

• Ecosystem compatibility determines adaptation speed: The platform’s MUSA software stack is highly compatible with the mainstream CUDA ecosystem at the programming model and runtime levels, allowing a large amount of model code from the open-source community to compile and run directly. In this adaptation, about 80% of the kernel code ran without modification, greatly improving migration efficiency.
• Heterogeneous compute paths require fine-grained management: Architectural complexity often manifests in the diversity of compute patterns—sparse and dense coexisting, long and short sequences alternating, attention and experts in parallel. A good software stack should provide a unified view of memory management and task scheduling so that upper-layer developers need not be aware of low-level hardware scheduling details.
• Auto-tuning should be a routine action: For the fuzzy-call expert routing scenario, the team used Triton and TileLang operator DSLs to write rapid validation prototypes and relied on the compiler’s auto-tuning mechanism to find the best configurations across different shapes and batches, ensuring decent performance even in long-tail scenarios.

4. Conclusion and Outlook

This Day-0 adaptation practice demonstrates the important value of a mature hardware-software co-design platform for industrial deployment of large models. When a model is open-sourced and released, the platform can already provide a complete inference solution in sync—not through sheer hardware stacking, but through a well-designed, clearly layered software ecosystem.

For engineering teams facing large-model deployment challenges, our recommendation is: prioritize building a highly compatible software stack, focus core effort on the model’s distinctive compute paths, and introduce flexible quantization and mixed-precision strategies to relieve VRAM pressure. Only deep hardware-software coordination can meet each iteration of ultra-large models with lower cost and greater speed.