Day-0 Adaptation of a Million-Token Context LLM on Chinese AI Chips: Engineering Practices and Insights

From “It Runs” to “It Runs Well”: The Day-0 Adaptation Challenge

In the race to bring large models into real-world applications, hardware-software co-optimization often decides whether a newly released model can be deployed efficiently on the day its weights ship. “Day-0 adaptation” means that as soon as new model weights become available, the inference system can complete model loading, service startup, and a fully connected basic inference path. It is not merely the starting point for performance tuning—it is a comprehensive stress test of the software stack’s completeness.

For a sparse MoE model with a million-token context window and trillions of parameters, the difficulty goes far beyond “fitting the model into memory.” The architecture introduces Dynamic Sparse Attention (DSA), cross-layer residual connections, joint inference with a vision encoder, and expert weights compressed in MXFP4 format. While these designs raise the model’s capabilities, they place demands on the underlying compute framework—in indexing logic, caching strategy, operator implementation, and numerical precision management—well beyond those of a conventional Transformer.

The Inference Engine Layer: Bridging Upstream Model Implementations

The first phase of adaptation centers on smoothly porting the community’s open-source model implementation onto the target hardware platform. Because mainstream inference engines—such as the Python-based SGLang framework—default to the CUDA ecosystem as their backend, the first step is to establish a compatible and efficient execution environment.

In practice, the team built an adaptation layer based on SGLang-MUSA to carry the upstream model code. This is far more than API translation: every operator in the model’s execution graph must be mapped and verified individually. The MUSA software stack provides a multi-layered toolchain for this work, consisting of the MATE operator library, the Triton-MUSA domain-specific language, and the MTCC compiler:

• MATE operator library: Delivers deeply optimized implementations of common operators—matrix multiplication, normalization, and various activation functions—ensuring efficiency along the fundamental compute paths.
• Triton-MUSA and TileLang-MUSA: Provide Python-like high-level programming interfaces, allowing developers to rapidly build and tune native implementations of memory-bound or compute-bound custom operators.
• MTCC compiler: Compiles high-level descriptions or standard C++ code into machine code optimized for the target architecture, forming the low-level foundation for final performance.

With this combined approach, model code originally written for a specific GPU ecosystem was transformed within days into inference service instances running on Chinese accelerator hardware, clearing the first hurdle of model loading and service startup.

Sparse Attention Adaptation: Balancing Indexing, Caching, and Compute Paths

Of all the technical challenges, adapting Dynamic Sparse Attention (DSA) is among the most demanding. Unlike traditional global attention, DSA dynamically selects the key-value (KV) pairs most relevant to the current query during inference, substantially cutting computation and memory usage in long-context scenarios.

The catch is that this sparsity is highly hardware-unfriendly. Compute units cannot follow a fixed matrix-multiplication pattern; instead, they must rely on an irregular, constantly changing index matrix to decide which data to read and what range to compute. To overcome this, the MUSA software stack pursued a three-pronged adaptation strategy:

1. Rebuilding the index selection pipeline: Sparse indices generated by the model are converted from global-memory indices into a local format suited for fast on-chip access, and the index sorting algorithm is optimized so that downstream operators can read data with well-coalesced access patterns.
2. KV cache layout optimization: Under dynamic sparsity, traditional contiguous memory allocation causes severe fragmentation. The adaptation adopted a block-based, non-contiguous cache management scheme, paired with a KDA state manager that allocates and reclaims cache blocks on demand, effectively improving memory utilization at million-token context lengths.
3. Separating Prefill and Decode paths: During Prefill, a high-throughput sparse matrix-multiplication implementation drives parallel computation; during Decode, the system switches to a low-latency sparse GEMV (matrix-vector multiplication) implementation. Optimizing the two paths independently ensures that both time-to-first-token and steady-state token generation speed reach strong levels.

In addition, the core sparse attention computation was fully integrated into the MATE operator library and chained with cross-layer residual connections and the vision attention module into a single execution path, avoiding the overhead of repeatedly launching kernels from the Python layer.

Online Conversion of Low-Bit Weights: The MXFP4 Expert Compute Strategy

Model compression is now a mainstream path toward lighter large models, and this model stores its expert network weights in MXFP4 (Microscaling FP4) format. FP4 can shrink weight storage to half its original size or less, but existing accelerator hardware often lacks native, high-throughput compute support for the FP4 data format.

Confronted with this hardware-algorithm gap, the team did not take the common route of “convert everything offline, then load for inference.” For a model with trillions of parameters, producing a complete offline FP8 (8-bit floating-point) copy of the weights would take hours and consume several additional terabytes of disk and memory. To sidestep this bottleneck, the engineering team designed an efficient pipeline that overlaps online conversion with computation.

During actual inference, the system loads only the native MXFP4 weights into memory. Before each computation, a lightweight decompression kernel converts the weights needed for the current batch from FP4 into BF16 or FP16 (or, through a reused workspace, into FP8) for the actual matrix operations. To reduce the overhead of frequent decompression calls, the system applies a cross-layer FP8 workspace reuse strategy: once an expert layer finishes computing, the FP8 workspace buffer it releases is immediately reused by the next layer. This approach hides conversion latency in the time dimension by overlapping decompression with computation, while in the space dimension it significantly lowers peak memory usage.

Conclusion and Outlook: Software Stack Depth Defines the Hardware Ceiling

These practices show that a Chinese AI chip achieving Day-0 adaptation of a frontier large model is, at its core, a systematic campaign fought across three layers of the software stack: framework, compiler, and operators. The MXFP4 handling strategy demonstrates the value of pragmatic engineering in overcoming resource bottlenecks, while the end-to-end adaptation of sparse attention confirms how decisive a mature software stack is for bringing new-architecture models into production quickly.

For developer communities and AI infrastructure teams, the methodology behind this effort is broadly applicable: as models iterate faster than ever, only by building coordinated capabilities across inference engines, operator implementations, cache management, and compiler optimization can heterogeneous hardware run diverse model types efficiently—truly bridging the gap from “usable” to “genuinely good to use.” Looking ahead, as unified programming models and AI compiler technology continue to advance, the cycle of adapting hardware to new algorithms is expected to shrink further, laying a more solid compute foundation for the next wave of applications built on top.