Day-0 Adaptation of a Trillion-Parameter LLM and Sparse Attention Optimization

As large language models surpass three trillion parameters and increasingly combine ultra-long context with multimodal capabilities, the underlying hardware and its supporting software stack face unprecedented adaptation challenges. Recently, a three-trillion-parameter model was successfully adapted on a self-developed AI accelerator and full-stack software ecosystem, achieving Day-0 readiness by completing model loading, service startup, and both text and multimodal inference. This effort validates the ability of heterogeneous domestic computing infrastructure to support extremely large and complex models, while also providing valuable engineering practices for low-level operator optimization, memory scheduling, and inference engine restructuring. This article examines the core technical path behind this adaptation.

1. Adaptation challenges posed by the evolution of massive model architectures

Current frontier large language models show three major architectural trends that directly reshape the development logic of the underlying computing stack:

1. Ultra-long context and sparsification: Models natively support context windows at the million-token scale. To mitigate the exponential growth in memory and computation caused by long sequences, models commonly introduce sparse attention mechanisms such as DSA. This requires the underlying software stack to move beyond the standard dense attention paradigm and support dynamic indexing and sparse memory access.

2. Extremely large MoE architectures: Model size reaches the three-trillion-parameter level and uses mixture-of-experts routing. Although only a limited number of parameters are activated per forward pass, the total weight footprint is enormous, placing extreme demands on memory capacity, memory bandwidth, and distributed tensor parallelism.

3. Low-bit quantization and mixed-precision computation: To reduce memory usage and improve communication efficiency, models store expert weights in extremely low-bit floating-point formats such as MXFP4. However, hardware compute units often do not natively support direct matrix operations on these formats, so the software stack must perform careful format conversion.

Given these challenges, achieving Day-0 adaptation, meaning the environment is ready and inference is functional on the day a model is released, is not a simple API integration effort. It requires deep engineering changes across the compiler, operator library, and inference engine.

2. Full-stack software ecosystem: the foundation for bridging heterogeneous computing

To support the rapid iteration of upstream community model code, a highly compatible and controllable software ecosystem is essential. In this adaptation effort, the team used a self-developed unified system architecture, the MUSA software stack, to build a fast path from high-level models to low-level hardware.

Specifically, the team used an open-source inference framework adapted to the self-developed chip, SGLang-MUSA, to host the upstream model implementation. Four key components form the backbone of this software stack:

• Core operator library (MATE): Provides highly optimized low-level computational primitives and serves as the physical foundation for model acceleration.
• Triton and TileLang adaptation layers: Triton-MUSA and TileLang-MUSA provide robust backend support for high-level operator development, enabling complex operators to be reused across platforms and significantly lowering the development barrier.
• Specialized compiler (MTCC): Translates high-level computation graphs and operator instructions into efficient machine code executable by the hardware, enabling the migration and integration of critical compute paths on the new architecture.

Through this closed-loop toolchain spanning framework, compiler, and operators, community model code originally written for general-purpose ecosystems can be efficiently transformed into high-performance inference services on the accelerator.

3. Core technical challenge: deep adaptation of sparse attention

For the sparse attention mechanism used in this massive model, adaptation was not a one-step process. The MUSA software stack required detailed reconstruction of index selection, memory management, and compute paths to accommodate DSA.

3.1 Index selection and KV cache layout optimization

In standard attention, Query-Key-Value computation follows a dense global mapping. Under DSA, the system dynamically selects a subset of KV entries according to a specific policy. The team implemented dedicated sparse attention operators in the MATE operator library with two core design goals:

• Dynamic index computation: Before kernel launch, the sparse index matrix is computed and passed quickly to avoid global traversal of irrelevant data.
• Non-contiguous KV cache access restructuring: Because sparse selection means KV reads are no longer contiguous memory blocks, low-level operators must optimize coalesced access to non-contiguous memory. By rearranging the physical layout of the KV cache, memory fragmentation is reduced, ensuring efficient data access during both prefill and decode stages.

3.2 Building a fused execution pipeline

Operator-level optimization alone cannot guarantee end-to-end performance. The MUSA software stack deeply integrates the DSA operators with the model’s other critical execution components:

• KDA state management: The intermediate state of sparse attention is combined with the low-level scheduler to maintain state consistency during long-context inference.
• Cross-layer residual and vision attention fusion: Large models often use multimodal architectures where complex residual and attention interactions exist between the vision encoder and the language model. By compiling DSA, cross-layer residual propagation, and vision attention operators into a unified execution pipeline, framework scheduling overhead and kernel launch gaps are reduced, enabling seamless end-to-end computation.

4. Dynamic conversion of MXFP4 expert weights and zero memory waste

Given the enormous memory pressure of a three-trillion-parameter MoE model, storing expert weights in MXFP4, a 4-bit floating-point format, is a common industry approach. However, executing 4-bit floating-point matrix multiplication directly on hardware presents many difficulties, making efficient weight processing a key factor for practical inference.

4.1 Pain points of traditional offline conversion

Traditional approaches usually rely on offline quantization conversion, where all MXFP4 weights are converted in advance to hardware-supported FP8 or FP16 formats and saved. For a three-trillion-parameter model, however, full conversion to FP8 would require several additional terabytes of storage. This is not only time-consuming but also substantially increases deployment costs.

4.2 Online conversion and cross-layer reuse mechanism

To address this problem, the team adopted an execution strategy based on online conversion and cross-layer reuse of an FP8 workspace:

• On-demand online conversion: During forward inference, only the currently activated expert weights are converted from MXFP4 to FP8 in real time.
• Cross-layer workspace reuse: In MoE models, only a very small subset of experts is active at any given time, and computation across different layers is serial. The system pre-allocates a fixed-size FP8 workspace in memory. After one layer finishes computing, the workspace is directly reused by the next layer’s activated experts for writing newly converted weights.

This mechanism has significant engineering value. First, it eliminates the cumbersome process of generating a complete offline FP8 weight set and the associated massive storage cost, enabling rapid Day-0 deployment. Second, by reusing a fixed workspace, the additional memory footprint is kept extremely small, ensuring stable operation of very large models within limited memory resources.

5. Summary and industry implications

The Day-0 adaptation of a three-trillion-parameter model shows that heterogeneous domestic computing infrastructure cannot compete in the era of large language models through hardware performance alone. A full-stack software-hardware co-design capability is essential.

Through deep coordination among a self-developed compiler, customized operator library, and high-level language backends, the underlying platform can quickly absorb and support the most advanced open-source model architectures. In particular, the engineering approaches demonstrated in handling sparse attention and extremely low-bit MoE weights, including dynamic index reconstruction and cross-layer memory reuse, not only address current computational bottlenecks but also provide reusable architectural patterns for adapting even larger models with longer contexts in the future. As the software-hardware ecosystem matures, building an efficient, autonomous, and compatible AI computing foundation will become a core engine for the continued development of the AI industry.