Domestic GPU Completes Day-0 Adaptation for a 3-Trillion-Parameter LLM: From Architecture Analysis to Inference Deployment

From Model Release to Inference Service: The Engineering Challenge of Day-0 Adaptation

When the open-source community releases a large language model with roughly 3 trillion parameters, the real challenge for AI infrastructure teams is not the model itself. It is completing the full adaptation from model weights to a production-ready inference path in the shortest possible time. Day-0 adaptation means that on the day model weights are publicly released—or within a very short window—the team can load the model, start the serving stack, and bring up basic inference capability. This tests how well the GPU hardware, software stack, and model architecture can work together.

Recently, a domestic GPU vendor announced that it had completed Day-0 adaptation for a 3-trillion-parameter open-source model and enabled support for two basic inference tasks: text and image inputs. The technical significance of this milestone is that it demonstrates the ability of a domestic AI computing stack to quickly keep pace with the most advanced large-model architectures. This article starts from the architectural characteristics of the target model, breaks down the key technical paths involved in the adaptation, and discusses general methods for deploying extremely large models in inference scenarios.

Core Architectural Characteristics of the Target Model

The model adapted in this effort is the first product in its series. Its notable features include:

Parameter scale: The model has roughly 3 trillion parameters in total, placing it in the top tier of currently available open-source models. At this scale, the weight files alone reach multiple terabytes, placing extreme demands on storage bandwidth, memory capacity, and compute-unit scheduling efficiency.

Context window: The model supports a context window of up to 1 million tokens. This causes KV cache usage during inference to grow linearly with sequence length, putting enormous pressure on memory capacity and memory bandwidth.

Hybrid architecture: The model uses sparse attention, specifically Dynamic Sparse Attention (DSA), in its attention mechanism, and employs MXFP4-format quantized weights in its expert modules. These two architectural choices target the computational waste of long-sequence inference and the memory bottleneck of extremely large models, but they also add complexity for inference framework adaptation.

Capability focus: After large-scale post-training and reinforcement learning, the model shows significant improvements in code generation, 3D creation, financial research, knowledge-based Q&A, and long-horizon agent tasks. Its results on code and agent benchmarks have reached or approached the level of top closed-source models.

Software Stack Compatibility: The First Gate to Rapid Adaptation

For any GPU hardware platform, the compatibility of the software stack is the decisive factor in getting a large model designed for the CUDA ecosystem running quickly. In this adaptation, the GPU vendor’s MUSA software stack served as the bridge. Its core design goal is high compatibility with mainstream AI programming models.

At the programming model level, MUSA provides interface mappings that are highly compatible with CUDA, so that operator code written for CUDA can be migrated to the MUSA environment at relatively low cost. At the runtime interface level, MUSA implements compatibility with the backend interfaces of mainstream inference frameworks. At the AI framework level, it directly adopts upstream model implementations. This means the team could load the original model code directly based on SGLang-MUSA, an inference serving framework branch optimized for this GPU, without large-scale framework rewrites.

On top of this foundation, the team coordinated work across multiple layers, including the MATE operator library, Triton-MUSA, TileLang-MUSA, and the MTCC compiler, to migrate and integrate the key computational paths of the new architecture. The essence of this process is converting the low-level compute logic written for NVIDIA GPUs in the community model into instruction sequences that can execute efficiently on the domestic GPU. Notably, this migration reused mature interface designs, allowing the adaptation work to focus on optimizing newly introduced computational paths rather than repeatedly solving infrastructure problems.

Full Computational Path Adaptation for Dynamic Sparse Attention

The Dynamic Sparse Attention mechanism used by the target model is one of the main technical challenges in this adaptation. Unlike conventional full attention, sparse attention uses preset index patterns to compute attention scores only between selected token pairs, reducing the computational complexity of long-sequence inference from O(n²) to approximately linear. However, this optimization also introduces implementation complexity:

1. Index selection and planning: The model needs to dynamically select which token positions require attention computation based on the input sequence. This process involves complex index generation logic.

2. KV cache layout adjustment: Because only the keys and values at selected positions need to be cached and used in subsequent computations, the KV cache storage layout must change from a dense format to a sparse-indexed format.

3. Prefill and decode path separation: During the prefill phase, the model needs to process attention computation for all input tokens in parallel. During the decode phase, it only processes attention between newly generated tokens and the existing cache. These two computation modes have very different compute and memory-access characteristics and require separately designed computational paths.

In the adaptation for this GPU, the team addressed all three layers through the MUSA software stack. At the operator level, they integrated sparse attention operators specifically implemented in the MATE library. At the overall execution level, they orchestrated sparse attention computation together with KV cache state management (KDA), cross-layer residual connections, and the visual attention module to ensure smooth forward computation across the entire model.

Online Conversion Strategy for MXFP4 Expert Weights

Another notable technical detail is the model’s use of MXFP4-format expert weights. MXFP4 is a microscaled FP4 format that quantizes floating-point values into 4-bit representations combined with fine-grained scaling factors, compressing model size as aggressively as possible while preserving accuracy. For a 3-trillion-parameter model, using FP8 or even FP4 formats is the only viable approach to fit it into a reasonably sized GPU cluster.

However, the GPU’s compute cores may not natively support operator execution in MXFP4 format. To address this, the team adopted an online conversion plus cross-layer reuse strategy:

• Online conversion: During model loading, MXFP4 weights are converted to FP8 format in real time rather than performing a complete offline conversion ahead of time. The advantage of this strategy is a significantly shorter preparation window before the model can go live. A full offline conversion would require processing and storing an entire additional set of FP8 weight files—at the 3-trillion-parameter scale, this means at least several terabytes of extra storage.
• Cross-layer FP8 workspace reuse: Because adjacent layers in the model often share the same scaling factors or quantization parameters, the team designed a mechanism for reusing the FP8 workspace. After online conversion is completed for one layer, its FP8 workspace can be directly reused by adjacent layers, avoiding the memory spikes and time costs associated with repeated allocation and repeated conversion.

This strategy provides a practical path for rapid deployment of extremely large models: compressing the cycle from model release to service availability as much as possible without sacrificing inference accuracy.

Image Inference Support: Seamless Integration of the Visual Encoder

In addition to pure text inference, this adaptation also covers image inference capability. This means the model has multimodal understanding and can accept image inputs and perform mixed image-text inference. In terms of technical implementation, image inputs first pass through a visual encoder and are converted into a set of visual feature vectors. These vectors are then concatenated with text token embeddings before being fed into the Transformer backbone.

In the GPU adaptation, the visual attention module and its corresponding computational path are packaged as an independent execution unit that works together with the text-side sparse attention through a shared KV state management mechanism. This design ensures that in multimodal inference scenarios, the visual and text data flows can proceed efficiently in parallel without significant inference latency increases caused by resource contention.

Implications and Outlook for AI Infrastructure

From this Day-0 adaptation case, several general lessons for AI infrastructure development can be drawn:

First, software stack compatibility matters more than peak hardware compute. In an era of rapidly evolving model architectures, no hardware platform can predict all future operator patterns. A software stack that is highly compatible with the mainstream AI ecosystem allows hardware to quickly take on new model deployment requirements. This has more practical value than pursuing peak TFLOPS alone.

Second, deploying extremely large models requires smarter quantization strategies. The online conversion approach for MXFP4 weights shows that when facing models with trillions of parameters, we do not need to follow the traditional path of completing full conversion before deployment. Strategies such as dynamic conversion and workspace reuse can substantially reduce deployment time and storage overhead without sacrificing accuracy.

Third, inference framework customization is unavoidable. Whether it is SGLang-MUSA or the MATE operator library, the reality is clear: to make general-purpose GPUs efficiently run the sparse attention and mixture-of-experts structures of specific models, deep adaptation of the inference framework is necessary. This adaptation work should be considered part of AI infrastructure development, not an optional enhancement.

Looking ahead, as model scale continues to move toward 10 trillion parameters and beyond, inference infrastructure will face simultaneous bottlenecks in storage bandwidth, memory capacity, and cross-node communication efficiency. The fact that a domestic GPU can complete adaptation for a 3-trillion-parameter model at the Day-0 mark shows that these challenges are not insurmountable. The real test is whether the software stack can sustain this rapid adaptability as model scale grows further and architectural innovation accelerates. This requires a tighter feedback loop between hardware vendors and the open-source community, as well as more developers engaged in the critical field of heterogeneous computing adaptation.