How Domestic GPUs Achieve Day-0 Large Model Adaptation: Sparse Attention Optimization in Practice with Kimi-K3.1 Pro

I. Background: The “Day-0 Adaptation” Challenge in the Era of Large Models

As open-source large models scale to trillions of parameters, the adaptation cycle between their architectural innovations (such as sparse attention and mixture-of-experts) and inference frameworks has become a critical bottleneck for deployment. “Day-0 adaptation” refers to completing deployment of a new model on a specific hardware/software stack on the very day its weights are released—covering the full pipeline from loading and startup to basic inference. This tests not only the compute power of the acceleration hardware but also how quickly and effectively the software stack can understand and port the latest model architectures.

The case study in this article involves a model with a 1-million-token context length that employs two cutting-edge techniques: deep sparse attention (DSA) and MXFP4 expert weights. For domestic accelerator cards outside the NVIDIA ecosystem, whether they can complete adaptation on launch day directly determines their competitiveness in the large model serving market. Below, we systematically break down the key technical paths involved in this process, for the reference of AI Infra engineers.

II. Overall Architecture: A Four-Layer Transformation from Community Code to Production-Grade Inference Service

Achieving rapid deployment of a new model on domestic accelerator cards is by no means a simple operator replacement. A typical technology stack includes the following layers, which must work in coordination:

Layer	Responsibility	Key Technical Points
Framework integration layer	Ingest upstream model definitions while maintaining community interface compatibility	Inference framework branch maintenance, model weight loading validation
Compiler/intermediate representation layer	Lower high-level model code into hardware-executable instructions	Dynamic shape support, loop fusion, memory allocation optimization
Operator library layer	Provide deeply optimized implementations of high-frequency compute kernels	Sparse attention, matrix multiplication, activation functions, etc.
Runtime/scheduling layer	Manage KV cache, memory pools, request scheduling	PagedAttention-style management, multi-stream concurrency

In this case, the team built on a self-developed inference framework branch (retaining upstream SGLang interface compatibility) and used a unified programming language compiler along with two domain-specific language frontends to gradually “translate” new operators in the model into hardware-native instructions. The core difficulty is this: 80% of computation time in modern large models is concentrated in a handful of “hot operators,” but the 20% of non-standard custom operators (such as DSA-related indexing logic) are often the main cause of adaptation failures.

III. Key Path One: Complete Migration of Sparse Attention DSA

3.1 Understanding the Execution Logic of DSA

The DSA mechanism used by this model does not simply discard attention scores. Its core consists of two steps:

• Index Selection: Based on local relevance between query vectors and key vectors, a lightweight scoring network selects the Top-K critical positions (or key KV blocks). In long-context scenarios (1 million tokens), this reduces computational complexity from O(N²) to O(N·K).
• Sparse Matrix Computation: Standard QK^T and softmax computations are performed only on the selected positions, and the output is then aggregated and updated.

3.2 Four Sub-Problems That Must Be Solved in the Software Stack

Sub-Problem	Native Implementation (Assumed)	Adaptation Approach
Insufficient parallelism in index selection	Relies on CUDA atomic operations and shared memory reduction	Rewritten as a vectorized sorting algorithm on the custom hardware; Top-K selection split into a two-stage reduction, using asynchronous copies to hide latency
KV cache layout conflicts	Contiguous storage assumption	Rearranged in a “block-sparse” pattern; a two-level page table structure introduced to support memory access continuity under irregular indices
Large one-shot computation in the prefill stage	Greedy scheduling	Long sequences split into overlapping windows; attention and index generation pipelined to avoid intermediate peak memory usage
Only one token per step in the decode stage	Repeated full-index inference	Index results from the previous round cached; only local positions of the new token are incrementally updated

Among these changes, the most fundamental is treating “sparse indexing” as a form of “data flow rearrangement” rather than “computation omission.” Traditional GPU implementations use control-flow masks to skip invalid computation, but domestic accelerator cards have coarser scheduling granularity. The team therefore pre-generates an “aggregated index table” at the operator library level, directly driving the hardware DMA engine to fetch KV blocks on demand, reducing indexing overhead from microseconds to nanoseconds.

3.3 Coordinated Adaptation with State Management and Residual Connections

DSA is not a standalone module. Its index results must be coordinated with KV cache state management (KDA)—for example, when cache eviction occurs, the corresponding sparse index tables must also be invalidated and rebuilt in sync. In addition, this model architecture includes a “cross-layer residual” feature, where the hidden state of one layer is directly injected into the attention computation of multiple subsequent layers. The team modified the computation graph compilation rules to move residual stream bindings forward to the tensor descriptor level, so that sparse indexing can automatically perceive offsets from different layers when selecting KV, avoiding logical misalignment.

IV. Key Path Two: A “No Offline Conversion” Approach for MXFP4 Expert Weights

4.1 The Nature of the MXFP4 Problem

This model stores expert network weights in a 4-bit MX (Microscaling Format) specification. While this significantly reduces memory footprint (4× savings compared to FP16), most accelerator Tensor Cores only support FP8 or higher-precision computation. Directly computing with 4-bit weights is not only slow but may also cause precision loss.

4.2 Engineering Wisdom for Avoiding Full Dequantization

A naive approach would be to convert all experts’ MXFP4 weights to FP16 or BF16 online during model loading. But for a large model with hundreds of experts and hundreds of millions of parameters per expert, this would consume substantial startup time and double memory usage.

The case in this article adopts an “online block-level upscaling + cross-layer workspace reuse” strategy:

1. Define FP8 workspaces: Fixed-size contiguous regions are reserved in memory (typically sized to hold one expert’s computation requirements).
2. On-demand scheduling: When the current layer needs to compute a particular expert, only that expert’s weights are read in a streaming fashion, and a dedicated DMA engine performs “MXFP4→FP8” conversion in real time (using hardware built-in scaling factor lookup instructions).
3. Workspace time-slice reuse: Since inference is serial across layers and experts are independent of each other, the previous expert’s FP8 workspace can be overwritten before the next expert begins computation. This keeps additional memory overhead below the size of a single expert’s FP8 weights—far less than a full offline conversion scheme.

The precision loss of this approach is nearly zero (only one extra rounding), while the memory access throughput of core computation improves by over 30% due to narrow-bit-width reads.

V. Summary and Insights: Day-0 Adaptation Is Not Mysterious, But It Requires Systematic Preparation

Looking back at the entire adaptation process, three core insights emerge:

1. Investment in compilers and operator libraries is the strongest lever. Directly modifying user model code (e.g., rewriting sparse attention as dense) is enormously labor-intensive and error-prone. Through compiler-level pattern matching and automatic scheduling, adaptation time can be compressed from weeks to days.
2. Data layout is the hidden deciding factor. Whether it is block-sparse KV cache rearrangement or MXFP4 workspace reuse, both prove one principle: optimizing “effective compute” (the compute actually utilized) is more valuable than boosting peak compute.
3. Hardware-software co-design is a long-term trend. Day-0 adaptation capability is essentially a reflection of the software stack’s “programmability” over hardware features. A unified programming language and multi-level intermediate representations form the infrastructure for such adaptation.

For technical teams looking to build their own large model inference platforms, the takeaway from this case is: don’t wait until a new model is released to begin adaptation work. Instead, abstract the “non-standard operators” in mainstream model architectures into reusable modules in advance. When a new model appears, only new indexing/layout logic needs to be implemented, and existing acceleration kernels can be reused. This is the underlying logic that makes “Day-0” possible.