Domestic AI Compute Stack Achieves Rapid Adaptation for Ultra-Large Models: Engineering Practice from 3-Trillion Parameters to Production-Grade Inference

The “Golden 24 Hours” from Model Release to Compute Deployment

As the large model race enters the “trillion-parameter era,” the gap between the pace of model architecture innovation and the ability of compute infrastructure to keep up has become a core bottleneck constraining AI application deployment. When a sparse mixture-of-experts (MoE) model with 3 trillion parameters and support for million-token context windows releases its full weights, achieving efficient inference on a domestic AI compute platform at first opportunity is not merely a demonstration of technical capability—it is a rigorous test of the entire software stack’s maturity.

“Day-0 adaptation” means that on the very day model weights are publicly released, the hardware platform must have the full inference pipeline operational—from model loading and service startup to text generation and image understanding. This involves far more than simple model file format conversion; it requires systematic engineering across computation graph rewriting, operator mapping, memory orchestration, communication topology, and compiler optimization. This article uses a domestic AI training-and-inference integrated accelerator card and its accompanying software stack as a case study to thoroughly examine the core technical pathways and engineering methodology behind this challenge.

1. The “Assembly Capability” of the Compute Foundation and Software Stack

Deploying any large model depends on tight coupling between hardware compute and software tooling. The compute foundation in this adaptation uses a training-and-inference integrated architecture, meaning the same hardware platform supports both the large-scale parallel computation of training and the low-latency, high-throughput real-time requirements of inference. The key characteristic of this hardware is its unified compute core design—programmable parallel compute units that accommodate both dense matrix operations and sparse computation patterns.

What truly determines adaptation efficiency, however, is the software stack built on top. A mature AI software stack should have a four-layer structure:

• Compiler layer: Provides translation and optimization from upper-level frameworks (such as general-purpose inference engines) to low-level hardware instructions;
• Operator library layer: Contains highly optimized high-performance compute kernels covering common neural network operators and specialized operators for specific model structures;
• Programming interface layer: Supports multiple development paradigms, including CUDA-like low-level programming languages, Python-based Triton-style DSLs, and compiler-oriented intermediate representations;
• Runtime layer: Handles task scheduling, memory management, multi-card communication, and other system-level functions.

In this adaptation practice, the engineering team built an end-to-end migration pipeline spanning “model–operator–compiler–runtime.” The core strategy was not to reproduce model logic from scratch, but rather to perform “translation-style migration” using the various toolchains in the software stack, building on the upstream model’s open-source inference implementation. Specifically, a compatibility layer interface receives the original model code, and then the underlying operator library and compiler redirect critical computation paths to optimized kernels for the proprietary hardware. This approach dramatically shortens the adaptation cycle, compressing support for entirely new model architectures from “months” to “days.”

2. Reconstructing the Computation Path for Sparse Attention Mechanisms

A technical core of ultra-long-context models such as Kimi-K3.1 Pro is its use of a Dense-Sparse Attention (DSA) mechanism. Unlike traditional full attention, sparse attention uses carefully designed indexing strategies to let each query attend to only a subset of key-value pairs, reducing computational complexity from O(n²) to O(n log n) or lower. This design places extreme demands on hardware execution:

1. Dynamic and irregular nature of index selection: The KV positions each token needs to attend to are not a fixed pattern but are dynamically determined by attention scores. This causes data-dependent branches in the computation graph at runtime, rendering static compilation optimizations ineffective.

2. Fragmented KV cache layout: Because only a subset of KV entries is used, the cache must adopt non-contiguous storage formats. Traditional contiguous KV cache strategies no longer apply; block tables and indirect addressing mechanisms must be introduced.

3. Significant divergence between prefill and decode phases: The prefill phase processes long user input sequences and must compute attention outputs for all positions in parallel; the decode phase generates tokens one at a time, processing only one new position’s query per step. The two phases have completely heterogeneous computation patterns and memory access characteristics.

To address these challenges, the engineering team implemented the following adaptation solutions in the software stack:

• Index generation and communication coordination: The index selection process is decoupled from the main computation path, with a separate parallel task pre-generating memory access index tables. A KDA state manager then maintains a unified logical view and physical address mapping of the KV cache.
• Operator-level fusion optimization: Index lookup, KV vector loading, attention score computation, and output weighted summation are fused into a single custom sparse attention operator, avoiding frequent kernel launch overhead and intermediate results round-tripping through memory.
• Dual-path execution engine: During prefill, a tensor-parallel-based long-sequence partitioning strategy maximizes compute throughput; during decode, the engine switches to low-latency mode, using continuous batching to improve GPU utilization.

3. Online Conversion and Execution Strategy for Mixed-Precision Expert Weights

Another notable technical feature is the model’s use of MXFP4-format expert weights. MXFP4 is a high-compression floating-point format designed for next-generation AI hardware, storing weights in 4 bits while providing expression capability approaching FP16 with controlled precision loss. However, most existing hardware cannot directly execute matrix multiplication on 4-bit weights. Achieving efficient computation under constrained precision becomes an engineering challenge.

The engineering solution has two layers:

Online conversion mechanism: Rather than pre-generating complete offline FP8 weight files—which would consume hundreds of gigabytes of additional storage and expose inference deployment latency—the approach decompresses MXFP4 weights to FP8 format in real time during model loading. This conversion is designed as a streaming pipeline operation: decompress and write to high-speed memory buffers simultaneously, keeping startup time to the minute level.

Cross-layer workspace reuse strategy: FP8 weights, while double the size of FP4, are still half that of FP16. However, for a 3-trillion-parameter model, peak memory footprint remains substantial. To address this, the team implemented a cross-layer shared FP8 temporary workspace. Because model inference executes layer by layer, once layer N finishes computation, its weight memory can be immediately released for layer N+1 to reuse. This dynamic memory planning mechanism makes it possible to fit enormous model parameters within limited per-card memory.

It is worth noting that this online conversion and workspace reuse strategy is not cost-free. The conversion operator itself introduces additional computational overhead. The engineering team employed efficient data-parallel decomposition algorithms to achieve conversion throughput in the tens of gigabytes per second, keeping conversion overhead within 5% of overall inference latency.

4. Integrating Vision Models and Cross-Modal Pipelines

Leading large models today universally incorporate multimodal capabilities, and Kimi-K3.1 Pro likewise supports image input. This means the vision encoder (Vision Transformer) and text decoder must coordinate efficiently. In the overall execution pipeline, the attention computation in the vision portion differs from the sparse attention mechanism in text decoding but shares the underlying KV cache management system.

The software stack achieves dynamic switching between the two attention modes through a unified multimodal attention computation graph description. When input contains images, visual token embeddings and text token embeddings are concatenated along the feature dimension and then fed into a unified Transformer decoding layer. This process poses additional challenges for memory bandwidth—image token counts are typically far larger than text token counts. The engineering team effectively controls attention computation scale through visual token pre-pruning and key-region sampling strategies.

5. The Systems Thinking Behind Performance Data

Ultimately, evaluating whether an adaptation succeeds requires returning to quantifiable performance metrics. The target platform in this adaptation demonstrated compelling data in authoritative evaluations: ranking first among open-weight models on an open-ended assistant leaderboard, scoring 61.2% on long-sequence code generation tasks, and achieving 65.7% accuracy on terminal command execution tests.

These results stem not from any single technological breakthrough but from systematic engineering optimization:

• Compute efficiency: Through joint optimization of sparse attention operators and mixed-precision execution, effective compute density reaches over 43% of hardware theoretical peak;
• Memory efficiency: Through workspace reuse and KV cache compression, single-machine inference memory requirements for the 3-trillion-parameter model are held to the hundreds-of-gigabytes range, enabling scaling from 8 cards to 64-card clusters;
• Scaling efficiency: Multi-node communication optimization tested on thousand-card clusters achieves linear scaling efficiency above 89%.

6. Conclusion and Outlook: Crossing from “Usable” to “Production-Ready”

This successful adaptation marks the domestic AI compute stack’s ability to track the world’s most advanced model architectures entering the “hours-level” response era. From an engineering methodology perspective, the core lessons can be summarized in three points: First, build an open programming interface ecosystem so that upstream open-source community code can migrate seamlessly; second, develop customized operators for model structural characteristics rather than relying on “hit-or-miss” matching with general-purpose operator libraries; third, establish a global dynamic scheduling mechanism across memory, compute, and communication.

Looking ahead, as model parameter scales advance toward 10 trillion and inference scenarios evolve from single-modal to multimodal and from short context to unlimited context, co-design between compute platforms and model architectures will become increasingly important. Day-0 adaptation capability will become a key benchmark for evaluating a compute vendor’s core competitiveness. The technical approach validated in this practice—adapting to model evolution through elastic software stack architecture and deep operator-level optimization rather than frequent hardware iteration—may well be the defining paradigm for next-generation AI infrastructure.