Domestic AI Chip Achieves Day-0 Adaptation for a Frontier 3-Trillion-Parameter Model with Full Text-Image Inference Support

Background: The urgency of compute adaptation in the large model era

As large model parameter counts move beyond the trillion level and architectures continue to evolve toward mixture-of-experts and sparse attention designs, the adaptation burden on underlying AI chips has increased sharply. Once open-weight models are released, the speed at which inference pipelines can be brought online directly determines how quickly AI infrastructure becomes usable in real deployments. This Day-0 adaptation capability has become a core measure of AI chip platform maturity.

Recently, an open-source large model with an estimated 3 trillion parameters was released with full weights. The model supports an ultra-long context window of up to 1 million tokens and demonstrates performance comparable to leading closed-source models in code generation, 3D creation, financial research, knowledge-based Q&A, and long-horizon agent tasks. Around the same time, a Chinese AI chip vendor announced that it had completed inference adaptation for the model, covering model loading, service startup, and full text-and-image multimodal inference. This marks a key step for domestic AI compute platforms in supporting frontier large models.

Core challenges in large model inference adaptation

1. Computational path reconstruction caused by architectural differences

Frontier large models increasingly adopt hybrid attention mechanisms, combining conventional full attention with sparse attention such as dynamic sparse attention (DSA) to reduce computational complexity in ultra-long-context scenarios. While this improves efficiency, it creates significant adaptation challenges for chips. General-purpose GPU architectures are typically optimized for dense matrix multiplication, whereas sparse attention involves irregular memory access patterns and index computation. Chip vendors must provide dedicated support at the operator library level.

2. Complexity of the multimodal inference pipeline

Text-image inference requires the model to process both high-dimensional visual encoder features and semantic information from text tokens. This involves vision-language cross-modal alignment, dynamic generation of attention masks, and several other stages. A missing computational path in any part of the pipeline can cause inference to fail, placing high demands on software stack completeness and compatibility.

3. The dual challenge of inference performance and stability

Even with quantization, a 3-trillion-parameter model still imposes substantial memory usage and latency during inference. The chip must preserve numerical accuracy while using operator fusion and memory reuse to improve efficiency. At the same time, it must remain stable over long-running services, avoiding memory leaks or computational anomalies that could interrupt deployment.

Key technical paths for Day-0 adaptation

1. Seamless support at the inference framework layer

In this adaptation effort, the engineering team selected the widely used SGLang inference framework as the integration layer. Through the interface between SGLang and the underlying chip software stack, the team progressively transformed the upstream Python implementation of the model into an executable compute graph on the target chip. This was not a simple code translation. The forward propagation logic had to be analyzed layer by layer to identify all computation nodes involving custom operators and design alternative implementations for them.

# Example: model loading and adaptation flow in an inference framework (pseudocode)
from sglang_like_runtime import Engine, ModelConfig
model_cfg = ModelConfig(
    model_path="./kimi_like_model",
    tokenizer_path="./tokenizer",
    tensor_parallel_size=8,
    attention_backend="sparse_dsa",  # enable sparse attention backend
    vision_backend="native_musa"      # backend for the visual encoder
)
engine = Engine(model_cfg)
engine.start_service(port=8090)

During framework adaptation, the team focused on three areas: whether the runtime scheduling logic was compatible with the chip memory management model, whether the framework operator dispatch mechanism could correctly identify compute kernels supported by the chip, and whether communication primitives in distributed inference could be mapped onto the chip multi-card interconnect protocol.

2. Coordinated optimization across operator libraries and compilers

Operator libraries are the critical link between AI frameworks and low-level hardware. For the newly introduced sparse attention computation, the team needed to implement specialized fused operators that combine index computation, data aggregation, and matrix multiplication into single kernels to reduce memory access. Specifically, DSA dynamically selects the key-value pairs that participate in attention based on input token similarity. This involves top-k selection and index reordering. Simulating these operations within a general matrix multiplication framework would reduce efficiency by several times.

The team used a Triton-like domain-specific language to write high-performance operator kernels and relied on the low-level compiler for automatic vectorization and register allocation optimization. At the same time, custom instruction extensions for operations such as sparse index lookup were applied to further reduce kernel latency. This three-layer approach of DSL implementation, compiler optimization, and instruction extension has become an effective methodology for rapidly landing new operators.

3. Engineering practice for multi-card communication and memory management

A 3-trillion-parameter model cannot fit entirely within a single card memory and must be deployed across multiple cards using tensor parallelism or pipeline parallelism. In this adaptation, the team established mapping between the chip-specific multi-card communication protocol and the distributed primitives of the inference framework, keeping communication overhead within an acceptable range. Specific measures included:

• Memory pooling: abstracting memory across cards into a unified pool with on-demand allocation and reclamation to reduce OOM risks caused by fragmentation.
• Communication-computation overlap: running communication and computation concurrently through independent threads and streams so that collective operations such as all-reduce are hidden behind kernel execution.
• KV cache management: designing a paged KV cache mechanism for the 1-million-token context scenario, supporting dynamic expansion and replacement to avoid memory spikes caused by context length variation.

4. Targeted optimization of the visual computation path

A core difficulty in text-image inference is how to efficiently combine the high-dimensional feature tensor output by the visual encoder, usually in the form of sequence length by hidden dimension, with text token sequences for subsequent attention computation. The team integrated a dedicated visual computation module into the software stack to support hardware acceleration for image patch embedding, positional encoding injection, and vision-language cross-attention. It also implemented an adaptive token pruning strategy based on information density differences across image regions. This reduces the number of visual tokens participating in attention without sacrificing inference accuracy, thereby lowering overall computation.

Benchmark performance and industry implications

According to third-party evaluations, the model achieved strong results on several authoritative benchmarks: 65.7% on a terminal command execution benchmark and 61.2% on a long-horizon code generation benchmark, an improvement of nearly 10 percentage points over the previous generation. Some code and agent task metrics were close to or on par with leading closed-source models. These results ranked first among open-weight models and 13th among all evaluated models.

The fact that a Chinese AI chip was able to complete full-chain adaptation within days of the model release reflects several industry trends:

First, AI chip software stack maturity is becoming a core competitive advantage. Hardware specifications are no longer the only criterion. The ability to rapidly adapt to constantly emerging model architectures determines whether a chip is usable in real business scenarios. The design philosophy demonstrated by the MUSA software stack in this effort, framework-agnostic, extensible operators, and automatic compiler optimization, represents the mainstream direction for AI chip software stacks.

Second, model-hardware co-design is becoming more important. Large model architecture innovation no longer depends solely on algorithm teams. Chip vendors need to participate deeply in model design discussions and bring hardware characteristics into architectural decision-making earlier. This collaborative model can significantly shorten adaptation cycles for future models.

Third, open-source ecosystem adaptability determines the vitality of AI infrastructure. This adaptation was built on an open-source model and completed using a community inference framework rather than waiting for a chip vendor-specific solution. This demonstrates that an open, standardized software interface layer is essential for a healthy AI chip ecosystem.

Outlook: from Day-0 adaptation to continuous optimization

Day-0 adaptation is only the starting point. As the model enters real business deployment, more challenges will emerge. Can inference latency under ultra-long context meet real-time interaction requirements? Can multimodal inference throughput support large-scale concurrent requests? These issues require long-term collaboration between chip vendors and model developers through continuous joint optimization.

Going forward, AI chip competition will likely focus on three dimensions: adaptation speed, meaning the time window from new model release to usability; runtime efficiency, meaning inference speed and energy consumption under the same hardware conditions; and ecosystem richness, meaning the operator libraries, toolchains, and community resources available to developers. This adaptation effort provides a useful reference for domestic AI chips across all three dimensions. For AI infrastructure builders, establishing an open, efficient, and iterable model adaptation pipeline will become a key engineering capability for supporting business innovation.