Domestic AI Chips Achieve Day-0 Adaptation for Next-Gen Multimodal LLMs: Unlocking the Vision-Language Reasoning Pipeline

Background: New Challenges in Multimodal LLM Inference

As large model technology evolves from pure text to vision-language multimodal paradigms, both model parameter counts and input context lengths continue to surge, placing higher demands on underlying compute chips and software stacks. Take the recently released new-generation trillion-parameter multimodal LLM as an example: it utilizes a Mixture-of-Experts (MoE) architecture with approximately 3 trillion parameters, supports a million-token context window, and possesses joint text and image reasoning capabilities. In complex scenarios such as Agent tasks, code generation, and financial research, this category of models demonstrates performance approaching top-tier closed-source models.

However, deploying advanced models into actual business applications depends not only on the model algorithms themselves but also on how quickly the underlying chips can adapt. For domestic AI chips in particular, achieving “Day-0 support” for new models—meaning deployment and inference can be completed on the exact day the model weights are released—is a crucial metric for measuring the maturity of their software stacks.

Adaptation Goals: From “Runnable” to “Optimized”

The core objective of this adaptation effort was to run the aforementioned new-generation multimodal LLM entirely on a domestic AI training and inference accelerator card, completing the full-link inference pipeline, including model loading, service startup, text generation, and image recognition. “Completing the pipeline” does not simply mean starting the model process on the hardware; it requires:

1. Complete mapping of the model structure to the underlying instruction set.
2. Highly efficient implementation of key operators across different data types.
3. Stable multi-card communication and distributed inference performance.
4. Correct attention mechanism computation without performance degradation in vision-language mixed-input scenarios.

Because the model employs a sparse attention mechanism—computing attention only for selected token combinations rather than traditional dense full attention—it imposes unique requirements on the chip’s operator library and runtime scheduling capabilities. The generation, sorting, and masking of sparse indices all require highly efficient primitive support from the underlying software stack.

Key Path: Software Stack Parsing and Restructuring

Achieving this adaptation relied on collaborative optimization across three layers of the chip’s software stack.

Layer 1: Framework Plugins Bridging Upstream Models

At the AI framework level, the team adopted the open-source SGLang inference framework as the upstream carrier. SGLang offers significant advantages in long-context scenarios due to its efficient RadixAttention mechanism and template-based fine-grained scheduling. Through the framework’s plugin mechanism, the adaptation mapped custom operators in the model definition to the chip’s operator library interfaces. A major challenge in this layer’s migration was that the new model’s visual encoder contains a massive amount of mixed convolution and Transformer computations. The team had to distribute different computation modes to the optimal execution paths while maintaining the integrity of the computation graph.

Layer 2: Operator Library and Compiler Integration

To address the new “linear attention + sparse index” hybrid path in the model, the team utilized a proprietary unified operator library and a multi-level compilation toolchain for restructuring. Specifically:

• The MTCC compiler automatically downscales and maps high-dimensional tensor operations in the model to the thread-block level. When processing 3D visual feature maps, it reduces data movement overhead through tensor memory layout transformations.
• Triton-MUSA and TileLang-MUSA, two programming language interfaces, allow operator developers to write high-performance kernels using Python-like syntax. In this adaptation, the team rewrote the index generation kernel for sparse attention using TileLang. This compressed logic that originally required hundreds of lines of CUDA code into a few dozen lines of declarative descriptions, while ensuring almost no performance loss.
• The MATE operator library provided pre-built fused operators, such as “Multi-Head Attention + Rotary Positional Encoding” fusion and “Visual Patch Embedding + Layer Normalization” fusion. This reduced kernel launch frequency and significantly improved response speed during small-batch inference.

Layer 3: Seamless Runtime and Communication

For multi-card inference, the model adopts a hybrid strategy of tensor parallelism and expert parallelism. Because each layer of the model contains multiple expert networks distributed across different chips, triggering them initiates all-to-all communication. The software stack uses a dynamic routing monitoring mechanism to prefetch the expert weights of the next token batch during communication gaps, achieving an overlap between communication and computation. Tests showed that at a scale of 64 cards, the communication overhead was kept under 15% of the end-to-end latency.

Data and Performance: Key Benchmark Results

On the code understanding and Agent tool invocation benchmark (Terminal-Bench 4.0), the model scored 65.7%. On the long-horizon programming task benchmark (Code Bench Long-horizon), it scored 61.2%, an increase of about 5 percentage points compared to the previous generation. These achievements rely not only on reinforcement learning strategies at the model algorithm level but also on the underlying inference engine’s efficient support for long sequences.

Notably, in vision-language mixed inference scenarios, the model’s visual encoder must split high-resolution images into hundreds of patches before concatenating them with text tokens. The MUSA software stack optimized the image preprocessing pipeline by parallelizing the patch embedding computation with the tokenization process. This kept the Time-To-First-Token (TTFT) latency under 2.1 seconds in a single-machine, 8-card environment, meeting the interactive requirements of production environments.

For Developers: Migration Toolchain and Debugging Experience

To lower the barrier for ecosystem migration, the software stack provides an automated migration tool that can convert model scripts written for general AI frameworks into versions runnable on domestic chips with a single click. The tool’s coverage includes:

• Dynamic Shape support: Automatically handles tensor dimension changes caused by inputs of varying resolutions.
• Mixed precision strategy: Defaults to BF16+FP32 mixed precision, keeping key loss layers in FP32 to ensure convergence accuracy.
• Debugging interfaces: Provides computation graph capture and per-operator profiling features similar to CUDA, enabling developers to quickly locate performance bottlenecks.

Furthermore, addressing the current demands for LLM service-oriented deployment, the software stack natively integrates an OpenAI-compatible API service interface, supporting streaming output, context caching, and batch scheduling. Developers can complete evaluations or go live simply by changing the endpoint configuration, without needing to modify business logic.

Future Outlook: Continuous Evolution from “Usable” to “Optimized”

As model capabilities deepen toward multimodal Agents, inference infrastructure will face three major trends: memory management under ultra-long contexts; non-contiguous memory access patterns brought by dynamic routing; and deep pipeline coupling between multimodal input preprocessing and generation decoding.

In response to these strategic shifts, the software stack has outlined the following development directions:

1. Cross-node inference optimization: For model parallelism strategies that exceed single-machine memory capacity, implementing hierarchical memory pool management to keep hot weights in high-bandwidth memory while dynamically swapping cold weights in and out.
2. Sparse computation hardware collaboration: Reducing index lookup instruction cycles through instruction-level support for atomic operations on compressed sparse row (CSR) storage formats.
3. Automatic kernel tuning: Introducing an operator parameter auto-tuner based on evolutionary search to automatically explore the optimal block size and pipeline stages after each model deployment.

For developers and enterprise users, when choosing AI infrastructure, it is essential to consider both the peak compute metrics of the chip and the portability and debuggability of the software stack. A mature software stack can shorten the adaptation cycle for new models from months to days, which is critical for the rapidly iterating LLM ecosystem.

The successful practice of this adaptation work validates that domestic AI chips now possess a viable deployment path for inference with the latest frontier models. As the software ecosystem continues to enrich, LLM applications based on proprietary compute power will usher in broader development space.