Day-0 Large Model Adaptation on Domestic AI Chips: Technical Path and Engineering Practice

1. The Significance and Challenges of Day-0 Adaptation

As large model iteration cycles continue to accelerate, the window between model weight release and inference service availability keeps shrinking. “Day-0 adaptation” refers to completing the full validation pipeline—from model loading and service startup to establishing the basic inference path—on the same day that model weights are publicly released. This tests not only the maturity of a chip’s software stack but also the engineering team’s depth of understanding of emerging model architectures.

The core challenges of Day-0 adaptation fall into three areas. First, new models often introduce computational paradigms not previously encountered, such as novel sparse attention mechanisms or low-precision expert weight formats, requiring rapid software stack response. Second, community model code is typically written for the general-purpose GPU ecosystem, and migrating it to domestic chips requires operator-level mapping and rewriting. Third, inference services demand end-to-end stability—a gap in any single component will break the entire pipeline.

This article uses the adaptation of a 3-trillion-parameter-class MoE model on a domestic AI training-inference accelerator as a case study. It outlines the complete technical solution, from computational path migration to inference service deployment, and distills reusable engineering methodologies.

2. Overall Adaptation Architecture

The overall approach can be summarized as “inheriting upstream implementations, layered mapping, and establishing the full pipeline”:

1. Inheriting upstream model implementations: Build on the chip adaptation branch of an open-source inference framework, maintaining semantic consistency with community model code and reducing the cost of tracking upstream updates.
2. Operator-level mapping: Map the key computational paths of the new architecture onto the target hardware layer by layer, using an in-house operator library, a Triton-style programming interface, a domain-specific compilation language, and a compiler toolchain.
3. Service-level validation: Complete end-to-end verification of model loading, service startup, and basic text and image inference paths, ensuring a closed loop from weight files to actual request responses.

The key principle of this architecture is not to cover every operator in one pass, but to prioritize the availability of the main computational path and gradually fill in edge paths.

3. Adapting the Sparse Attention Mechanism

The model’s sparse attention scheme (DSA) was the primary challenge. Unlike conventional full attention, DSA requires index selection before computation to determine the subset of key positions each query should attend to, introducing an additional computational pre-processing path.

The adaptation work addressed three layers:

3.1 Index Selection Path

Index selection involves scoring, sorting, and filtering, with a computational pattern quite different from conventional matrix multiplication. The engineering team decomposed it into segmentable operator combinations, implemented using sparse attention primitives from the operator library to avoid introducing inefficient generic implementations on the critical path.

3.2 KV Cache Layout

Sparse attention accesses the KV cache in a non-contiguous pattern, so the cache layout must be optimized for the output of index selection. The team redesigned the KV cache storage arrangement so that selected key-value pairs can be read with high locality, reducing memory access overhead during inference.

3.3 Separating Prefill and Decode Paths

The prefill phase is compute-intensive, while the decode phase is memory-intensive, and they place different requirements on sparse attention implementation. The two paths were handled separately: the prefill phase focused on operator fusion and parallelism tuning, while the decode phase focused on cache access efficiency and batching strategy.

In addition, sparse attention must work in concert with state management modules, cross-layer residual connections, and visual attention paths to form a complete execution pipeline. Any missing component would result in incomplete multimodal inference capability.

4. Online Conversion of Low-Precision Expert Weights

The model stores expert weights in MXFP4 format, a block-scaled 4-bit floating-point format that significantly compresses memory usage while preserving accuracy. However, the target hardware’s compute units have more mature support for FP8 and higher-precision formats, so executing MXFP4 computation directly is not the optimal path.

The engineering team’s solution was online conversion + cross-layer reuse of an FP8 workspace:

• Online conversion: Before expert computation, MXFP4 weights are dequantized/converted to FP8 format in real time. The conversion overlaps with the compute pipeline to avoid additional synchronization stalls.
• Cross-layer workspace reuse: Expert computations for different layers are staggered in time, so they can share the same FP8 workspace memory without allocating a separate converted weight buffer for each layer.
• No offline pre-conversion required: The entire process does not require pre-generating complete offline FP8 weight files, significantly reducing deployment storage overhead and preparation time.

The trade-off here is accepting a small amount of runtime conversion overhead in exchange for a substantial reduction in memory usage and deployment complexity. For a model with trillions of parameters, the weight copies produced by offline pre-conversion would be unacceptable in terms of both storage and loading time.

Illustrative pseudocode:

for each expert in activated_experts:
    fp8_weight = convert_mxfp4_to_fp8(expert.weight, workspace)  # online conversion
    output += matmul(token_states, fp8_weight)                   # workspace reuse

5. Toolchain and Compiler Optimization

The implementation of operator mapping relies on a multi-layered toolchain:

• Operator library: Provides hand-optimized implementations of high-frequency primitives such as sparse attention, matrix multiplication, and normalization, serving as the performance baseline.
• Triton-style programming interface: Enables rapid implementation of new computational patterns not yet covered by the operator library, lowering the development barrier.
• Domain-specific compilation language: Used for writing high-performance kernels targeting specific computational patterns, balancing development efficiency with runtime performance.
• Compiler toolchain: Responsible for kernel scheduling, memory planning, and target code generation, serving as the unified outlet through which all layers above are deployed to hardware.

The division of labor is: mature primitives go through the operator library, novel computations go through the programming interface and domain-specific language, and everything ultimately converges through the compiler. This layered approach avoids both the high cost of hand-writing every operator and the performance loss of relying entirely on automatic compilation.

6. Key Engineering Takeaways

The following reusable lessons emerged from this adaptation effort:

1. Prioritize the main pipeline: Ensure model loading, service startup, and basic inference work first, then fill in multimodal and edge paths.
2. Maintain semantic consistency with upstream: Keep the adaptation branch as close to the community implementation as possible to reduce duplicate work during subsequent model updates.
3. Choose precision strategies based on requirements: The timing of low-precision weight format conversion (online vs. offline) should be weighed against model scale and deployment constraints.
4. Cache layout is the performance key for sparse attention: Data locality after index selection determines actual memory access efficiency.
5. Converge through a layered toolchain: The operator library, programming interface, and domain-specific language each serve distinct roles—avoid redundant development.

7. Conclusion

The essence of Day-0 adaptation is validating the responsiveness and engineering completeness of a chip’s software stack against rapidly evolving model architectures. The emergence of features such as sparse attention and low-precision expert weights requires adaptation work to evolve from “operator replacement” to “computational path reconstruction.” Through layered mapping, online conversion, and workspace reuse, domestic AI chips can now complete the full pipeline—from weights to service—on the same day a model is released, providing another viable path for large-scale deployment of large model inference.