How Domestic GPUs Achieved Day-0 Adaptation for a 3-Trillion-Parameter LLM

1. Background: Engineering Challenges in Large-Model Inference Deployment

As large models scale from hundreds of billions to trillions of parameters, a core tension in inference deployment is becoming increasingly prominent: a significant gap exists between continuous architectural innovation—sparse attention, mixture of experts, ultra-long context—and the adaptation cycle of hardware and software stacks. Under the traditional model, a newly released model often requires weeks or even months of adaptation before inference services can actually be deployed on specific hardware.

“Day-0 adaptation” refers to completing the entire pipeline—from model loading and service startup to basic inference paths—on the same day the model weights are publicly released. This places extremely high demands on a hardware vendor’s software stack capabilities: it must not only cover conventional dense computation paths but also provide complete operator support for newly introduced features such as sparsification and low-precision quantization.

Recently, a domestic AI training-and-inference accelerator, together with its software stack, completed Day-0 adaptation for a 3-trillion-parameter large model, enabling basic text and image inference pipelines. This article breaks down the key technical paths and engineering practices behind this adaptation process.

2. Analysis of Model Architecture Features

The target model for this adaptation has the following notable technical characteristics:

2.1 Ultra-Long Context Support

The model supports a context length of 1 million tokens. This means KV cache memory usage becomes a critical bottleneck. Estimated at conventional FP16 precision, the KV cache requirement for million-level tokens could reach hundreds of gigabytes, which must be optimized through sparsification, low-precision storage, and paged management.

2.2 Sparse Attention Mechanism

The model adopts a dynamic sparse attention (DSA) scheme. Unlike traditional dense attention, DSA requires index selection over attention scores during inference, performing attention computation only on top-k key-value pairs. This requires the hardware software stack to support:

• Efficient implementation of index selection operators
• Non-contiguous layout management for KV caches
• Differentiated computation paths for the Prefill and Decode stages

2.3 Mixture of Experts and Low-Precision Weights

The model stores expert weights in MXFP4 format. MXFP4 is a 4-bit floating-point format that further compresses memory usage compared to FP8, but requires format conversion before computation. If an offline conversion approach were used, a complete FP8 weight copy would need to be generated in advance, introducing additional storage overhead and deployment latency for large-scale models.

3. Key Technical Paths for Software Stack Adaptation

3.1 Overall Architecture

The adaptation work was carried out based on the hardware backend of the inference framework. The overall pipeline can be summarized as:

Model code (community implementation)
    ↓ Operator mapping and graph optimization
Compiler layer (MTCC)
    ↓ Code generation
Operator library layer (MATE / Triton / TileLang)
    ↓ Hardware instructions
AI accelerator hardware

By building on upstream model implementations, the team transformed community model code into an actual inference service on the target hardware. In this process, the compiler and operator libraries carried out the core adaptation work.

3.2 Complete Execution Pipeline for Sparse Attention

For the DSA mechanism, the software stack completed the following key adaptations:

Index selection and KV cache layout: The core of DSA lies in dynamically selecting the key-value pairs that participate in computation. The software stack needed to implement an efficient index selection operator and support non-contiguous KV cache layouts. In the Prefill stage, because sequence lengths are large, the computational overhead of index selection is significant; in the Decode stage, incremental KV cache updates must be handled.

Computation path adaptation: The Prefill and Decode stages have clearly different computational characteristics—the former is compute-intensive, while the latter is memory-access-intensive. The software stack optimized computation paths separately for the two stages, integrated sparse attention operator libraries, and formed a complete execution pipeline together with the following components:

• KDA state management: handles cross-layer transfer of attention states
• Cross-layer residual: manages cross-layer data flow for residual connections
• Visual Attention: supports visual feature processing in multimodal inputs

3.3 Online Conversion of Low-Precision Expert Weights

For MXFP4-format expert weights, the team adopted an approach combining online conversion with cross-layer reuse of an FP8 workspace:

# Illustrative pseudocode: online weight conversion and workspace reuse
def expert_forward(x, expert_weights_mxfp4, fp8_workspace):
    # Online conversion: convert MXFP4 weights to FP8 on demand
    w_fp8 = convert_mxfp4_to_fp8(expert_weights_mxfp4)
    # Cross-layer reuse: write converted weights into a shared workspace
    fp8_workspace.update(w_fp8)
    # Execute expert computation
    return matmul(x, fp8_workspace.current())

The core advantages of this approach are:

1. No need to generate complete FP8 weights offline: avoids additional storage overhead and deployment preparation time
2. Cross-layer workspace reuse: multiple expert layers share the same FP8 workspace, reducing peak memory usage
3. On-demand conversion: conversion occurs only when expert computation is needed, reducing unnecessary computation

3.4 Multi-Level Operator Library Collaboration

During adaptation, the team used multiple operator implementation approaches in combination:

• MATE operator library: provides deeply optimized core operators, including key computations such as sparse attention and matrix multiplication
• Triton-MUSA: supports writing custom operators using Triton-like syntax, balancing development efficiency and performance
• TileLang-MUSA: targets lower-level operator development, supporting fine-grained scheduling and memory control
• MTCC compiler: compiles high-level operator descriptions into instruction sequences for the target hardware

This multi-level collaborative strategy allowed the team to choose the most appropriate implementation based on operator characteristics—using the MATE library for core operators to ensure performance, Triton for rapid iteration on new operators, and TileLang for fine tuning of special operators.

4. Key Considerations in Engineering Practice

4.1 Memory Management

Memory usage for a 3-trillion-parameter model is a core challenge. In addition to the weights themselves, KV caches, intermediate activations, and workspaces all require careful management. The following strategies are recommended:

• Layered weight loading: inactive layer weights can be temporarily stored in host memory
• KV cache paging: allocate and reclaim KV cache pages on demand
• Workspace reuse: share workspace memory across different computation stages

4.2 Overlapping Computation and Communication

In distributed inference scenarios, the communication overhead of expert parallelism cannot be ignored. By overlapping communication with computation, end-to-end latency can be effectively reduced.

4.3 Balancing Precision and Performance

Low-precision computation (FP8, MXFP4) brings memory and bandwidth advantages but may also introduce precision loss. In practice, an appropriate precision strategy must be selected based on task characteristics—retaining high-precision paths for precision-sensitive tasks and using low-precision paths for throughput-sensitive tasks.

5. Performance Reference and Validation

According to public information, the model performs as follows on multiple evaluation benchmarks:

Benchmark	Score/Ranking
Agent leaderboard (open-weight models)	1st
Agent leaderboard (all evaluated models)	13th
Work/coding/chat categories (open models)	All ranked first
Terminal-Bench 4.0	65.7%
Code Bench 3.0 Long-horizon	61.2%

These data indicate that the model has strong overall capabilities among open-weight models. The completion of Day-0 adaptation means developers can begin inference deployment and business validation on domestic hardware on the same day the model is released.

6. Summary and Outlook

The core value of this adaptation work lies in validating a feasible Day-0 adaptation methodology for large models:

1. Architecture first: analyze model architecture features in advance and identify key computation paths
2. Layered operators: deeply optimize core operators, rapidly iterate on new operators, and finely tune special operators
3. Online conversion: convert low-precision weights on demand to avoid offline preparation overhead
4. Complete pipeline: connect the full pipeline from model loading to service startup

For developers, this means that when choosing inference hardware, they should evaluate not only peak compute but also the software stack’s ability and speed in adapting to new model architectures. As the pace of model architecture innovation accelerates, “Day-0 adaptation” capability will become an important dimension of inference hardware competitiveness.