How Domestic GPU Software Stacks Achieve Day-0 Adaptation for Large Language Models

1. Why Day-0 Adaptation Has Become a Critical Challenge for LLM Deployment

When a new-architecture large language model with trillions of parameters is released, infrastructure teams typically have only hours to days to respond. Once model weights are made public, the ability to load them, start an inference service, and run a basic inference pipeline within the same day directly determines the pace of subsequent evaluation, tuning, and business integration. This “adapt at release” capability is known in the industry as Day-0 adaptation.

The difficulty of Day-0 adaptation lies not in whether the model can run at all, but in achieving three goals simultaneously: model loading does not crash, service startup does not fail, and the compute path is not slow. For models that adopt new attention mechanisms and expert weight formats, generic operator libraries often cannot directly cover their critical compute paths. The chip vendor’s software stack team must dive into the model implementation and complete operator-level migration and integration.

This article uses a three-trillion-parameter model that supports million-token context lengths as a case study to examine the core technical challenges and engineering solutions involved in Day-0 adaptation on a domestic GPU software stack.

2. Adaptation Pipeline Overview: From Community Model Code to Production Inference

The complete Day-0 adaptation pipeline can be broken down into four stages:

1. Model loading: Parse weight formats, complete parameter mapping, and plan memory layouts.
2. Service startup: Initialize the inference engine, register custom operators, and configure parallelization strategies.
3. Prefill stage: Process long-context inputs, execute attention computation, and write KV cache.
4. Decode stage: Generate tokens one by one, executing sparse attention index selection and expert routing computation.

Along this pipeline, the software stack must serve as both a “translation layer” and an “acceleration layer”—translating generic implementations in community model code into instruction streams executable on the chip, while also customizing compute paths for the chip’s architectural characteristics.

3. Operator-Level Adaptation for Sparse Attention

3.1 Why Sparse Attention Requires Dedicated Adaptation

Traditional dense attention exhibits quadratic computational complexity when sequence lengths reach millions of tokens, and KV cache usage quickly exceeds available memory. The core idea behind sparse attention is that not every token needs to attend to all historical tokens. Instead, an index selection mechanism identifies only the top-K most relevant historical positions for attention computation.

This mechanism introduces three new computational components:

• Index selection: Based on the current query vector, select the top-K most relevant positions from historical KV entries.
• KV cache layout: Under sparse access patterns, the physical layout of the KV cache must support non-contiguous, strided reads.
• Prefill/Decode path separation: The Prefill stage must build a complete index structure, while the Decode stage reuses the index and updates it incrementally.

3.2 Integrating the Index Selection Module

Index selection is typically performed by a small scoring network whose computation pattern differs from standard attention: its input consists of similarity scores between the query vector and historical key vectors, and its output is a sparse index set. The software stack must implement dedicated operators for this purpose, supporting batched top-K selection and mask generation.

In practice, the team provided sparse attention base operators through the operator library, covering three sub-steps: index scoring, top-K filtering, and mask application. At the same time, the compiler mapped the index selection logic from the community implementation onto the chip’s vector compute units, avoiding the performance penalties of element-wise loops.

3.3 KV Cache Layout Optimization

Sparse attention accesses the KV cache in an “on-demand” manner. A conventional contiguous memory layout would cause a large number of ineffective cache line loads. Optimization strategies include:

• Blocked storage: Organize the KV cache into fixed-size blocks, with index selection operating at block granularity to reduce random access frequency.
• Index-aware memory alignment: Prefetch adjacent blocks based on the distribution characteristics of top-K selection results.
• Separate Prefill/Decode management: Use one layout for KV cache written during Prefill and another for incremental writes during Decode, linked through the index structure.

3.4 Coordination with State Management and Cross-Layer Residuals

Sparse attention is not an isolated module. In this model, it forms a complete execution pipeline together with the state-space management mechanism, cross-layer residual connections, and visual attention modules. The software stack must ensure that:

• The state management module updates state vectors after sparse attention writes to the KV cache.
• Cross-layer residual connections correctly perform additions between sparse attention outputs and original inputs.
• The sparse patterns of visual attention and text attention remain compatible, avoiding conflicts between two index structures.

4. Online Conversion Strategy for Mixed-Precision Expert Weights

4.1 Background

The model’s expert layer weights use a low-precision floating-point format (MXFP4), while the inference engine’s expert compute units are typically optimized for a different low-precision format (FP8). Using the original weights directly prevents the compute units from reaching peak performance. Conversely, pre-generating complete offline FP8 weights incurs additional storage overhead and conversion time.

4.2 Online Conversion and Cross-Layer Workspace Reuse

The team adopted an approach of online conversion combined with cross-layer workspace reuse:

• At the start of expert computation, convert the current layer’s MXFP4 weights to FP8 format in real time, writing them into a pre-allocated FP8 workspace.
• Immediately after conversion completes, execute expert matrix multiplication.
• Reuse the workspace across layers, avoiding separate FP8 weight buffers for each layer.
• Pipeline the conversion process with computation to hide conversion latency.

The advantage of this approach: there is no need to pre-generate a complete offline weight set, saving storage space and offline conversion time, while ensuring that compute units always execute in FP8 format to maintain inference throughput.

5. The Role of the Compiler Toolchain in Architecture Migration

5.1 From Community Code to Chip Instructions

Community model code is typically written using general-purpose frameworks, and its operator implementations are not necessarily optimized for a specific chip architecture. The software stack team needs a complete compiler toolchain to progressively lower community implementations:

1. High-level graph optimization: Identify computation graph patterns in the model and merge redundant operators.
2. Operator mapping: Replace framework-level operators with chip-optimized operator library implementations.
3. Instruction generation: Compile operator implementations into chip instruction streams via the compiler.
4. Memory planning: Complete tensor memory allocation and reuse at compile time.

5.2 Key Tool Components

In this adaptation effort, the software stack leveraged the following tool components:

• Operator library: Provides base operators such as sparse attention, matrix multiplication, and normalization.
• Compiler frontend: Supports compiling Python-level operator descriptions into intermediate representations.
• Compiler backend: Maps intermediate representations to the chip instruction set, performing register allocation and instruction scheduling.
• Tensor compiler: Generates highly optimized kernel code for specific computation patterns such as sparse attention.

Together, these tools form the bridge from “community model code” to “actual inference service on the chip.”

6. Distilling Reusable Adaptation Principles

Reviewing the entire Day-0 adaptation process, four reusable engineering principles emerge:

Principle 1: Prioritize an end-to-end pipeline before point optimizations. First ensure the model can load, the service can start, and basic inference works. Then pursue deep optimization of hot paths such as sparse attention and expert computation.

Principle 2: Advance operator adaptation and compiler optimization in parallel. The operator library provides functionally correct compute units, while the compiler maps those units efficiently to hardware. Both are indispensable.

Principle 3: Precision conversion strategies must be designed in coordination with the storage hierarchy. The feasibility of online conversion depends on workspace size and conversion latency, which must be planned together with the chip’s storage hierarchy and compute pipeline.

Principle 4: Build a reusable adaptation checklist. Codifying model loading, operator registration, memory layout, precision conversion, and parallelization strategy into checklist items can significantly shorten the adaptation cycle for subsequent models.

7. Conclusion

Day-0 adaptation for large language models is a systems engineering effort spanning model parsing, operator development, compiler optimization, memory management, and precision conversion. The introduction of new features such as sparse attention and mixed-precision expert weights further raises the technical bar. Through operator integration, layout optimization, and online conversion strategies at the software stack level, domestic GPUs are progressively building the capability for rapid adaptation to new-architecture large models. For developers, understanding the key nodes and engineering trade-offs along this pipeline can help deliver more efficient model deployment and performance tuning in their own business scenarios.