Domestic GPU Achieves Day-0 Adaptation for a Next-Generation Ultra-Large Model: Engineering Practices for Sparse Attention and Low-Precision Expert Computation

1. Background: Deployment Challenges for Ultra-Large Models

As large model parameters enter the trillion-scale range, deployment faces a growing core contradiction: model capability improves with parameter size and context length, while the memory capacity, compute throughput, and interconnect bandwidth required for inference grow nonlinearly. Recently, a domestic AI training-inference accelerator, together with its supporting software stack, completed Day-0 adaptation for a 3-trillion-parameter model, establishing working paths for model loading, service startup, and basic text and image inference. This effort provides a referenceable engineering paradigm for deploying ultra-large models on domestic compute platforms.

The model supports a 1-million-token context and adopts a sparse attention mechanism with low-precision expert weights. It ranks among the top open-weight models on agent benchmarks and ranks first among open models in the work, coding, and chat categories. Its terminal task benchmark score is 65.7%, and its long-context code task score is 61.2%. These metrics place stringent demands on operator coverage, memory management efficiency, and compute path adaptation of the underlying compute platform.

This article focuses on the key technical paths in this adaptation, analyzing the adaptation methods for sparse attention, the online conversion strategy for low-precision expert weights, and how the software stack transforms community model code into a practical inference service.

2. Overall Software Stack Architecture and Adaptation Strategy

2.1 Layered Handling: From Model Code to Hardware Execution

Day-0 adaptation of a large-scale model is essentially the process of rapidly mapping an upstream model implementation onto target hardware. This adaptation adopted a layered handling strategy:

• Service layer: Built on an inference framework to host the upstream model implementation, responsible for request scheduling, batching, and streaming output.
• Operator layer: Uses an in-house operator library, a general-purpose operator compilation tool, and domain-specific languages to migrate and integrate the key compute paths of new architectures.
• Compilation layer: Uses a compiler to transform high-level operator descriptions into hardware-executable code, handling memory allocation, instruction scheduling, and pipeline optimization.

The value of this layered design is that when a model introduces new structures such as sparse attention, new normalization variants, or modified residual connections, only the operator and compilation layers need corresponding implementations. The service layer can complete adaptation without major changes.

2.2 What Day-0 Adaptation Means

Day-0 adaptation means the inference path can be established on the same day model weights are released. This requires three capabilities from the software stack:

1. Rapid operator generation: The ability to quickly generate efficient implementations for newly introduced compute patterns through a general-purpose operator compilation tool.
2. Flexible memory management: Support for dynamic memory allocation and reuse to handle KV cache expansion from ultra-long contexts.
3. Composable compute paths: Operators that can be flexibly combined into complete Prefill and Decode execution pipelines.

3. Sparse Attention Adaptation in Practice

3.1 Compute Characteristics of Sparse Attention

The sparse attention mechanism used by this model differs from standard dense attention: it does not compute attention weights over all historical tokens. Instead, it uses an index selection mechanism to identify the most relevant subset of tokens for the current query. This design reduces attention computational complexity from quadratic growth with sequence length to approximately linear growth, which is key to supporting a 1-million-token context.

However, sparse attention imposes new requirements on hardware adaptation:

• Index selection: Efficiently computing relevance between queries and historical tokens and selecting the Top-K indices.
• KV cache layout: Key-value pairs of selected tokens need to be accessed in a contiguous or semi-contiguous manner to improve cache hit rates.
• Variable-length computation: Different queries may select different numbers of tokens, causing the compute graph to change dynamically.

3.2 Index Selection and KV Cache Layout Optimization

During adaptation, the software stack completed index selection, KV cache layout, and Prefill and Decode compute path integration. Specifically:

For the index selection stage, a block-based computation strategy was adopted: historical tokens are divided into fixed-size blocks, block-level relevance scores are computed first, and then token-level filtering is performed within the highest-scoring blocks. This two-level filtering reduces index computation overhead while maintaining recall.

For KV cache layout, the cache structure was reorganized for sparse access patterns. Traditional dense attention stores KV cache contiguously in sequence order, while sparse attention requires dynamic gathering based on indices. The adaptation uses paged cache management, dividing the KV cache into fixed-size pages. Index selection results are mapped to page number lists, and pages are loaded during computation. This design supports both non-contiguous access and memory reuse.

Prefill and Decode paths were optimized separately: the Prefill stage has longer sequences and emphasizes index selection parallelism and KV write bandwidth utilization, while the Decode stage grows the sequence incrementally and emphasizes single-token inference latency and incremental cache update efficiency.

3.3 Composition of the Complete Execution Pipeline

Sparse attention is not an isolated operator. It works together with state management, cross-layer residuals, and vision attention to form a complete execution pipeline:

• State management: Maintains cross-layer consistency of sparse indices to avoid redundant index computation at each layer.
• Cross-layer residuals: Handles gradient and numerical stability of residual connections under the sparse computation graph.
• Vision attention: For image inputs, the vision encoder also uses attention mechanisms and needs to coordinate scheduling with text-side sparse attention.

The software stack integrates an in-house sparse attention operator to connect these components into an end-to-end executable pipeline, ensuring that basic text and image inference both run correctly.

4. Online Conversion of Low-Precision Expert Weights

4.1 Storage Challenges of MXFP4 Expert Weights

The model’s expert layers store weights in MXFP4 format. MXFP4 is a 4-bit floating-point format that significantly reduces memory usage compared to FP8 or FP16, making it possible to load a 3-trillion-parameter model within limited memory. However, most hardware compute units natively support FP8 or FP16 operations, so executing MXFP4 computation directly requires additional format conversion.

Generating a complete offline FP8 weight copy before inference creates two problems: first, conversion time delays the Day-0 adaptation window; second, memory usage doubles, negating the advantage of low-precision storage.

4.2 Online Conversion and Cross-Layer Reuse Strategy

The adaptation team used online conversion with cross-layer reuse of an FP8 workspace to execute expert computation, eliminating the need to pre-generate a complete offline FP8 weight copy. The core approach is:

• On-demand conversion: Before expert computation, the currently needed MXFP4 weight blocks are converted to FP8 in real time and written into a pre-allocated FP8 workspace.
• Cross-layer reuse: Different layers or experts share the same FP8 workspace. The workspace size is allocated based on peak single-computation demand rather than total model weight size.
• Pipeline overlap: Conversion and computation are pipelined so that the next weight block is converted while the current block is being computed, hiding conversion latency.

This strategy balances memory usage and compute efficiency: the memory increase is only the size of one workspace, far smaller than a full FP8 weight copy, and conversion overhead is hidden through pipelining, keeping end-to-end latency impact manageable.

5. Engineering Practices and Best Practices

5.1 Validation Methods for Operator Integration

When converting community model code into a practical inference service, a layer-by-layer validation strategy is recommended:

1. Unit testing: For each newly integrated operator, construct small-scale inputs and compare numerical accuracy against a reference implementation.
2. Inter-layer validation: Combine several layers and verify consistency of intermediate activations.
3. End-to-end validation: Run full model inference and compare output token sequences against the reference implementation.

5.2 General Principles for Memory Management

In ultra-large model deployment, memory management follows these principles:

• Peak-first: Allocate workspaces based on single-computation peak demand rather than total weight size.
• Reuse-first: Buffers with non-overlapping lifetimes share the same memory.
• Paged management: Large caches use paging mechanisms to support non-contiguous access and dynamic scaling.

5.3 Composable Design of Compute Paths

To support rapid adaptation to new model structures, compute paths should adopt a composable design:

# Illustrative pseudo-code: compute path composition
class ExecutionGraph:
    def add_node(self, op, inputs):
        # Register an operator node
        ...
    def compile(self):
        # Compile into hardware-executable code
        ...
    def run(self, inputs):
        # Execute and return results
        ...
# When adapting a new model, register new operators and compose them
graph = ExecutionGraph()
graph.add_node(SparseAttentionOp, [query, key_cache, value_cache, index])
graph.add_node(StateManagementOp, [sparse_output])
graph.add_node(ResidualOp, [state_output, residual_input])

This design transforms adaptation of new model structures from modifying the execution engine into registering new operators and recomposing them, significantly shortening the Day-0 adaptation cycle.

6. Summary and Outlook

This adaptation demonstrates that a domestic AI training-inference accelerator with a mature software stack is capable of supporting a 3-trillion-parameter model with a 1-million-token context. The key technical paths include index selection and KV cache layout optimization for sparse attention, online conversion and cross-layer reuse of low-precision expert weights, and rapid adoption of community model code through a layered software stack.

From a broader perspective, deployment of ultra-large models is shifting from hardware stacking to hardware-software co-design. As model architectures continue to evolve with sparser attention, more aggressive low-precision formats, and more complex expert routing, the software stack’s operator generation capability, memory management flexibility, and compute path composability will become the core competitiveness of compute platforms. For developers, understanding these underlying adaptation methods helps in making more informed technical decisions during model deployment and performance optimization.