How to Achieve Day-0 Adaptation for Trillion-Scale Sparse Models on Domestic GPU Software Stacks

1. Background: Deployment Challenges of Ultra-Large Models

Current Large Language Models (LLMs) have entered the trillion-parameter threshold, while context windows have expanded from a few thousand tokens to the million-token level. This trend poses three major challenges for underlying compute platforms:

• Memory Capacity and Bandwidth Pressure: Even with low-bit quantization, trillion-level parameters result in massive weight sizes. Furthermore, the KV cache overhead introduced by million-level contexts grows linearly with sequence length.
• Computational Path Complexity: To control attention computation costs, sparse attention mechanisms have been introduced. To compress expert weights, mixed-precision quantization schemes are widely adopted. These new architectures prevent the direct reuse of inference code designed for traditional dense models.
• Adaptation Timeliness: Users expect to complete deployment verification shortly after model weights are publicly released. Day-0 adaptation capability has become a key metric for evaluating the maturity of a compute platform’s software stack.

This article references the deployment practice of a trillion-parameter sparse model to discuss the engineering methods used in domestic GPU software stacks across three levels: sparse attention, low-bit expert weights, and the inference service pipeline.

2. Overall Strategy: From Community Model Code to Usable Inference Services

Migrating upstream community model code to a specific hardware platform typically requires the following tasks:

1. Model Loading Pipeline: Weight format parsing, shard loading, and memory allocation strategies.
2. Service Framework Integration: Integration with the inference engine’s scheduler, batching strategies, and request lifecycle management.
3. Key Computation Path Migration: Hardware adaptation for core operators such as attention, expert computation, and state management.
4. Accuracy and Performance Verification: Correctness validation for text and multimodal base inference pipelines.

A viable engineering path is to leverage the hardware adaptation branch of a mature inference framework to inherit the upstream model implementation. Subsequently, custom operator libraries and compiler toolchains are used to migrate and integrate key computational paths of the new architecture. This approach reuses the logical correctness of community models while achieving usable execution efficiency on the target hardware.

3. Key Points for Sparse Attention Adaptation

3.1 Computational Characteristics of Sparse Attention

Sparse attention is not simply about “skipping part of the computation.” Taking a common dynamic sparse scheme as an example, its core workflow includes:

• Index Selection: Dynamically selecting the set of key positions to participate in the computation for each query position, based on the relevance between query vectors and key vectors.
• Sparse Computation: Executing attention scoring and weighted summation only on the selected key positions.
• KV Cache Management: The selected key-value pairs must be organized in a specific layout to support efficient Gather/Scatter operations.

This means the adaptation work must cover index selection, KV cache layout, and both the Prefill and Decode computational paths.

3.2 Index Selection and KV Cache Layout

The core of the index selection phase is a Top-K or threshold filtering operation, whose output determines the memory access pattern of subsequent sparse computations. During hardware adaptation, attention should be paid to:

• Index Tensor Memory Layout: Indices typically exist as integer tensors. Their layout directly impacts the merged memory access efficiency of subsequent Gather operations.
• Physical Layout of KV Cache: Under sparse attention, the KV cache is no longer accessed as a continuous sequence but rather randomly accessed via index sets. Adopting a paged or blocked layout, combined with index sorting, can improve cache hit rates.

A simplified logic for index selection is shown below:

# Pseudocode: Sparse index selection and cache access
def sparse_attention(query, key_cache, value_cache, topk):
    # 1. Compute relevance scores between query and keys
    scores = compute_relevance(query, key_cache)
    # 2. Select topk key position indices
    indices = topk_select(scores, k=topk)  # [batch, heads, topk]
    # 3. Gather key-value pairs based on indices
    selected_k = gather(key_cache, indices)
    selected_v = gather(value_cache, indices)
    # 4. Execute attention computation on the sparse set
    attn_out = dense_attention(query, selected_k, selected_v)
    return attn_out

3.3 Differentiated Processing for Prefill and Decode

The Prefill phase features long sequences and is compute-intensive; the benefit of sparse attention here lies primarily in reducing invalid computations. The Decode phase has a sequence length of 1 and is memory-access intensive; the benefit of sparse attention here lies mainly in reducing the amount of KV cache read. The optimal operator implementations for these two paths differ and require separate adaptations:

• Prefill Path: Tends to fuse index selection with sparse computation to reduce memory round-trips of intermediate tensors.
• Decode Path: Tends to optimize Gather efficiency and co-schedule sparse attention with state management operators.

3.4 Coordination with Other Attention Mechanisms

In actual models, sparse attention rarely exists in isolation. It may form a complete execution pipeline alongside state management mechanisms like linear attention, cross-layer residual connections, and visual attention modules. During adaptation, it is necessary to ensure:

• Correct data dependency transmission between state management operators and sparse attention operators.
• Numerical stability of cross-layer residuals outside the main computational path.
• Consistency in the KV cache management logic shared between visual attention and text attention.

4. Online Conversion of Low-Bit Expert Weights

4.1 Problem Description

A portion of this model’s expert weights is stored in MXFP4 format. MXFP4 is a 4-bit micro-scaled floating-point format that significantly compresses weight volume while maintaining numerical representation capability. However, inference computation typically requires higher precision, such as FP8 or FP16.

The traditional approach is to pre-convert MXFP4 weights to FP8 weights offline and store them. However, this introduces two issues:

• Doubled Storage Overhead: Both the original and converted weights must be stored simultaneously.
• Extended Adaptation Cycle: Offline conversion requires additional engineering workflows and time.

4.2 Online Conversion and Workspace Reuse

A more efficient strategy is online conversion: completing the MXFP4 to FP8 conversion on-the-fly during model loading or inference, eliminating the need to pre-generate full offline FP8 weights. Its key designs include:

• On-Demand Conversion: Only converting the corresponding weight shards when an expert is activated, avoiding the startup latency of full conversion.
• Cross-Layer FP8 Workspace Reuse: Expert computations across different layers can reuse the same FP8 workspace memory, because only a subset of experts is active at any given moment. This significantly reduces peak memory usage.

The schematic flow is as follows:

# Pseudocode: Online weight conversion and workspace reuse
fp8_workspace = allocate_fp8_buffer(max_expert_size)
def expert_forward(x, expert_id):
    # 1. Load MXFP4 weight shard from storage
    mx_weight = load_mxfp4_weight(expert_id)
    # 2. Online convert to FP8, writing to the reused workspace
    fp8_weight = convert_mxfp4_to_fp8(mx_weight, out=fp8_workspace)
    # 3. Execute expert computation
    return expert_compute(x, fp8_weight)

The core benefit of this strategy is keeping memory usage within acceptable limits without adding offline conversion workflows, while ensuring the numerical precision of expert computation meets inference requirements.

5. Completing the Inference Service Pipeline

After adapting the key paths mentioned above, the modules must be integrated into a usable inference service. The complete pipeline includes:

1. Model Loading: Parsing weight file formats, loading shards into memory, and completing format recognition and metadata validation.
2. Service Startup: Initializing the inference engine, registering custom operators, and configuring batching and scheduling parameters.
3. Text Inference Pipeline: Input tokenization, embedding lookup, layer-by-layer forward computation, and output decoding.
4. Image Base Inference Pipeline: Vision encoder forward pass, cross-modal feature alignment, and text generation.
5. Health Check and Monitoring: Collection and alerting for metrics such as request queue status, memory usage, and per-step latency.

In this pipeline, the compiler toolchain is responsible for transforming high-level operator descriptions into hardware instructions. By providing hand-optimized core operators through a custom operator library and supporting automatic generation from Python-level operator descriptions to hardware code via the compiler, a balance between development efficiency and execution performance can be achieved.

6. Engineering Practice Recommendations

For teams looking to deploy ultra-large models on domestic compute platforms, the following points are worth noting:

• Prioritize Reusing Community Model Logic: Keep model structure definitions and weight naming conventions aligned with upstream as much as possible to reduce maintenance costs.
• Layered Operator Adaptation: First establish the dense path to ensure basic usability, then gradually replace it with sparse and low-bit optimized operators.
• Proactive Memory Planning: Estimate the peak memory for weights, KV cache, intermediate activations, and workspaces early in the adaptation phase to avoid late-stage rework.
• Regularized Accuracy Verification: Each adapted operator should have corresponding numerical comparison tests to ensure consistency with the reference implementation.
• Focus on End-to-End Latency Over Single Operator Performance: The benefits of sparse attention and expert computation should be measured across the complete service pipeline to avoid local optimizations causing global bottleneck shifts.

7. Conclusion

Day-0 adaptation for ultra-large sparse models is fundamentally a systems engineering challenge. It requires the software stack to collaborate across multiple levels, including model loading, computation path migration, memory management, and service framework integration. Index selection and KV cache layout for sparse attention, alongside online conversion and workspace reuse for low-bit expert weights, represent two highly technical components. Through a sound layered adaptation strategy and compiler toolchain support, domestic GPU software stacks are fully capable of hosting inference services for such models, providing a practical engineering path for scenarios involving million-level contexts and trillion-level parameters.