Day-0 Adaptation of Ultra-Large Sparse Models on Domestic AI Accelerators: A Reasoning Pipeline Breakdown

I. Background: When Ultra-Long Contexts and Trillion Parameters Become the New Norm

The competition among large language models has shifted from simply pursuing parameter counts to a combined goal of “longer context + stronger reasoning + lower deployment costs.” Current industry models have expanded context windows to the million-token level while using sparse attention and Mixture of Experts (MoE) architectures to control computational volume. Although these models are more capable, they impose stricter requirements on inference infrastructure:

• A single card’s memory often cannot hold the complete model weights;
• The prefill stage for long contexts demands massive computation;
• Sparse attention requires novel indexing and caching strategies;
• If MoE weights use low-bit formats, they introduce additional format conversion and operator adaptation costs.

In this context, “Day-0 adaptation” has become a crucial metric for evaluating an inference platform’s engineering capabilities. Day-0 refers to completing the basic inference pipeline immediately after model weights are released, without waiting for official adaptation packages or relying on specific training frameworks. Instead, it involves quickly integrating open model implementations with a proprietary software stack.

This article focuses on a Day-0 inference adaptation practice for a model with million-level context and a three-trillion-parameter scale, breaking down its technical path and reusable engineering experiences.

II. Day-0 Adaptation Goals and Overall Architecture

Day-0 adaptation does not require achieving ultimate throughput or lowest latency upon launch. Instead, it prioritizes the following goals:

1. The model can be loaded correctly;
2. The inference service can start normally;
3. The inference pipelines for both text and image inputs are fully functional;
4. Output results align with the reference implementation.

A further engineering goal is to quickly migrate community implementations to the target accelerator without modifying the model’s structural semantics, maximizing the reuse of existing operator libraries and compiler capabilities.

The entire adaptation process is typically divided into five layers:

• Model Layer: Understand the model structure, identifying special designs such as long context, sparse attention, MoE, and vision branches.
• Framework Layer: Leverage existing inference frameworks to support the model implementation, completing compute graph construction and execution scheduling.
• Operator Layer: Supplement, fuse, and tune missing or suboptimal operators.
• Runtime Layer: Handle memory allocation, KV cache management, weight format conversion, and workspace reuse.
• Hardware Layer: Map operators to the target instruction set via the compiler, performing necessary memory access and concurrency optimizations.

During a real-world adaptation, the team utilized an existing unified programming model for AI training and inference cards. Relying on the upstream implementation of an open-source inference framework, they collaborated across operator libraries, multiple compilation channels, and the underlying compiler to quickly migrate the key computation paths of the new architecture.

III. Sparse Attention Adaptation: From Indexing to Computation Paths

The model employs the Deep Sparse Attention (DSA) mechanism to reduce the attention computation overhead of long contexts. Unlike traditional dense attention, sparse attention does not compute the dot product of all queries and keys. Instead, it first uses a sparse selection strategy to determine which positions need to participate in the computation, and then performs precise attention on local blocks or specific positions.

Key aspects of sparse attention adaptation include:

1. Index Selection

During the prefill or decode stage, sparse indices must be generated based on the relationship between the current token and historical tokens. This index defines which KV positions are visible to the current query. In practice, index selection can be based on positional structure or content relevance, and must remain consistent with the parameters in the model weights.

During adaptation, first ensure that the sparse selection logic perfectly matches the upstream implementation, then map it to tensor operators. For index generation processes that cannot be directly mapped, element-wise operators or small-scale custom operators can be used to avoid frequent switching between the CPU and the accelerator.

2. KV Cache Layout

In long-context scenarios, the KV cache becomes the primary consumer of memory. While sparse attention reduces computation, it increases cache management complexity. Storing the entire KV densely offers limited memory gains, while scattering storage based on sparse indices leads to non-contiguous memory access and dynamic allocation issues.

In practical adaptation, a common approach is to maintain a contiguous KV cache as the primary storage while recording valid positions in the index. During computation, the corresponding segments are fetched from the contiguous cache based on the index to form a temporary dense block for calculation. This maintains memory access friendliness and avoids complex dynamic storage management.

3. Prefill and Decode Computation Paths

The prefill stage processes a large number of input tokens at once, where the advantage of sparse attention is mainly reflected in reducing unnecessary KV comparisons. The decode stage processes only one new token at a time, requiring the selection of relevant parts from historical KV based on sparse indices.

Because the computation shapes of the two paths differ, they must be adapted separately. The prefill stage is better suited for large block matrix multiplication, while the decode stage requires specialized small-batch attention operators. In actual adaptation, the prefill path typically performs batch index filtering and then calls high-performance dense attention operators. The decode path connects to a dedicated sparse attention operator, reducing latency through grouping and pipelining.

IV. Online Conversion and Computation of Mixed-Precision Expert Weights

The model uses the MXFP4 format to store expert weights in the MoE layers. Low-bit weights effectively reduce storage requirements but introduce new challenges for inference: general matrix multiplication on hardware typically does not directly support FP4 computation, requiring conversion to higher precision or the use of mixed-precision computation.

1. Online Conversion Strategy

Pre-generating complete offline FP8 weights is feasible but incurs extra storage costs and lacks flexibility during the Day-0 phase where weights are updated frequently. In actual engineering, an “online conversion + cross-layer workspace reuse” approach is adopted:

• Keep the original FP4 format when loading expert weights;
• Dynamically convert the weights to FP8 or FP16 format before executing a specific expert’s computation;
• Release the conversion results after computation, retaining only the original FP4 weights.

This prevents memory bloat and ensures the inference service starts quickly.

2. Workspace Reuse

Expert layers typically contain multiple experts, and the weight conversion across different layers can reuse the same workspace memory. By analyzing the dependencies in the compute graph, the workspace’s lifecycle can be determined and allocated to different time segments in a streamed execution. While common in training frameworks, using this technique in inference pipelines requires more caution because inference services prioritize predictable latency and stable memory footprints.

In the actual implementation, the workspace size can be set to the maximum conversion size of a single expert’s weights. Synchronization and reuse mechanisms ensure that conversion tasks in different layers do not overwrite each other. It is crucial to explicitly manage the reuse strategy in the compute graph to prevent data errors caused by asynchronous execution.

3. Mixed-Precision Computation

The converted expert weights participate in matrix multiplication in FP8 format, while activations usually remain BF16 or FP16. Mixed-precision computation requires underlying operators to support a mixed accumulation mode for FP8 and FP16/BF16 inputs. For hardware lacking direct instructions, this can be achieved through splitting or software simulation, though performance will suffer. Therefore, the Day-0 phase focuses primarily on correctness, with subsequent performance optimization handled through operator fusion and quantization strategies.

V. Operators and Compilation Pipeline: Rapidly Landing Community Code

The critical path from community model implementation to an actual inference service on a target accelerator lies in operator mapping and automatic compilation capabilities.

1. Role of the Operator Library

The base operator library provides high-performance implementations for common tensor operations, including matrix multiplication, normalization, activation functions, convolution, and attention. During Day-0 adaptation, first verify whether the base operators used in the model structure are covered. For missing operators, fill the gap by combining existing operators or implementing temporary alternatives.

2. Compilation Channels

For model definitions in the Python ecosystem, multiple compilation paths usually exist. For example, Triton-style auto-tuning mechanisms can automatically map high-level tensor operations to underlying hardware. Domain-Specific Languages (DSLs) allow for more precise control over computation and memory access. The underlying compiler handles final instruction selection, register allocation, and memory transfer optimization.

During actual adaptation, different paths can be selected based on operator type:

• For compute-intensive operators like matrix multiplication, call pre-optimized implementations from the operator library;
• For lightweight operators like element-wise operations or custom activations, use automatic compilation channels to generate kernels;
• For complex fused operators like sparse attention, use DSLs for fine-grained control;
• Rely on the underlying compiler to complete final device code generation.

3. Typical Operator Example: Sparse Attention Kernel

Below is a simplified representation of the sparse attention computation process, illustrating the relationship between indexing, KV cache, and computation:

# Example: Selecting valid blocks from contiguous KV cache based on sparse indices
# selected_positions: [batch, num_heads, top_k]
# key_cache: [batch, num_heads, seq_len, head_dim]
selected_keys = gather_along_seq(key_cache, selected_positions)
selected_values = gather_along_seq(value_cache, selected_positions)
# Execute efficient attention on the selected dense blocks
attn_scores = matmul(query, selected_keys.transpose(-1, -2))
attn_scores = scale(attn_scores)
attn_weights = softmax(attn_scores)
output = matmul(attn_weights, selected_values)

In actual implementation, gather_along_seq can be completed using index operators, and then high-performance dense matrix multiplication can be directly reused. This design translates sparsity into an index selection and data movement problem, maximizing the utilization of existing computational capabilities while maintaining algorithmic flexibility.

VI. Multimodal and Cross-Layer Structure Adaptation

This model is not a pure text model; it also includes an image input pipeline. For the vision branch, processing typically involves image patching, vision encoder computation, and cross-modal feature fusion. Day-0 adaptation must ensure that the text and vision branches can work collaboratively on the same compute graph.

Cross-layer residual and state management are also critical. The model might pass state information between different layers or apply special handling to residual connections. Adaptation must ensure these structures remain consistent with the original implementation state, avoiding state passing sequence errors caused by compute graph optimizations.

VII. Engineering Insights from Day-0 Adaptation

Several valuable engineering experiences can be extracted from this practice:

First, connect the pipeline before optimizing performance. The core goal of Day-0 adaptation is correct execution, not extreme performance. Once the compute graph runs successfully, gradually locate bottleneck operators and perform targeted optimizations.

Second, prioritize reusing existing frameworks and operator libraries. In the open-source model ecosystem, many implementations already exist as generic framework code. Adaptation should not start from scratch but quickly inherit model structures through the upstream implementations of these frameworks.

Third, manage format conversion and memory reuse as independent issues. Low-bit weights are a future trend. Online conversion and workspace reuse are effective transitional solutions, especially during phases of rapid weight iteration.

Fourth, value the flexibility of the compilation pipeline. Different operators suit different compilation paths. A single automatic compilation path or pure manual optimization cannot cover all scenarios. Combining operator libraries, automatic compilation, and DSLs balances efficiency and controllability.

Fifth, maintain alignment with community implementations. Day-0 adaptation usually lacks official benchmarks, so it is necessary to build input-output alignment tests independently. Ensure that the computation results of each layer match the reference implementation to prevent errors from propagating deeply.

Through these practices, newly released models with million-level contexts and trillion parameters can be integrated into proprietary inference platforms in a short timeframe, laying a solid foundation for subsequent performance tuning and production deployment.