Engineering Path for Day-0 Adaptation of Trillion-Scale Sparse Models on Domestic AI Accelerators

The rapid iteration of large model architectures is continuously widening the time gap between model release and hardware readiness. When a model with a trillion-level parameter scale, a context length exceeding a million tokens, and introduced sparse attention alongside low-precision expert weights is publicly released, whether an inference platform can complete the entire pipeline—from weight loading to service startup—on Day-0 has become a critical benchmark for measuring the engineering maturity of a software stack.

Based on a Day-0 adaptation practice of a 3-trillion-parameter model, this article focuses on three core issues: the migration of computation paths for sparse attention, the online quantization conversion of expert weights, and the localized implementation of upstream model code onto self-developed AI accelerators. We will break down the technical difficulties and corresponding engineering solutions.

1. Background: Why New Architecture Models Are Not “Plug-and-Play”

The difficulty of deploying trillion-parameter models for inference lies not only in total compute power but also in the matching degree between computation paths and hardware execution models. These models typically share several common characteristics:

• Ultra-long context: The context window reaches the million-token level. The computational overhead of attention grows non-linearly with sequence length, necessitating sparsification methods to control the compute volume.
• Sparse attention structure: An index selection mechanism computes attention only on a subset of historical tokens. The storage and access patterns of the KV cache are entirely different from dense attention.
• Low-precision expert weights: Mixture-of-Experts (MoE) architectures use low-bit formats like MXFP4 to store expert parameters to compress memory usage. However, hardware execution units typically require higher-precision data formats.
• Multimodal inference pipeline: Basic inference pipelines for text and images need to share the same execution backend. Visual attention and language attention require unified scheduling at the operator level.

This means that simply “getting the model to run” does not equate to “running the model correctly.” Adaptation work must go deep to the operator level, reorganizing data layouts and computation sequences to ensure the model runs both correctly and efficiently on the target hardware.

2. Overall Pipeline: From Upstream Implementation to Local Inference Service

The overall adaptation strategy can be summarized into four layers, from bottom to top: the compilation and operator layer, the computation graph migration layer, the serving framework layer, and the model integration layer.

┌─────────────────────────────────────────┐
│  Model Integration: Weight loading / Service startup / Multimodal pipeline  │
├─────────────────────────────────────────┤
│  Serving Framework: Request scheduling / Batching / KV cache management   │
├─────────────────────────────────────────┤
│  Graph Migration: Key path migration and integration for new architectures        │
├─────────────────────────────────────────┤
│  Compilation & Operator: Operator library / Domain compiler / Graph compilation     │
└─────────────────────────────────────────┘

At the operator and compilation layer, high-frequency basic operators are handled by a general operator library, while new architecture operators are generated and optimized using a domain-specific language for tensor computation and a graph compiler. At the computation graph migration layer, upstream community model code is mapped path-by-path to the local execution backend to ensure semantic consistency. At the serving framework layer, a mature serving framework is reused for request scheduling and cache management. At the model integration layer, end-to-end verification is performed for weight loading, service startup, and both text and image inference pipelines.

The value of this layered structure is that adapting a new model does not require rewriting from scratch. Instead, the “incremental changes brought by the new architecture” are localized to specific layers, replacing or extending them incrementally, thereby compressing the Day-0 adaptation engineering cycle.

3. Sparse Attention Adaptation: Indexing, Cache, and Computation Paths

Sparse attention is the most representative technical challenge in this adaptation. Its core idea is: for each query token, first use a lightweight index selection process to pick the most relevant tokens from the historical sequence, and compute attention only on these tokens. This mechanism directly alters the implementation of three stages.

3.1 Integration of the Index Selection Operator

Index selection is essentially a scoring and Top-K filtering process. Unlike standard attention operators, its output is not a continuous attention matrix but a discrete set of indices. Subsequent computations must perform non-contiguous memory accesses based on these indices.

During adaptation, two issues must be resolved: first, whether the scoring computation can reuse existing matrix multiplication operators to avoid introducing inefficient implementations solely for index selection; second, the parallelization strategy for Top-K filtering on large-scale candidate sets, which must match the hardware’s thread organization. In practice, index selection is split into three segments: “scoring — reduction — filtering.” The first two reuse general matrix computation and reduction operators, while the third is implemented as a custom operator, ensuring both correctness and acceptable throughput.

3.2 Non-contiguous Layout of the KV Cache

The KV cache for dense attention is usually allocated contiguously, arranged sequentially by sequence position. In sparse attention, however, the selected token indices change dynamically. If the cache is still organized in a contiguous layout, every computation requires additional address rearrangement.

The engineering approach is to change the KV cache to a blocked layout: managing the cache in fixed-size blocks. The index selection results map to block numbers and intra-block offsets, and data is loaded at the block granularity during computation. The trade-off is that cache management logic becomes more complex, but the benefit is that memory access during the attention computation phase is more regular, reducing bandwidth waste from random memory access. Meanwhile, the block granularity must align with the hardware’s cache line width to avoid read amplification.

3.3 Differentiated Paths for Prefill and Decode

The Prefill phase features long sequences and high parallelism. Index selection can be completed at once over a large candidate range, prioritizing throughput. The Decode phase processes only a few new tokens per step but requires repeated access to the historical cache. The candidate set for index selection grows with decoding steps, making memory access the bottleneck.

Therefore, the two paths must be adapted separately: the Prefill path focuses on batching index selection and operator fusion to reduce intermediate result disk writes; the Decode path focuses on the locality of cache access and lightweight state management. Implementing them separately rather than forcing a shared kernel is necessary to guarantee end-to-end performance.

3.4 Coordination with Other Structures

Sparse attention does not exist in isolation. The complete execution pipeline must also coordinate with state management of state-space modules, cross-layer residual connections, and visual attention. The state management module needs to maintain recursive states across steps; residual connections require each layer’s output to be consistent in precision and layout; visual attention brings tensor shapes of different dimensions. During adaptation, these types of computations are unified under a single set of operator interfaces and scheduled by the execution engine according to the graph, avoiding fragmentation where “some paths are on the accelerator while others fall back to general implementations.”

4. Execution Strategy for MXFP4 Expert Weights

In MoE architectures, expert weights are stored in MXFP4 format to accommodate more experts in memory. However, compute units typically execute in FP8 or higher precision. Directly using low-bit weights requires format conversion.

An intuitive approach is to pre-generate complete FP8 weight copies offline, but this introduces two problems: first, memory usage doubles, negating the benefits of low-bit storage; second, the weight file size increases, prolonging loading times.

A more reasonable strategy is online conversion + workspace reuse:

• Before expert computation, dynamically convert the expert weights hit by the current batch from MXFP4 to FP8, writing the results into a pre-allocated FP8 workspace.
• This workspace is reused cyclically at the expert granularity. Different layers and different token batches use the same memory block sequentially, avoiding keeping separate copies for each expert.
• When reusing across layers, the lifecycle management of the workspace must be correct—meaning a workspace can only be overwritten by the next layer after the previous layer’s expert computation is complete. This requires the execution engine to have explicit constraints on the scheduling order of expert computations.

The pseudocode for this process is as follows:

# Pseudocode: Online conversion and workspace reuse for expert weights
fp8_workspace = allocate_workspace(size=max_expert_size, dtype=fp8)
for layer in decoder_layers:
    for expert_id in layer.routed_experts(batch_tokens):
        # Online conversion: MXFP4 -> FP8, writing to the shared workspace
        convert_mxfp4_to_fp8(
            src=expert_id.weight,
            dst=fp8_workspace
        )
        # Execute expert forward pass using the workspace
        out = expert_forward(input=batch_tokens, weight=fp8_workspace)
        # The workspace can be overwritten by the next expert after this computation round

The key benefit of this strategy is that there is no need to pre-generate offline weights. The weight files maintain their low-bit footprint, keeping loading and storage overheads under control. Meanwhile, the FP8 workspace only occupies a fixed block of memory, making the overall memory budget predictable. The trade-off is the introduction of runtime conversion overhead, which needs to be hidden within the computation pipeline through operator fusion and scheduling overlaps.

5. Engineering Implementation and Verification Methods

The acceptance criteria for Day-0 adaptation is not “can it load” but “can it provide correct inference services.” Therefore, a layered verification method must be established in engineering practice:

1. Single-operator verification: For new operators like index selection, sparse attention, and expert conversion, use small-scale inputs and reference implementations for element-by-element comparison to ensure numerical consistency.
2. Single-layer verification: Using a single decoder layer as a unit, compare the complete forward output to check if the combination of state management, residual connections, and attention is correct.
3. End-to-end verification: Connect weight loading, service startup, and basic text and image inference pipelines. Compare output consistency using fixed prompts and observe cache behavior in long-context scenarios.
4. Performance baseline observation: Record throughput and latency for Prefill and Decode phases separately to locate whether the bottleneck falls in index selection, cache access, or expert conversion.

Furthermore, the batching strategy of the serving framework needs to align with the dynamic indexing characteristics of sparse attention: different requests have different index sets, and cache access patterns vary significantly during batching. Request grouping should be performed at the scheduling layer to prevent individual long-sequence requests from inflating overall latency.

6. Conclusion

The Day-0 adaptation of trillion-scale sparse models on domestic AI accelerators is essentially an engineering practice of “computation path remapping.” The technical key points can be summarized into three aspects: sparse attention requires systematic adaptation from index selection and cache layout to separate Prefill/Decode paths, rather than point-to-point operator replacement; low-bit expert weights should adopt the route of online conversion and workspace reuse, trading predictable memory usage for runtime overhead; and a layered verification method is a necessary means to ensure adaptation correctness.

As model architectures continue to evolve towards sparsity, low precision, and multimodality, the competitiveness of an AI software stack will increasingly depend on “whether it can transform model code into available inference services in the shortest possible time when a new architecture emerges.” Building this capability relies precisely on the collaborative evolution of operator libraries, compilers, and serving frameworks based on layered decoupling.