How Domestic AI Accelerator Cards Achieve Day-0 Adaptation for Trillion-Parameter MoE Models

1. Day-0 Adaptation: The “Time Race” in the Large Model Era

The iteration speed of the open-source large model ecosystem is continuously compressing the response window for compute platforms. “Day-0 adaptation” refers to completing model loading, inference service startup, and basic inference pipelines for text and images on the same day that model weights are officially released.

Recently, a multimodal MoE (Mixture of Experts) model with 3 trillion parameters and support for million-level context published its complete weights. A domestic AI accelerator card team immediately announced Day-0 adaptation based on its self-developed software stack. Since this model ranks among the top open-weight models in multiple public benchmarks and demonstrates outstanding capabilities for complex tasks and agent execution, the ability to support such frontier models immediately has become a litmus test for the maturity of domestic compute software stacks.

The value of Day-0 is straightforward: the time gap between model release and available compute directly determines whether downstream enterprises can conduct business validation within the competitive window. The prerequisite for achieving Day-0 is a software stack with general engineering capabilities that can “follow upstream architecture evolution”—not one-off customization for a single model, but a reusable adaptation methodology.

2. Where the Difficulty Lies: Four Challenges from the New Architecture

1. Ultra-large MoE structure. A 3-trillion-parameter model must reside entirely in memory (or be efficiently scheduled on demand). Although each inference pass activates only a small number of experts, routing computation, expert load balancing, multi-card memory partitioning, and interconnect bandwidth utilization are all hard requirements.

2. Dynamic sparse attention. Traditional attention computes similarity between every query and all KV entries; sparse attention first uses a lightweight index network to dynamically select a small number of key KV positions for each query, then computes attention only over those positions. The computation graph shifts from “regular large matrix operations” to “dynamic irregular memory access”—directly applying general implementations makes it difficult to meet either accuracy or performance targets.

3. Cross-layer state and residual design. The model introduces an incremental state management mechanism passed across layers and a cross-layer residual structure. Attention is no longer “stateless,” and data dependencies emerge between layers, breaking the typical inference engine assumption of “parallel within layers, no coupling between layers.”

4. MXFP4 low-precision expert weights. Expert weights are stored in MXFP4 (microscaling 4-bit floating point) format, grouped by blocks with shared scaling factors. If the hardware lacks a native 4-bit compute path, a precision conversion strategy must be designed without significantly increasing memory footprint or startup time.

3. Layered Software Stack: Turning Community Code into Hardware Services

The first principle of Day-0 adaptation is to maximize upstream ecosystem reuse rather than rewriting from scratch. A successful adaptation typically involves four collaborating layers:

• Inference framework layer: Mainstream open-source inference serving frameworks already provide request scheduling, continuous batching, multimodal preprocessing, and other general capabilities. The core adaptation work is to integrate upstream model implementations and connect the framework’s model execution path to the target hardware backend;
• Operator library layer: A high-performance fused operator library handles key computation paths such as GEMM, normalization, and sparse attention, replacing core computations in community model code with optimized implementations targeting the target hardware;
• Operator DSL layer: A domain-specific language for AI operators handles the large amount of kernel code in the community that targets general-purpose GPUs. Computation logic needs no rewriting—only the compilation target is redirected;
• Compiler layer: The compilation toolchain handles operator-to-target-instruction-set mapping, memory layout, and scheduling optimization, serving as the foundation for all upper-layer capabilities.

The combination of “framework integration + operator migration + compiler fallback” efficiently transforms community model code into actual inference services on accelerator cards—this is the prerequisite for turning Day-0 from a slogan into reality.

4. Sparse Attention Adaptation: Three Key Stages and a Complete Pipeline

(A) Index Selection: The First Gate for Accuracy and Performance

Index selection determines which KV entries participate in attention computation and is the foundation of the entire sparse scheme:

• Index computation must be implemented as operators running on the accelerator card. If index computation falls back to CPU, every layer and every step during Prefill/Decode triggers host-device interaction, and overall performance collapses immediately;
• Index correctness directly determines model accuracy. Missing key KV positions significantly degrades performance in long-text scenarios, so adaptation requires layer-by-layer comparison of index distributions against the reference implementation to ensure filtering behavior is consistent.

# Pseudocode: Core sparse attention flow
idx = index_net(q, kv_summary)        # ① Index selection: dynamically filter key KV positions
k, v = gather(kv_cache, idx)          # ② Gather data from rearranged cache by index
out = attention(q, k, v)              # ③ Compute attention only over selected KV

(B) KV Cache Layout: Redesigning Physical Storage for “Irregular Memory Access”

Sparse patterns naturally produce large amounts of gather-style random reads. With a contiguous layout, bandwidth utilization drops sharply. Adaptation work must rearrange the cache physical layout around sparse access characteristics: keeping KV entries hit by indices within contiguous memory segments as much as possible to reduce fragmented scattered reads; at the same time, million-level context requires the cache to support paged dynamic expansion to avoid memory waste from pre-allocating at the upper limit. The layout design must also be compatible with both Prefill and Decode access granularities—this is the real difficulty of this stage.

(C) Prefill and Decode Dual-Path Division

• The Prefill stage processes the entire input at once and is compute-intensive. After index filtering selects key segments, attention can be transformed into multiple small-scale dense computations; optimization focuses on improving GEMM throughput and operator fusion;
• The Decode stage generates token by token with only a few queries per step and is memory-intensive. Index and gather latency is amplified layer by layer; optimization shifts to memory access coalescing under small batches, cache hit rates, and operator launch overhead.

Both paths must be optimized separately while sharing the same index and cache semantics to guarantee consistent output.

(D) Pipeline Integration: Sparse Attention Cannot Be “Adapted in Isolation”

Sparse attention must ultimately form a complete execution pipeline together with cross-layer state management, cross-layer residuals, and the vision encoder’s attention. If any link is missing, long-text or image-text capabilities will fail. The completion criterion for this adaptation is precisely that both text and image inputs can traverse the full loop of “encoding → indexing → sparse computation → state update → decoding output.”

5. MXFP4 Expert Weights: Online Conversion + FP8 Workspace Cross-Layer Reuse

MXFP4 shares scaling factors per block, achieving extremely high storage efficiency, but it is not friendly to hardware lacking a native 4-bit compute path. A common approach is to convert to FP8 before computation; the question is when the conversion happens:

• Offline conversion: Pre-convert all expert weights to FP8 and persist to disk. Drawbacks are obvious—converting trillion-level parameters takes a long time; FP8 copies nearly double storage; and every weight update requires redoing the conversion;
• Online conversion: Convert only the expert weights hit by routing to FP8 when expert computation actually occurs, writing into an FP8 workspace. This workspace is reused across layers and does not retain a full FP8 copy.

# Pseudocode: Online conversion of MXFP4 expert weights with workspace reuse
ws = alloc_fp8_workspace()                        # Allocate FP8 workspace reusable across layers
for layer in model.layers:
    for expert_id in activated_experts(layer, x):  # Convert only routed experts
        w = mxfp4_to_fp8(weights[layer][expert_id], ws)
        out[routed_tokens] = fp8_gemm(x[routed_tokens], w)

This approach delivers three benefits: first, only one workspace is needed in memory rather than a full copy, which is especially critical for trillion-parameter models; second, it eliminates long conversion waits, significantly accelerating service cold start—naturally aligned with the Day-0 goal; third, MoE inference activates only a small number of experts per step, and “on-demand conversion” precisely matches the sparse activation characteristic. The cost is that conversion overhead is amortized into runtime, requiring workspace reuse and caching strategies to avoid repeated conversion—a classic design trade-off of trading compute for memory.

6. Validation Loop: Day-0 Acceptance Criteria and Future Evolution

Day-0 adaptation cannot stop at “it runs.” Complete acceptance should cover four dimensions:

• Functional pipeline: Model loading, service startup, and text and image inference are fully reproducible end to end;
• Accuracy alignment: Output consistency and perplexity comparison against the official reference implementation, with focus on validating numerical behavior of index selection, sparse attention, and MXFP4 conversion;
• Extreme scenarios: Stress validation for million-level context and mixed image-text inputs;
• Service stability: Throughput and stability observation under long-running continuous operation.

Beyond this, Day-0 addresses “usability,” while performance optimization (deep operator fusion, compilation scheduling, parallel strategies) is a long-term iterative effort.

7. Conclusion

The “time gap” from model release to compute availability is becoming a core metric for evaluating AI compute platform competitiveness. This Day-0 adaptation practice offers an answer: for frontier architectural features such as sparse attention and MXFP4, “layered software stack ecosystem integration + operator-level targeted adaptation + online precision conversion” constitutes a reusable methodology. For developers and enterprise users, understanding the four levers—index selection, KV cache layout, Prefill/Decode division, and workspace reuse—captures the main thread of heterogeneous large model adaptation and enables more confident planning of model deployment paths in today’s multi-compute landscape.