Achieving Day-0 Adaptation for Trillion-Parameter Models: A Software Stack Engineering Approach

The iteration cycle of large language models is measured in days, whereas the traditional hardware adaptation cycle is measured in months. In the industry, “Day-0 adaptation” refers to the ability to load, launch a service, and run basic text and image inference on an AI accelerator on the exact day a new model’s complete weights are publicly released. Recently, a domestic training-inference integrated AI accelerator card and its accompanying software stack announced the successful Day-0 adaptation for a 3-trillion-parameter MoE model with a 1-million-token context. They publicly shared their solutions for two critical challenges: sparse attention and low-precision expert weights. This model ranks among the top open-weight models in multiple third-party evaluations for agents, coding, and long-context tasks, placing extremely high demands on computing platform compatibility. This article uses this adaptation as a case study to deconstruct the general engineering methods behind it.

1. The Challenges of Day-0 Adaptation: Three Concurrent Variables

New variables on the model side. The latest generation of models introduces novel architectural components: index-selective sparse attention, gated incremental state management (a mechanism similar to linear attention), cross-layer residual connections, and vision attention branches. Most of these components lack off-the-shelf kernel implementations, and the reference implementations in community inference frameworks often mismatch the execution models of new hardware.

Scale-side pressure. In a 3-trillion-parameter MoE model, expert weights occupy the vast majority of storage. Meanwhile, a 1-million-token context causes the KV cache and attention computation overhead to expand dramatically. Loading, VRAM scheduling, and memory bandwidth are pushed to their absolute hardware limits.

New precision format requirements. The model uses MXFP4 (microscaling block-quantized 4-bit floating-point format) to store expert weights. This requires the software stack to form a complete closed loop for quantization format parsing, online conversion, and numerical precision preservation.

Crucially, Day-0 adaptation is not just about “running a simple example.” It requires打通ing the complete chain of “model loading → service startup → text inference → image inference,” with clear acceptance criteria for every stage.

2. Software Stack Layering: “Translating” New Models to New Hardware

The adaptation team’s publicly shared path reflects a classic decoupled layering approach, divided roughly into four layers from top to bottom:

1. Inference Service Framework Layer: Directly interfaces with upstream model implementations, ensuring model logic and scheduling behavior align with community versions to avoid introducing extra deviations where “adaptation equals rewriting.”
2. High-Performance Operator Library Layer: Hotspot computations like attention, MoE, and normalization are handled by optimized fused operators.
3. Kernel Development Language Layer: Uses a Python DSL compiler for GPU programming and a tile-level programming language to quickly write and validate long-tail operators.
4. Compiler Toolchain Layer: Responsible for graph optimization, memory planning, and instruction generation, compiling high-level descriptions into efficient execution code on the accelerator card.

The value of this layering is “modifying locally without affecting globally.” When a new model arrives, engineering effort is concentrated on genuinely new computation paths, while the rest of the chain is reused. The team’s approach was to first run the framework’s native implementation to ensure logical correctness, then progressively replace critical computation paths with high-performance operators, ultimately transforming community model code into a practically usable inference service on the accelerator card.

3. Sparse Attention Adaptation: Indexing, KV Layout, and Dual Paths

Why sparsification is necessary. A 1-million-token context means standard attention computation grows quadratically with sequence length, and the KV cache expands linearly. Full computation is unaffordable in terms of cost. The core idea of index-selective sparse attention is to perform an “index selection” for each query first, allowing only a few of the most relevant tokens to participate in attention computation, compressing computation and memory access from “full” to “critical subset.”

Three key adaptation hurdles:

• Index Selection: Selecting top-k relevant tokens from ultra-long sequences involves sorting and sampling kernels. The index results must be strictly aligned with the main computation; an off-by-one index will completely distort the output.
• KV Cache Layout: Sparse reading requires the KV cache to be organized in a way directly addressable by the index (paged or blocked layout). The layout design directly determines memory access efficiency and is the lifeline of sparse attention performance on custom hardware.
• Prefill and Decode Dual Paths: The Prefill phase handles one-time encoding of long sequences, while the Decode phase handles sparse reading by index during step-by-step generation. The memory access patterns of these two phases are entirely different, requiring independent kernel implementations.

Complete execution graph. Sparse attention is not an isolated module; it must form a complete execution graph alongside gated incremental state management, cross-layer residual connections, and vision attention. Missing any link will cause an end-to-end failure. A sensible validation order is step-by-step alignment: index consistency → attention output consistency → single-layer output consistency → end-to-end output consistency. Locking only one variable at each step allows for quick localization to the specific operator if issues arise.

4. MXFP4 Expert Weights: Online Conversion and Workspace Reuse

The problem. In trillion-parameter models, expert weights dominate storage. Using the conventional approach of offline conversion to FP8 requires generating and storing a complete second copy of the weights, incurring high disk usage and conversion time costs. In a Day-0 scenario where “weights are just released,” the offline pipeline is unaffordable.

The solution. The loader directly reads the raw MXFP4 shards, converting them to FP8 on-demand during inference for expert computation. Simultaneously, the FP8 workspace is reused across layers. For each MoE layer, experts are only converted when activated. After a layer switch, the workspace is overwritten by the next layer. This reduces VRAM usage from “an entire copy of weights” to “a single-layer workspace.” The schematic pseudocode is as follows:

# Pseudocode: Online conversion and workspace reuse for MoE expert weights
workspace = alloc_fp8_buffer(size_of_single_layer)   # Workspace sized only for a single layer
for layer in model.layers:
    for eid in activated_experts(layer, tokens):      # Experts activated for this batch of tokens
        raw = load_mxfp4_shard(layer, eid)            # Directly read raw MXFP4 shards
        workspace[eid] = mxfp4_to_fp8(raw)            # Online conversion
        out[eid] = matmul_fp8(workspace[eid], x)
    # Upon entering the next layer, workspace is overwritten and reused, no need to hold all expert weights

Trade-off analysis. Online conversion introduces a small amount of extra computation overhead but eliminates the offline conversion pipeline and double storage. It is naturally compatible with scenarios where “weights are just released and there is no time for offline preprocessing.” This “runtime precision adaptation” approach is universally applicable and can be extended to support other low-precision formats.

5. End-to-End Verification Checklist

The final mile of Day-0 adaptation is service validation. A reference checklist includes:

• Model Loading: Shard integrity verification, VRAM mapping, and loading time baseline.
• Service Startup: Health checks, batching, and concurrency parameter baselines.
• Text Inference: Short-context regression use cases, combined with long-context sampling use cases.
• Image Inference: Multimodal input chain and vision branch output verification.
• Numerical Alignment: Comparing key intermediate variables with reference implementations, focusing primarily on sparse indexing and low-precision conversion.
• Performance Monitoring: Continuous tracking of metrics like Prefill/Decode latency, KV cache footprint, and expert activation hit rate.

6. Methodology Distillation: Making the Next Day-0 Faster

This adaptation case yields five reusable experiences: First, layered decoupling is the prerequisite for Day-0 adaptation. The framework layer remains consistent with upstream, while all differences are converged into the operator and compiler layers. Second, long-tail operators should be quickly bridged using DSLs, while hotspot operators are consolidated into the operator library, forming a “fast and slow combined” operator production system. Third, for new architectural components, adhere to “logic first, performance later” — ensure end-to-end correctness first, then optimize segment by segment. Fourth, runtime handling of low-precision weights should be consolidated into a general platform capability rather than a one-off breakthrough. Fifth, a comprehensive regression and evaluation system determines the speed of the second and third adaptations.

The essence of Day-0 adaptation is not a competition of single-point computing power, but the systematic maturity of the software stack. When layered decoupling, operator production, precision adaptation, and verification systems form a closed loop, “usable upon model release” will transition from a special case to the norm. This is the core of the next phase of competition for AI computing platforms.