Engineering Day-0 Adaptation for Trillion-Parameter Sparse Attention LLMs

When open-weight large language models (LLMs) iterate on a monthly basis, the adaptation speed of inference infrastructure is shifting from a value-add to a critical necessity. Day-0 adaptation refers to the ability of a compute platform to complete model loading, inference service startup, and the connection of basic text and image inference pipelines on the exact day the model weights are released. Users can download the weights and run them on the same day.

Recently, a new generation of open-weight Mixture-of-Experts (MoE) models with up to 3 trillion parameters publicly released its complete weights. It supports a 1-million-token context and ranks at the top of open-weight models in public evaluations for agents and long-range programming. Concurrently with the release, compute software teams announced the completion of Day-0 adaptation for this model based on integrated training and inference accelerator cards and a self-developed software stack. For the engineers doing the adaptation, this model incorporates almost all the latest architectural “new components” found in current LLMs:

• A trillion-parameter MoE architecture with expert weights stored in the low-bit MXFP4 format.
• A sparse attention mechanism oriented towards ultra-long contexts, replacing traditional global dense attention.
• A linear attention variant alternately stacked with sparse attention, requiring the maintenance of state across time steps.
• Cross-layer shared residual designs, along with a visual encoding pipeline supporting image inputs.

Each “new component” implies new operators, new memory access patterns, and new precision risks. Using this adaptation as a case study, this article breaks down a reusable engineering path: how the software stack layers divide responsibilities, how sparse attention challenges are tackled, and how MXFP4 weights run with “zero redundancy.”

1. Overall Methodology: Upstream Integration and Layered Software Stack

Day-0 feasibility is never achieved through a last-minute sprint; it is the natural outcome of a layered software stack design. Three engineering decisions from this adaptation are worth noting.

1. Integrate upstream, avoid hard forking. The team directly adopted the model implementation from mainstream open-source inference frameworks in the community—model structure definitions, scheduling logic, and sampling strategies all reused upstream code, while they only maintained the hardware backend. The benefit is straightforward: whatever the model looks like in the framework, the backend takes over the corresponding compute graph. Fixes and optimizations from the community for the upstream code can be synced at a low cost, eliminating the need to rewrite model code for every new model. In essence, this compresses the workload of “adapting a new model” into “adapting a batch of new operators.”

2. Layered adaptation, clear division of labor. A typical layered structure contains four layers:

• High-performance operator library: Provides deeply optimized implementations of core operators like GEMM, dense/sparse attention, and quantization/dequantization. This is the foundation of performance.
• High-level operator DSL: Uses scripting languages to describe compute logic. New operators can be rapidly prototyped, with the compiler handling hardware details without requiring everything to be written in low-level hardware languages.
• Tile-level scheduling language: Suited for memory-intensive and dynamically shaped computations like attention, facilitating manual control over data movement and pipelining.
• Low-level compiler: Compiles the intermediate representation generated by the frontends into target hardware instructions and performs general scheduling optimizations.

The key compute paths of the new architecture are migrated through a combined approach: “using the operator library to cover common patterns, DSL to quickly fill in the increments, and the compiler to guarantee correctness and a performance baseline.”

3. Minimal closed-loop validation. The acceptance criteria for adaptation are not just “operators run successfully,” but a minimal viable pipeline: successful weight loading → normal inference service startup → normal text inference output → normal image understanding inference output. Only pipeline-level validation can expose inter-module integration issues, such as tensor layout inconsistencies between the visual encoder and the text backbone, which single-operator tests cannot detect.

2. Tackling Sparse Attention: Index Selection, Cache Layout, and Dual-Path Optimization

This is the most technically demanding part of the adaptation. First, a brief background on the mechanism: taking a 1-million-token context as an example, if traditional global dense attention were used, the compute complexity would grow quadratically with sequence length, and the memory footprint of the KV cache alone would be substantial. The core idea of sparse attention is a two-phase “filter then compute” approach. The first phase uses a lightweight, coarse-grained relevance scoring to select high-correlation fragments from historical KV. The second phase executes precise attention only on the selected fragments, bringing the compute and memory overhead of long contexts down to a manageable range.

At the software stack level, three key tasks must be completed.

1. Operatorization of index selection. Coarse-grained scoring is an entirely new compute pattern—aggregating KV by blocks, computing similarity with the query, and selecting the most relevant blocks. This step has two typical pitfalls. First, the numerical output must align with the reference implementation; if the index selects the wrong block, the error will be amplified by the precise attention in the second phase, directly degrading generation quality, and this is extremely difficult to trace from the final output. Second, this step is memory-intensive and highly parallel, making it suitable for manual optimization using a tile-level scheduling language. Relying solely on auto-compilation often fails to achieve ideal performance.

2. Redesigning the KV cache layout. The sparse access pattern is “index by block, gather and read.” The cache layout must support efficient gather operations and coordinate with the paged KV cache mechanism to avoid memory fragmentation. More complex is the fact that this model does not use pure sparse attention: the linear attention variant stacked alternately with it maintains a fixed-size state vector rather than a KV list that grows token by token. The software stack must establish unified lifecycle management for both “KV lists” and “state vectors.” Any state read/write desynchronization will cause precision issues that are exceptionally difficult to troubleshoot in long-context scenarios.

3. Separate optimization for Prefill and Decode dual paths. The bottlenecks for the two phases are entirely different. The Prefill phase has many queries and key-values, suited for a batch processing paradigm of “compute indices in batches first, then aggregate computation uniformly.” The Decode phase has only one query per step; the bottleneck lies in the read bandwidth of the sparse KV, requiring pipelining of data movement and computation. The two paths require kernel implementations of different forms and separate precision alignment. One set of code cannot handle both.

It must also be emphasized that sparse attention is not an isolated island: it must form a complete execution chain together with the state updates of linear attention, the writes of cross-layer shared residuals, and the attention computation in the visual encoder. All four are indispensable. Taking the Prefill path as an example, it can be simplified as follows:

# Sparse Attention Prefill Path (Pseudo-code)
index = block_score_and_select(q, kv_blocks)   # Phase 1: Coarse-grained index selection
k, v  = gather(kv_cache, index)                # Gather KV by selected blocks
out   = precise_attention(q, k, v)              # Phase 2: Precise attention
state = linear_attn_update(state, x)            # Linear attention state maintenance
x     = residual_stream_write(out, state)      # Cross-layer residual write

3. MXFP4 Expert Weights: Online Conversion and Cross-Layer Workspace Reuse

Another major feature of this model is that the MoE expert weights are stored in the MXFP4 format. MXFP4 is a block-quantized 4-bit floating-point format. In industry-standard specifications, weights are grouped into fixed-size element blocks (commonly 32 elements per block), sharing a scaling factor within the block (usually represented in FP8). This compresses the weight volume by half compared to FP8. Since an MoE model only activates a subset of experts per token, the inference throughput bottleneck is often on the memory access of expert weights, which low-bit storage is specifically designed to address. The trade-off, however, is that most computing hardware currently cannot natively execute FP4 arithmetic, necessitating format conversion first.

There are two route options for this:

• Offline conversion: After downloading the weights, dequantize all expert weights to FP8 and save them entirely to disk. The problem is that for a 3-trillion-parameter model, this needlessly creates an extra TB-level weight replica. The combined time for downloading, converting, and storing turns Day-0 into Day-N.
• Online conversion: Load the MXFP4 weights as they are, dequantize them block by block during expert computation, and discard them after use. The team ultimately adopted this route and made two key designs.

# Online Dequantization for Expert Computation (Pseudo-code)
for block in expert_weights[mxfp4]:      # Read MXFP4 weights block by block
    w8 = dequant(block, scale_fp8)        # Block-level dequantization to FP8
    ws.put(slot_of(block), w8)            # Write to shared FP8 workspace
    gemm_fp8(x, ws[slot_of(block)])      # Execute matrix multiplication in FP8 and accumulate

First, cross-layer reuse of the FP8 workspace: The computation of different layers and experts in MoE is naturally staggered in time. The same FP8 workspace can be reused repeatedly, reducing memory footprint from “number of experts × workspace size” to “concurrency × workspace size.” Second, pipelining conversion and computation: Block-level dequantization is a typical memory-access operation that can be pipelined with GEMM execution, hiding the conversion latency behind the computation.

The direct benefit of this route is that there is no need to pre-generate complete offline FP8 weights. With zero storage redundancy, the inference service can be started as soon as the weight download is complete. For the Day-0 goal, this is hardly an optimization item; it is a mandatory requirement.

4. Validation Checklist and Engineering Recommendations

For teams undertaking similar adaptation efforts, the following check items are worth incorporating into the plan early:

• Precision alignment: Beyond comparing intermediate results operator by operator, focus on confirming the block selection consistency of sparse indices—selection discrepancies are the number one source of long-context precision issues. Use perplexity and output comparisons for end-to-end validation.
• Performance baseline: Record Prefill throughput and Decode latency separately, and specifically test the memory footprint and long-term stability under a full 1-million-token context.
• Multimodal pipeline: Image inputs go through the visual encoder, whose attention structure differs from the text side. This requires separate operator coverage and precision validation; testing text alone is insufficient.
• Servitization capabilities: Concurrent scheduling, continuous batching, and memory recycling strategies should all be validated within the Day-0 pipeline, rather than as an afterthought.
• Cloud deployment: Weight files can be hosted on object storage, and inference services can be orchestrated using containers, paired with logging and monitoring alerts to observe service quality. If utilizing cloud platforms like Baidu AI Cloud, you can directly combine their general product capabilities—such as containers, object storage, and monitoring alerts—to further compress environment preparation time.

5. Conclusion

Looking back, the essence of Day-0 adaptation is the rapid following of platform capabilities to model innovation. Integrating upstream implementations reduces the migration cost on the model side; operator DSLs and compilers shorten the deployment cycle for new operators; and online quantization conversion bridges the gap between new storage formats and hardware capabilities. As LLM architectures continue to evolve on a monthly basis, this systematic adaptation capability is becoming the true moat for compute infrastructure.