Day-0 Adaptation of Ultra-Large MoE Models: Engineering Stable Inference on Domestic AI Chips

1. Background: Deployment Challenges of a 3-Trillion-Parameter Model

As large model parameter scales enter the trillion-level, model architectural complexity has far surpassed that of dense models. Recently, a research team publicly released the complete weights for a 3-trillion-parameter MoE large model. This model supports a million-level context length and has undergone large-scale post-training and reinforcement learning, showing significant improvements in code generation, 3D content creation, financial research, knowledge QA, and long-range agent tasks. In public agent evaluation benchmarks, the model ranks at the top of the open-weight category, with outstanding performance in work, programming, and chat subcategories. On code and agent-related benchmarks, its long-range task scores increased by about 5 percentage points compared to the previous generation, reaching or approaching the level of top-tier closed-source models.

For compute platforms, the “Day-0 adaptation” of such models—completing weight loading, service startup, and the basic inference pipeline on the day of weight release—is a systematic engineering challenge. The challenges primarily stem from three areas:

1. Parameter Scale and Memory Pressure: Even with low-precision storage, a 3-trillion-parameter model imposes extreme requirements on memory capacity, weight sharding, and loading efficiency.
2. Architectural Novelty: The model employs sparse attention, a Mixture-of-Experts (MoE) architecture, and low-bit expert weight formats. General inference frameworks often lack the corresponding high-performance operator implementations.
3. Multimodal Pipeline: Text and image inference coexist, requiring separate adaptation for the attention computation path between the vision encoder and the language backbone.

This article outlines the complete technical path for the Day-0 adaptation of an ultra-large model using a typical domestic AI training and inference chip. It focuses on engineering methods for framework compatibility, migration of key computational paths, low-bit weight processing, and multi-card communication.

2. Overall Adaptation Strategy: Reusing Mature Interfaces, Optimizing New Paths

Faced with an ultra-large model with a novel architecture, writing an inference backend from scratch is neither realistic nor necessary. A sound engineering strategy is to use the mature upstream model implementation as a baseline, leverage the compatibility layer of the software stack to host the model code, and then map the new computational paths to the target hardware one by one.

Specifically, the adaptation work can be divided into four layers:

• Framework Layer: The inference framework handles model structure parsing, scheduling, and batching. The adaptation focus is enabling the upstream model implementation to run directly on the target software stack.
• Operator Layer: Integrate key operators such as sparse attention, expert computation, and vision attention into the target hardware’s high-performance operator library.
• Compilation Layer: Use graph and operator compilation tools to transform the new architecture’s computation graph into device-executable code.
• Runtime Layer: Cover device runtime, memory management, and multi-card communication to ensure the large model runs stably under tensor parallelism and expert parallelism.

The value of this layered strategy lies in the fact that most general logic (such as scheduling, sampling, and KV cache management) can reuse mature interfaces. Engineering effort is concentrated on the genuinely “new” computational paths, thereby compressing the adaptation cycle to a matter of days.

3. Migration and Adaptation of Key Computational Paths

3.1 Sparse Attention: Index Selection and KV Cache Layout

The sparse attention mechanism (DSA) used by this model is core to its long-context capability. Compared to standard attention, sparse attention requires index selection before computation—filtering the key-value positions most relevant to the current query from the historical context and computing attention only on the selected parts. This imposes two additional requirements on the inference system:

1. Efficient implementation of the index selection algorithm itself: If index scoring and Top-K filtering are placed on general operators, they become a bottleneck during the prefill phase. Therefore, a dedicated sparse attention operator is needed to fuse index selection with attention computation.
2. Redesigning the KV cache layout: Under sparse access patterns, the KV cache is no longer a simple continuous append but must support discrete reads by index. The blocking granularity and alignment of the cache will affect memory access efficiency.

During adaptation, the team mapped the index selection, KV cache layout, and the attention computation paths for both Prefill and Decode phases separately, integrating the corresponding high-performance sparse attention operators. Furthermore, sparse attention is not an isolated module; it must form a complete execution chain with state management mechanisms (for maintaining intermediate states across layers and time steps), cross-layer residual connections, and vision attention. Any missing link will cause output deviation, so every operator in the chain must be individually verified for numerical correctness.

3.2 Low-Bit Expert Weights: Online Conversion and Workspace Reuse

The model’s expert layer uses the MXFP4 format for weight storage. MXFP4 is a block floating-point low-bit format that maintains numerical stability while keeping storage overhead low. However, general inference stacks usually do not directly support matrix multiplication for this format. A common practice is to pre-generate FP8 weight copies offline, but this introduces two issues: first, weight conversion is time-consuming and requires extra storage space; second, in a Day-0 scenario, offline conversion becomes a hard bottleneck for launch time.

A more efficient solution is online conversion + workspace reuse:

• Before expert computation, dynamically convert MXFP4 weights to FP8 at runtime.
• The converted FP8 weights are not written to disk or made persistent but are written to an FP8 workspace reused across layers.
• Because different experts are activated in each layer of the MoE model, the workspace is managed in a “serial within layers, reused across layers” manner. The memory overhead is only related to the maximum expert scale of a single layer, rather than the sum of all experts.

This solution avoids generating complete offline FP8 weight copies, significantly lowering the deployment threshold for ultra-large models. The tradeoff is that conversion must be executed during every forward pass, so the efficiency of the conversion kernel directly impacts inference throughput and requires sufficient pipelining overlap with the expert matrix multiplication.

3.3 Vision Inference Pipeline

Multimodal capabilities require the model to integrate a vision encoder alongside the language backbone. The computation pattern of vision attention differs from the language side: the attention of image patch sequences is often denser and requires feature alignment with text tokens at specific layers. During adaptation, vision attention was integrated into the operator library as an independent computational path and merged with the language-side sparse attention at cross-layer residuals, ensuring numerical consistency across the entire pipeline under multimodal input.

4. Software Stack Compatibility: Infrastructure for Day-0 Adaptation

Achieving adaptation on the day of weight release requires a software stack with sufficient compatibility. From an engineering perspective, the following three capabilities are most critical:

(1) Compatibility of Programming Models and Runtime Interfaces

Upstream implementations of inference frameworks are typically written based on mainstream programming models. If the software stack can provide compatible mappings at the runtime interface level, the model code can be launched on the target device without massive rewrites. This requires the runtime to maintain consistency with upstream baselines in basic semantics such as stream management, event synchronization, and memory allocation.

(2) Integration of AI Frameworks and Operator Libraries

The adaptation team used the hardware backend of an open-source inference framework to host the upstream model implementation. They then integrated the new architecture’s key computational paths via a high-performance operator library, a Triton-like operator programming tool, and a graph compiler. The key here is “minimal modification”: model structure definitions and scheduling logic remain as close to upstream as possible, replacing only the operator implementations.

(3) Coverage of Compilation and Multi-Card Communication

Ultra-large models inevitably rely on multi-card parallel deployment. The software stack must simultaneously cover operator compilation (compiling new operators into device code), device runtime (kernel launch, synchronization), and multi-card communication (collective communication, All-to-All for expert parallelism). With all three in place, model adaptation can progress from “running on a single card” to “stable multi-card service.”

5. Engineering Verification and Launch Checklist

The completion criteria for Day-0 adaptation are not merely “being able to start,” but rather “being able to stably provide text and image inference services.” In practice, it is recommended to verify item by item according to the following checklist:

1. Weight Loading: Integrity check for shard loading; verify peak memory is within budget.
2. Numerical Correctness: Compare sparse attention index selection results layer-by-layer against the reference implementation.
3. Low-Bit Path: Ensure the error of the MXFP4->FP8 online conversion is within an acceptable range.
4. Multimodal Pipeline: Verify feature alignment between vision attention and the language backbone under image input.
5. Long Context: Stress test KV cache capacity and memory access bandwidth under million-level context.
6. Multi-Card Scaling: Evaluate communication overhead and load balancing under expert parallelism.
7. Servitization: Verify batch scheduling, time-to-first-token, and throughput under concurrent requests.

Among these, numerical correctness verification is the most easily underestimated. The index selection in sparse attention is discrete; a single-position index deviation can cause significant output drift. Therefore, layer-by-layer comparison is necessary rather than just looking at the final perplexity.

6. Summary and Future Outlook

The Day-0 adaptation of ultra-large models is essentially the process of decomposing a “novel architecture” into “known paths + new paths”: reusing mature interfaces for known paths and focusing optimization efforts on new paths (sparse attention, low-bit expert computation, vision attention). Combined with methods like online weight conversion and workspace reuse to lower memory thresholds, it is possible to complete the full pipeline connection—from loading to multimodal inference—on the day the weights are released.

From a broader perspective, as MoE architectures, low-bit formats, and sparse computation become mainstream, the competitiveness of inference software stacks will increasingly depend on the “speed of integrating new operators” and the “efficiency of cross-layer resource reuse.” For enterprise users, choosing a compute platform with comprehensive compilation, operator, and communication capabilities is a prerequisite for the rapid deployment of ultra-large models. On the application side, the capability improvements of such models in code generation, long-range agents, and multimodal understanding are providing a new technical foundation for scenarios like intelligent programming and knowledge QA.