Day-0 Adaptation of a 3-Trillion-Parameter MoE Model on Chinese Domestic AI Accelerators: Technical Pathways Explained

As large language models evolve, parameter counts moving from the hundred-billion to the trillion scale have become a defining line for frontier models. Once a model reaches roughly three trillion parameters while supporting a million-token context window, the challenge for the inference system is no longer a single-dimension compute problem. It becomes a tangle of interwoven constraints: memory capacity, memory bandwidth, computation-graph scheduling, and precision management.

Recently, a Chinese AI compute vendor announced that, using its unified training-and-inference AI accelerator card and accompanying software stack, it had completed Day-0 adaptation of a 3-trillion-parameter MoE model, establishing model loading, service startup, and the basic text and image inference pipelines. The technical value of this work lies not in “getting it to run” itself, but in the complete engineering methodology it reveals for what it takes for a domestically developed compute platform to absorb frontier model architectures. This article analyzes the effort along three dimensions: the adaptation layers, the key technical pathways, and the engineering practice.

A note on terminology: in industry usage, Day-0 adaptation means the inference pipeline is working on the very day the model weights are publicly released. The difficulty is that the team cannot obtain the model’s structure and weight details in advance; only after the model is open-sourced can they quickly map operators and rework compute paths based on the released code. This demands a software stack with sufficient generality and extensibility.

The Four-Layer Structure of a Model Adaptation

From an engineering perspective, a complete large-model adaptation decomposes into four layers with clear responsibilities and dependencies:

Layer 1: Model ingestion. Handles weight loading, format parsing, and parameter mapping. At 3-trillion-parameter scale, the weight volume is enormous, so loading itself must account for sharding strategies, asynchronous loading, and memory planning.

Layer 2: Compute graph integration. The inference framework takes over the upstream model implementation, mapping the operator calls in the community model code to operators executable on the target hardware. This layer determines whether the model “runs at all.”

Layer 3: Key operator adaptation. Provides dedicated operator implementations for the model’s distinctive structures—attention mechanisms, state management, residual connections. This layer determines whether the model “runs correctly.”

Layer 4: Performance and precision optimization. Through low-precision quantization, workspace reuse, fused compute paths, and similar techniques, it raises throughput and reduces memory usage while preserving numerical correctness. This layer determines how “well” the model runs.

The adaptation effort described here made substantive investments at all four layers, each examined below.

Compute Path Migration: From Community Code to a Serving Stack

Inference Framework Support Backed by an Operator Library

Open-source model code is usually written against a general-purpose deep learning framework, and its operator semantics do not map directly onto the target hardware. The adaptation team used an open-source inference framework as the carrier, and completed the migration and integration of the new architecture’s key compute paths through a combination of: an in-house operator library, Triton implementations targeting the hardware, an operator description language built on TileLang, and an in-house compiler.

The point of this combination is a “multiple entry points, single exit” operator supply capability: existing general-purpose operators are provided directly by the operator library; new compute patterns emerging from the community that the library does not yet cover can be quickly written in Triton or TileLang and compiled to the target hardware; and the compiler handles scheduling and memory planning at a lower level. Working together, the three mean the team does not have to wait for the operator library to be filled in operator by operator when a brand-new architecture appears—a compute path can be brought up in a short time.

The Adaptation Workflow, Step by Step

Turning community model code into a real inference service on the target hardware roughly involves the following steps:

1. Model structure parsing — Read the model configuration files and weight index to pin down the layer count, hidden dimensions, number of experts, attention type, and other key parameters.
2. Operator mapping audit — Scan the model’s forward-pass code layer by layer, list every operator call, compare it against the existing operator library, and flag missing items.
3. Implementing missing operators — Classify missing operators by compute pattern; reuse existing operator combinations where possible, and write Triton/TileLang implementations where reuse is not possible.
4. Weight loading and layout conversion — Convert weights from their original format into a memory layout friendly to the target hardware, paying attention to sharding and alignment requirements.
5. Service packaging and validation — Wrap the model as an inference service, then perform layer-by-layer numerical alignment and end-to-end output validation.

Steps 3 and 4 typically concentrate most of the workload, and they are where the two key techniques discussed in the following sections come into play.

Adapting Sparse Attention: Indexing, Caching, and Compute Paths

What Makes This Sparse Attention Different

The sparse attention mechanism used by this model (commonly described in the industry as DSA-style dynamic sparse attention) rests on a simple idea: when processing long sequences, do not compute the full attention matrix. Instead, a lightweight index-selection step first filters out the most relevant key-value positions for each query position, and attention is computed only over those positions. At million-token context lengths, this significantly reduces compute and memory traffic—but it also introduces three new problems for the implementation:

• Index selection is itself a compute step. It requires a dedicated operator, and its output affects every downstream compute path.
• The KV cache layout is no longer contiguous. The selected positions are dynamic and non-contiguous, so cache management must support sparse, indexed access.
• Prefill and Decode have different sparse patterns. Each stage must be adapted separately.

Three Keys to the Adaptation

Against these problems, the adaptation team made three changes at the software stack level:

First, integrating the index-selection operator. The index-selection process was implemented as a dedicated sparse attention operator so that scoring and filtering run efficiently on the target hardware. Its output—a set of position indices—serves as the input to the subsequent attention computation.

Second, redesigning the KV cache layout. With dense attention, the KV cache is stored contiguously by sequence position; sparse attention needs random access by index. The adaptation adjusted the cache layout specifically so that the memory-access efficiency of sparse indexed access stays within an acceptable range.

Third, adapting the Prefill and Decode paths separately. In Prefill, sequences are long and parallelism is high, and the sparse pattern is block-like in distribution; in Decode, the sequence grows step by step, per-step compute is small, and the sparse pattern depends more heavily on historical indices. The two compute graphs were adapted separately, then combined with state management operators, cross-layer residual connections, and the vision attention module into one complete execution chain.

The phrase “complete execution chain” deserves emphasis: sparse attention does not exist in isolation. Together with state management (which maintains state information across sequence positions), cross-layer residuals (which affect numerical stability), and vision attention (the multimodal input path), it forms a whole. A missing link anywhere renders the end-to-end output unusable. Adaptation therefore has to be validated as a chain, not operator by operator in isolation.

Online Conversion of Low-Precision Expert Weights

The Problem: Expert Weights Cannot Be Pre-Quantized Offline

The model stores its expert weights in a low-precision floating-point format (an MXFP4-style 4-bit float). This format has a clear advantage in memory footprint, but the target hardware’s matrix compute units offer limited direct support for 4-bit floating point, so computation typically has to be performed in a more general 8-bit float format.

The conventional approach is to pre-generate a complete FP8 copy of the weights offline. At three trillion parameters, however, this creates two unacceptable problems: it requires an additional amount of storage equal to the original weights, and the conversion process takes so long that it collides with the Day-0 adaptation window.

The Solution: Online Conversion with a Cross-Layer Reusable Workspace

The team’s strategy was online conversion plus a cross-layer reusable FP8 workspace:

• Online conversion: Immediately before an expert computation executes, convert just the 4-bit expert weights currently needed into 8-bit format. The converted result lives only on-chip or in a temporary workspace—it is never written to disk and never materialized as a full offline copy.
• Cross-layer reuse: Because expert computations in different layers do not overlap in time, the same block of FP8 workspace can be reused layer after layer. Workspace memory therefore scales with the expert weights of a single layer, not with the entire model’s weights.

The engineering payoff is twofold: memory usage drops dramatically, making a 3-trillion-parameter model deployable on a single node or a limited number of nodes; and skipping the offline quantization pass is precisely what makes Day-0 timing feasible.

One caveat: online conversion raises the bar for the compute pipeline. Conversion itself consumes compute and must overlap well with matrix computation, or it becomes a throughput bottleneck. That generally requires compiler-level scheduling that orchestrates conversion operations and compute operations into a pipeline.

Engineering Takeaways

Drawing on the adaptation process above, several lessons generalize to deploying large models on heterogeneous or newly built compute platforms:

First, a layered, decoupled software stack is the prerequisite for fast adaptation. With the operator library, the operator description language, and the compiler each playing its own role, a new architecture can be handled at whichever layer fits best, without a full-stack rewrite from the bottom up.

Second, validate new-architecture adaptations at the level of the “chain.” Sparse attention, state management, residuals, and the vision path are coupled both numerically and in scheduling; passing every operator test in isolation does not guarantee end-to-end correctness.

Third, for low precision, distinguish “storage format” from “compute format.” The format in which weights are stored and the format in which they participate in computation are two independent decisions. Online conversion exploits exactly this separation: compact on the storage side, hardware-friendly on the compute side.

Fourth, building Day-0 capability matters more than any single adaptation result. The operator supply, compilation and scheduling, and precision management capabilities this adaptation relied on are reusable infrastructure for future models. Their long-term value exceeds the adaptation of any one model.

Closing Thoughts

Completing Day-0 adaptation of a 3-trillion-parameter MoE model on a Chinese domestic AI accelerator is, in technical substance, a complete heterogeneous-computing adaptation project: an inference framework carries the model implementation; multi-entry operator supply fills in the compute paths; dedicated sparse attention work opens up the long-context chain; and online low-precision conversion resolves the twin constraints of memory and time. For developers working to deploy large models on emerging domestic compute platforms, this layered methodology—and the engineering trade-offs within it—is more instructive than the specific adaptation result itself.