3-Trillion-Parameter Model on Domestic AI Accelerators: From Software Stack Compatibility to Hybrid Attention Inference

1. Background: The Deployment Challenge at Trillion-Parameter Scale

As large models push into the trillion-parameter range, the engineering challenge of deployment has shifted from “can the weights be loaded” to systematic, full-pipeline optimization. A research lab recently released its latest 3-trillion-parameter model, which supports 1M-token context lengths and delivers across-the-board improvements in code generation, 3D content creation, financial research, knowledge-driven Q&A, and long-horizon agent tasks. On a third-party agent leaderboard, the model ranks first among open-weight models, and its results on coding and agentic benchmarks are at or near the level of top closed-source models.

At 3 trillion parameters, however, every stage of the pipeline—weight loading, memory management, attention computation, and expert routing—faces extreme resource pressure. Achieving Day-0 adaptation on China’s domestically developed training-and-inference AI accelerator cards—meaning deployment and inference are possible on the day the model ships—has become a key yardstick for the maturity of the domestic AI software stack.

This article breaks down the adaptation path across three dimensions: software stack compatibility, sparse attention integration, and low-precision expert weight deployment.

2. Software Stack Compatibility: The Foundation for Fast Porting

2.1 Compatible Programming Models and Runtime Interfaces

Fast adaptation of an ultra-large model presupposes a software stack compatible enough that community model code can be ported to the target hardware at low cost. The foundation of this adaptation was a stack that maintains a high degree of compatibility with the mainstream ecosystem at the programming model, runtime interface, and AI framework levels.

Concretely, the adaptation team carried the upstream model implementation on top of an inference framework, then used in-house operator libraries, a Triton compatibility layer, a TileLang compatibility layer, and a compiler toolchain to migrate and integrate the key compute paths of the new architecture. The core logic: map the standard operator calls in community model code to efficient, hardware-executable implementations, rather than asking developers to rewrite the model.

From an engineering standpoint, this compatibility design delivers two direct benefits:

• Reuse of mature interfaces: model adaptation can heavily reuse existing interfaces for framework execution, operator compilation, device runtime, and multi-card communication, cutting down duplicate work;
• Focus on incremental optimization: the team only needs to optimize the newly added compute paths (such as the new attention mechanism and low-precision expert computation) instead of rewriting everything.

2.2 Full-Stack Coverage

A complete inference service involves multiple layers working together. The software stack must cover all of the following:

┌──────────────────────────────────────────────────┐
│  Framework layer (model loading / scheduling)                │
├──────────────────────────────────────────────────┤
│  Operator compilation layer (graph optimization / operator fusion)  │
├──────────────────────────────────────────────────┤
│  Device runtime layer (kernel execution / memory management)    │
├──────────────────────────────────────────────────┤
│  Multi-card communication layer (collectives / pipeline parallelism)  │
└──────────────────────────────────────────────────┘

Only when all four layers are fully supported can a model move from “the code compiles” to “the service runs.” By covering the full stack, this adaptation enabled model loading, service startup, and the basic text and image inference pipelines.

3. Sparse Attention and Vision Compute: Wiring Up Hybrid Attention

3.1 Adapting the Sparse Attention Compute Path

For long contexts, a 3-trillion-parameter model using standard dense attention would face quadratic growth in compute complexity with sequence length, making both memory usage and compute cost unacceptable. The model therefore adopts a sparse attention mechanism (DSA), which computes attention only over a selected subset of key tokens via an index-selection strategy.

At the hardware level, integrating sparse attention involves several key pieces:

1. Index selection: the indices of the key-value pairs that participate in the computation must be selected dynamically based on the similarity between query and key vectors. This involves sorting or Top-K operations and places heavy demands on hardware parallelism.
2. KV cache layout: under sparse attention, the KV cache must be stored in a layout that matches the index-selection results to avoid wasted memory accesses. A well-designed layout can significantly reduce memory bandwidth pressure.
3. Prefill and decode path adaptation: the prefill phase (processing the input sequence) and the decode phase (generating tokens one at a time) exhibit different compute patterns under sparse attention and must be optimized separately.

The adaptation team integrated in-house sparse attention operators and combined them with state management, cross-layer residuals, and the vision attention module into a complete execution chain, keeping these compute paths running stably.

3.2 Unified Handling of Vision Attention

The model supports multimodal inference over text and images, which means features from the vision encoder must be fused with text tokens at the attention level. Vision attention differs from text attention in its compute pattern: there are far more vision tokens, and they carry rich spatial structure, requiring tailored memory layouts and compute strategies.

In the unified execution chain, vision attention shares some operator implementations with text sparse attention, but each manages its own KV cache and indexing strategy independently. This “shared + separated” design lowers development cost while keeping the two compute paths independent.

4. Low-Precision Expert Weights: MXFP4 Storage with an FP8 Workspace

4.1 The Storage Challenge of Expert Weights

The model uses a Mixture-of-Experts (MoE) architecture, with expert weights stored in MXFP4 format. MXFP4 is a 4-bit floating-point format that compresses weight storage to one quarter of FP16, substantially reducing memory usage. However, low-precision formats must be converted to higher precision (such as FP8 or FP16) at compute time to maintain numerical stability.

The traditional approach is to pre-generate a complete set of offline FP8 weight files. But for a 3-trillion-parameter model, even FP8 weights require hundreds of gigabytes of storage, making both the conversion time and the storage cost of offline conversion hard to ignore.

4.2 Online Conversion with Cross-Layer Workspace Reuse

This adaptation took a more efficient approach: online conversion with a cross-layer reusable FP8 workspace. The core ideas are:

• Online conversion: during inference, MXFP4 expert weights are converted to FP8 on demand, rather than being fully converted in advance;
• Cross-layer reuse: expert computations in different layers reuse the same block of FP8 workspace memory, avoiding a separate converted copy of the weights for every layer.

The advantages of this strategy:

Dimension	Offline pre-conversion	Online conversion + workspace reuse
Storage overhead	Hundreds of GB of extra storage	A single workspace
Startup time	Full conversion required first	No pre-conversion needed
Memory footprint	Higher	Significantly lower
Implementation complexity	Lower	Moderate

With this approach, rapid deployment of an ultra-large model becomes feasible: without waiting for a lengthy offline conversion, the service can go live on the day the model is released.

5. Engineering Takeaways

5.1 Key Milestones in the Adaptation Process

From a delivery perspective, the Day-0 adaptation of an ultra-large model comes down to these milestones:

1. Weight loading: verify weight format compatibility and ensure sharded loading and memory management work correctly;
2. Service startup: confirm the inference framework can initialize the model, allocate the KV cache, and start the service processes;
3. Basic inference validation: complete basic inference checks for both text and image inputs;
4. Performance tuning: apply operator-level optimizations to critical paths such as sparse attention and expert computation.

5.2 What Developers Should Keep in Mind

For developers deploying ultra-large models on domestic AI accelerator platforms, several points deserve attention:

• Software stack compatibility comes first: choosing a stack that is highly compatible with mainstream frameworks greatly reduces migration cost;
• Check sparse attention support: for long-context workloads, sparse attention is a must—confirm that the stack provides the corresponding operators;
• Low-precision deployment strategy: the ability to convert low-precision formats such as MXFP4 on the fly directly affects deployment efficiency and memory usage;
• Validate the full pipeline: from model loading to service startup to inference output, every step needs end-to-end verification to avoid a model that “compiles but doesn’t run.”

6. Conclusion

Adapting a 3-trillion-parameter model to domestic AI accelerators is a comprehensive stress test of a software stack’s end-to-end capability. From programming model compatibility and sparse attention integration to low-precision expert weight deployment, every step demands deep engineering optimization and careful debugging and validation. Completing this Day-0 adaptation shows that the domestic AI software stack can support rapid deployment of ultra-large models, laying the engineering groundwork for even larger models and more complex architectures ahead.

For developers, understanding the technical logic behind these adaptation paths not only helps with today’s deployment work but also provides a reusable methodology for tackling the even larger models to come.