Adapting a Trillion-Parameter Sparse Model on a Domestic AI Accelerator Platform: A Practical Breakdown

Why Adapting Ultra-Large Sparse Models Has Become an Engineering Focus

Recently, an ultra-large sparse model with approximately three trillion parameters was released with complete open weights for the first time. It supports context lengths of up to one million tokens. After large-scale post-training and reinforcement learning, the model shows clear improvements in code generation, 3D content creation, financial research, knowledge-based QA, and long-horizon agent tasks. On agentic benchmarks, its performance is approaching leading closed-source models. In some long-horizon task benchmarks, scores have risen from roughly 50% in the previous generation to more than 60%.

For developers and enterprises, stronger model capabilities expand the application space, but they also introduce serious engineering challenges: how to quickly adapt and reliably run a new model on a domestic AI accelerator platform. This article walks through a typical Day-0 adaptation effort, from model weight release to inference service availability, and is intended for developers working on LLM deployment and inference optimization.

Adaptation Goals and Technical Challenges

The adaptation had three core objectives:

1. Model loading: correctly load a three-trillion-parameter sparse model into accelerator memory.
2. Service startup: launch inference service on a mainstream inference framework and establish basic inference paths for both text and image inputs.
3. Stable operation: maintain stability and scalability of the inference service in a multi-card environment.

Several technical challenges were involved:

• Sparse attention mechanism: the model uses dynamic sparse attention (DSA), which requires separate adaptation for index selection, KV cache layout, and the Prefill and Decode computation stages.
• Mixture-of-experts weight format: the model stores expert weights in MXFP4 format. Offline conversion to FP8 would create significant storage and conversion overhead.
• Visual computation path: the model supports image inference, so the visual attention module must work together with the text-side execution path.
• Cross-layer state management: KDA state management and cross-layer residual connections need to be combined with sparse attention operators to form a complete execution path.

Software Stack Compatibility as the Foundation for Fast Migration

How quickly a new model can run on a domestic AI accelerator platform depends heavily on how compatible the software stack is. The stack used in this adaptation provided support in the following areas:

• Programming model and runtime interfaces: the programming paradigm remains consistent with mainstream AI frameworks, allowing community model code to be ported at low cost.
• Operator compilation and execution: operator libraries, compilers, and domain-specific language support are used to turn the key computation paths of the new architecture into actual inference services on target hardware.
• Multi-card communication: mature interfaces for device operation and multi-card communication can be reused during adaptation.

Specifically, the adaptation team used a ported version of a mainstream inference framework to take over the upstream model implementation, then completed migration of the new architecture’s key computation paths through operator libraries and compiler toolchains. This strategy of reusing mature interfaces while optimizing only the newly introduced paths significantly shortened the time from weight release to service availability.

Practical Integration of Sparse Attention and Visual Computation

Key Points for Adapting Sparse Attention

For the model’s dynamic sparse attention mechanism, the adaptation focused on three areas:

Index selection: the core of sparse attention is dynamically selecting which key-value pairs participate in computation. The adaptation needed to ensure that index selection logic runs efficiently on target hardware and avoids pipeline stalls caused by branching or data dependencies.

KV cache layout: access patterns for the KV cache in sparse mode differ significantly from dense attention. A well-designed memory layout can reduce ineffective memory access and improve Decode-stage throughput.

Separate Prefill and Decode paths: Prefill is compute-bound, while Decode is more constrained by memory bandwidth. Adapting the two paths separately makes it easier to optimize for the bottleneck of each stage.

The team integrated sparse attention operators and combined them with state management, cross-layer residual connections, and visual attention modules into a complete execution path, enabling unified support for both text and image inference.

Online Conversion of Mixture-of-Experts Weights

In a three-trillion-parameter model, expert layers account for most of the parameters. The model uses MXFP4 for expert weights. Converting them offline to FP8 would require storing a full additional copy of the weights and would lengthen deployment preparation.

The adaptation instead used online conversion with cross-layer reuse of an FP8 workspace:

• MXFP4 weights are converted to FP8 on demand during inference.
• Different layers reuse the same FP8 workspace, avoiding separate memory allocation for every layer.
• No full offline FP8 weight set needs to be generated in advance, which supports fast deployment of very large models.

This approach balances memory usage and deployment efficiency, and it is broadly relevant as model sizes continue to grow.

Engineering Insights and Best Practices

Several practical lessons can be drawn from this adaptation:

First, prefer mature interfaces. Software stack compatibility directly determines migration cost. When choosing an inference framework and operator libraries, prioritize toolchains that stay aligned with the mainstream ecosystem rather than building from scratch for each new model.

Second, optimize only the new computation paths. Reuse does not mean no optimization. New features such as sparse attention and mixture-of-experts weight formats require dedicated operator implementations and memory management strategies.

Third, design memory and communication together. For ultra-large models in multi-card environments, performance often depends on how well computation, memory, and communication overlap. Multi-card communication should be part of the overall design from the beginning, not an afterthought.

Fourth, build a reusable adaptation process. Standardizing steps such as model loading, service startup, and inference path validation makes it easier to respond quickly when new models are released.

Conclusion

Adapting an ultra-large sparse model to a domestic AI accelerator platform is a systems engineering effort involving operators, compilation, memory management, and multi-card communication. Through software stack compatibility and targeted optimization for the new architecture, this practice completed the path from weight release to working text and image inference in a relatively short time. For teams working on domestic deployment of large models, understanding the key decision points in this adaptation path can help avoid unnecessary detours and turn model capabilities into usable business services more quickly.