Technical Path for Adapting Domestic AI Compute Cards to Trillion-Parameter Models

Technical Path for Adapting Domestic AI Compute Cards to Trillion-Parameter Models

As open-weight models scale into the trillion-parameter range, rapidly deploying new models on domestic AI compute platforms has become a core engineering challenge for developers and enterprise users. Recently, a model team released an open-weight model with approximately three trillion parameters. On the day of its release, a domestic integrated AI training and inference compute card successfully completed its adaptation, enabling model loading, service startup, and basic text and image inference pipelines. Achieving this “Day-0 adaptation” involved addressing critical technical aspects such as software stack compatibility, sparse attention integration, and low-precision expert weight processing. This article breaks down the implementation path and optimization strategies of this adaptation process from an engineering perspective.

1. Engineering Challenges in Deploying Trillion-Parameter Models

Deploying a three-trillion-parameter model is not as simple as “loading weights and running.” The engineering challenges primarily manifest in the following areas:

First, significantly increased model structural complexity. Current ultra-large models generally adopt composite structures such as Mixture of Experts (MoE), cross-layer residual connections, and sparse attention. The computation graph is no longer a single dense matrix multiplication chain but contains branching logic like conditional routing, dynamic index selection, and state management. If the inference framework cannot fully support these new computation paths, the model will fail to execute correctly.

Second, non-linear growth in memory and bandwidth pressure. Even when stored in a low-precision format, a three-trillion-parameter model demands extremely high memory capacity and inter-card bandwidth. Effectively managing weight layout, KV cache management, and on-demand expert weight scheduling within limited hardware resources is critical to deployment success.

Third, new computation path requirements from long-context and multimodal inference. Million-level context windows mean that attention computation and cache management strategies must be specifically designed for sparsification. Meanwhile, supporting both text and image inference requires the visual attention branch and the main language computation path to execute cooperatively.

Fourth, strict timelines for adaptation. Once an open-weight model is released, users expect deployment validation in the shortest possible time. This requires the software stack to have robust interface compatibility and reusable operator capabilities to avoid building the entire inference pipeline from scratch.

2. Software Stack Compatibility: Infrastructure for Rapid Migration

The primary prerequisite for rapid model migration is a highly compatible software stack across programming models, runtime interfaces, and AI frameworks. In this adaptation process, the team built upon the upstream model implementation of a mainstream open-source inference framework rather than rewriting the model code. This approach significantly shortened the adaptation cycle.

Specifically, the adaptation relied on the following infrastructure components:

• Operator Library: Provides tuned basic and fused operators, covering common computation patterns like matrix multiplication, normalization, activation, and attention.
• Compiler Toolchain: Responsible for lowering high-level computation graphs into device-executable code and applying targeted optimizations to new computation paths.
• Graph Compilation and Runtime Interfaces: Handles execution scheduling at the framework level and abstracts device differences.
• Multi-Card Communication Library: Supports distributed inference strategies such as tensor parallelism and expert parallelism.

The significance of this layered design is that when a new model introduces novel computation patterns, the adaptation team can prioritize reusing mature interfaces and only develop operators and migrate paths for incremental parts. This transforms “community model code” into an actual inference service on the target hardware. The software stack simultaneously covers framework execution, operator compilation, device runtime, and multi-card communication, ensuring a controllable migration process.

3. Integration of Sparse Attention and Visual Computation Paths

The sparse attention mechanism (DSA) used by this model was a focal point of this adaptation. Unlike dense attention, sparse attention requires dynamically selecting indices during inference—specifically, extracting key-value fragments relevant to the current query from the historical context. This introduces three engineering challenges:

1. Computation path adaptation for index selection: Index selection involves scoring and Top-K filtering, requiring dedicated operator support and efficient integration with the main attention computation path.
2. KV cache layout adjustments: After sparsification, the cache is no longer accessed as a continuous sequence. The cache organization must support discrete reads by index, and access patterns differ between the Prefill and Decode phases.
3. State management and cross-layer coordination: The model’s state management module, cross-layer residual connections, and visual attention branch must form a complete execution chain with sparse attention. Missing any component will result in incorrect inference outputs.

The adaptation solution integrated dedicated sparse attention operators and separately adapted computation paths for the Prefill phase (batch processing input sequences) and the Decode phase (token-by-token generation). This enabled sparse attention to work in tandem with state management, cross-layer residuals, and visual attention. Fully connecting this pipeline was the prerequisite for the model to support both text and image inference.

4. Online Conversion Strategy for Low-Precision Expert Weights

Another major engineering hurdle for a three-trillion-parameter model is the storage and loading of expert weights. The model uses a low-precision floating-point format (MXFP4) to store expert weights, while inference computation typically requires a higher-precision workspace to balance performance and numerical stability.

The traditional approach is to pre-generate a complete high-precision weight copy offline. However, for a three-trillion-parameter model, this entails massive additional storage overhead and prolonged preprocessing times, contradicting the goal of rapid deployment. The strategy adopted here was online conversion and cross-layer reuse:

• Convert low-precision weights to the target computation precision online during expert computation execution.
• Reuse a fixed-size FP8 workspace for expert computations across different layers, avoiding separate weight buffer allocations for each layer.
• Eliminate the need to pre-generate a complete set of high-precision weights offline.

The core benefit of this strategy is the simultaneous reduction of memory footprint and deployment preparation time. The trade-off is the introduction of some runtime conversion overhead. To mitigate this, operator fusion and scheduling optimizations are applied to overlap the conversion operations with expert computations, minimizing the impact on overall throughput. For rapid deployment scenarios of ultra-large models, trading a small amount of runtime overhead for significant storage and time savings is a sound engineering compromise.

5. Multi-Card Parallelism and Inference Service Organization

A three-trillion-parameter model cannot be deployed on a single card; it relies entirely on multi-card parallelism. Common organizational strategies include tensor parallelism (splitting a single layer’s weights across multiple cards) and expert parallelism (assigning different experts to different devices). When starting the inference service, the following steps are executed:

1. Weight shard loading: Split and distribute weights to each device according to the parallel strategy.
2. Communication group initialization: Establish communication domains for tensor parallel and expert parallel groups.
3. Computation graph construction: Map the model structure into a device-executable inference graph.
4. Service registration: Expose text and image inference interfaces and enter a request-ready state.

During this process, the multi-card communication library implements collective communication operations such as All-Reduce and All-to-All. Its efficiency directly determines whether the communication overhead introduced by expert routing and tensor sharding remains manageable. By reusing mature communication interfaces, the adaptation team ensured the stability of the service startup pipeline.

6. Engineering Insights and Best Practices

From this adaptation effort, several broadly applicable lessons can be derived for developers:

Prioritize reusing upstream model implementations. Directly adopting the model code from open-source inference frameworks and focusing adaptation efforts on operators and execution paths—rather than model logic itself—is an effective way to shorten the Day-0 adaptation cycle.

Isolate issues by layer; avoid full-stack rewrites. Breaking down the adaptation into framework, operator, compiler, and communication layers to verify compatibility sequentially helps quickly locate bottlenecks.

Apply incremental optimization to new computation paths. Novel structures like sparse attention and visual branches are the focus of adaptation. Mature paths can be directly reused, concentrating resources on the incremental parts.

Explicitly balance storage and runtime overhead. The online conversion strategy for low-precision weights demonstrates that accepting a certain amount of runtime overhead to drastically reduce memory and preparation time is viable for rapid deployments, provided its impact is controlled through overlapping execution.

Validate full-pipeline integrity rather than isolated components. Model loading, service startup, text inference, and image inference constitute a complete chain. If any link fails, the service cannot be delivered. Adaptation acceptance should be based on end-to-end pipeline verification.

Conclusion

Achieving Day-0 adaptation of a three-trillion-parameter model on a domestic integrated AI compute card is fundamentally a comprehensive test of the software stack’s capabilities. Compatible programming models and framework interfaces provide the foundation for migration, dedicated operators and compiler toolchains support the integration of new computation paths, online precision conversion strategies solve the deployment challenges of ultra-large weights, and multi-card communication and parallel strategies ensure service viability. For developers looking to deploy cutting-edge open-weight models on emerging compute platforms, understanding the layered logic and trade-offs of this technical path provides more long-term value than memorizing any specific conclusion.