Domestic GPU Completes Adaptation for 3-Trillion-Parameter Model, Full Text-and-Image Inference Pipeline Now Operational

Background: New Challenges as LLM Inference Enters the Trillion-Parameter Era

As large language model parameter counts surpass the trillion mark, the computational complexity and memory pressure during inference grow exponentially. This is especially true for models with long-context capabilities (e.g., million-token contexts), where inference involves not only traditional forward computation but also complex paths such as sparse attention, mixture-of-experts structures, and multimodal encoders. For computing infrastructure, efficiently supporting the stable operation of such models has become a key indicator of technical maturity.

Recently, an open-weight large language model series released its most powerful version—a model with a parameter scale of 3 trillion. On the day of its release, adaptation work was completed on a domestic AI training-and-inference acceleration card and its accompanying software stack, achieving “Day-0 support” and successfully connecting the complete foundational pipeline from model loading and service startup to mixed text-and-image inference. This progress marks an important step forward in the engineering capability of domestic AI infrastructure to handle ultra-large-scale models.

Notably, the model performs strongly across multiple authoritative benchmarks: it ranks first among open-weight models and within the top 15 across all evaluated models; it leads all open models in three core task categories—work, coding, and chat. In code generation and agent tasks in particular, its scores approach those of top closed-source models—for example, 65.7% on a terminal-operation benchmark and 61.2% on a long-horizon coding benchmark, representing clear improvement over the previous generation.

Technical Core: Adaptation Engineering from Model Architecture to Hardware Mapping

1. Migration and Optimization of the Sparse Attention Mechanism

The model employs a novel sparse attention (DSA) mechanism, whose core idea is to focus only on a small set of key-value pairs highly relevant to the current query position when computing attention scores, rather than performing full computation across all positions. This design substantially reduces computational complexity during long-sequence inference, but it also introduces new challenges for hardware adaptation: sparse index generation, sorting, and masking operations must work closely with low-level compute kernels.

In this adaptation effort, the software stack team refactored the index computation logic for the DSA mechanism, mapping it to parallel tasks that execute efficiently on the acceleration card. Specifically, custom kernels fuse the sparse index generation process with the attention score computation process, reducing the number of intermediate-result memory reads and writes and significantly improving inference efficiency in long-context scenarios.

2. Integrating Hybrid Attention and Visual Computing Paths

Beyond text processing, the model also supports mixed text-and-image inference, meaning the inference pipeline must include computation logic for both text encoders and visual encoders. Data from the two modalities passes through different feature extraction paths within the model and is fused at specific layers. For hardware adaptation, this requires the software stack to flexibly schedule different compute kernels and efficiently switch between heterogeneous computing units.

During implementation, the development team added a unified device abstraction layer at the software stack’s runtime level, enabling convolution operations and attention computation in the visual computing path to share the same memory management mechanism and kernel scheduling strategy, avoiding the device-switching overhead common in traditional multimodal inference.

3. Coordinated Work of the Compiler Toolchain and Operator Libraries

Adaptation of modern AI inference frameworks depends heavily on whether the underlying compiler toolchain can efficiently translate high-level model descriptions into hardware-executable code. This adaptation adopted a two-layer strategy: “upstream framework integration + in-house compiler stack.” On one hand, an industry-standard serving framework (such as SGLang) is used as the upper-layer inference engine to ensure compatibility with community model code; on the other hand, in-house operator libraries, compilers, and domain-specific language (DSL) tools handle the migration and integration of key computation paths in the new architecture at the lower level.

The advantage of this layered design is that the upper-layer framework can stay synchronized with the community, while the lower-layer toolchain can perform deep optimization for specific hardware architectures. For example, when implementing a novel activation function, the compiler can automatically generate a micro-kernel optimized for that hardware’s instruction set without modifying upper-layer framework code.

The Key Role of the Software Stack: Infrastructure for Rapid Adaptation

1. Unified Programming Model and Runtime Interface

The core enabler of “Day-0 support” is that the software stack provides a complete programming model and runtime interface. This interface offers upper layers an abstraction similar to standard parallel programming, allowing developers to port model operators without deep knowledge of underlying hardware details. Specifically, its runtime interface covers device management, memory allocation, kernel launch, and multi-card communication, ensuring scalability during distributed inference.

2. Multi-Card Communication and Load Balancing

For a 3-trillion-parameter model, single-card memory is far from sufficient to hold all weights, so multi-card parallel inference is mandatory. In this adaptation, the software stack provides native support for both tensor parallelism and pipeline parallelism. In the multi-card communication stage, topology-aware optimization of the collective communication library reduces cross-card communication latency; measured at 64-card scale, linear speedup remained above 85%.

3. Stability and Fault Tolerance

Inference services for ultra-large models demand extremely high stability. During this adaptation, the software stack introduced fine-grained fault detection and recovery mechanisms that can automatically migrate the workload of a failing compute card to other idle cards, achieving service continuity. At the memory management level, paged memory management techniques allow requests with dynamic input lengths to be scheduled more efficiently.

Practical Reference: Key Metrics and Tuning Directions for Inference Deployment

For teams looking to deploy models of similar scale on their own infrastructure, the following points merit attention:

Tuning Dimension	Key Consideration	Recommended Practice
Memory planning	Model weights + KV Cache + activations	Prioritize paged memory management with dynamic cache allocation
Compute precision	FP16/BF16 mixed precision vs. INT8 quantization	Retain high precision for visual encoding paths; enable quantization for text paths
Request scheduling	Dynamic batching vs. static batching	Adopt continuous batching to improve throughput
Multi-card topology	Intra-node NVLink vs. cross-node IB network	Prefer restricting tensor parallelism to the same compute node

Additionally, for long-context inference scenarios, enabling Async Prefill mode is recommended—separating prompt processing from the decoding stage to avoid excessively high first-token latency caused by long prompts.

Summary and Outlook

This adaptation effort demonstrates that domestic AI acceleration cards and software stacks are now capable of independently supporting the latest generation of ultra-large-scale open-source models. By building a unified programming model, a complete compiler toolchain, and flexible runtime mechanisms, the time window from model release to available service can be significantly shortened.

In the future, as model architectures continue to evolve (e.g., larger parameter scales, more modality combinations), co-design of hardware and software stacks will become increasingly important. The industry broadly favors the approach of pushing inference performance ceilings higher through joint hardware-software optimization, which will also become a core competitive differentiator for AI infrastructure vendors. For developers, understanding and mastering the adaptation methodologies described above will help them deploy large model applications more efficiently in heterogeneous computing environments.