Engineering Practices for Adapting Trillion-Parameter Sparse Model Inference

I. Background: Deployment Challenges of Trillion-Parameter Sparse Models

In recent years, the parameter scale of large language models has continued to grow, with some models reaching the trillion-parameter level. Unlike dense models, these models typically adopt a mixture-of-experts architecture, where only a subset of experts is activated per inference, achieving stronger expressive capability while controlling computational cost. However, this design also introduces significant pressure on inference deployment:

1. Massive total weight size: Even with low-precision quantization, the complete weights still far exceed single-card memory capacity, requiring multi-card partitioning or dynamic loading mechanisms.
2. Sparse activation characteristics: Expert routing causes the data path of each forward pass to differ, placing higher demands on computation graphs, cache management, and communication patterns.
3. Long context support: Million-token context lengths mean the key-value cache in attention computation grows substantially, requiring dedicated sparse attention strategies to reduce memory access overhead.
4. Multimodal extension: When a model supports both text and image inputs, the interaction between the vision encoder and the language backbone further increases the complexity of the inference pipeline.

Against this backdrop, quickly completing model adaptation on new accelerator hardware has become a key focus. “Day-0 support” refers to completing the basic inference pipeline on the day model weights are publicly released, including model loading, service startup, and forward inference for both text and image inputs. This tests not only the hardware’s computational capability but also the portability and engineering efficiency of the software stack.

II. Adaptation Goals and Overall Approach

The goal of this adaptation work was clear: complete inference deployment of a trillion-parameter sparse model on domestic AI training-and-inference accelerator cards using a proprietary software stack, and establish both text and image basic pipelines. Around this goal, the team formulated the following overall approach.

2.1 Reducing Migration Cost Through Software Stack Compatibility

In large-scale model adaptation, the biggest uncertainty often comes from the software ecosystem. Hardware platforms typically need to provide a software stack aligned with mainstream programming models, runtime interfaces, and AI frameworks. When upper-layer model code changes, if the underlying software stack can quickly accommodate newly introduced operators and execution patterns, the adaptation cycle can be significantly shortened.

Therefore, the first step was not to manually rewrite the model layer by layer, but rather to assess the existing software stack’s coverage of the model’s required key capabilities, and then fill gaps in a targeted manner.

2.2 Handling Model Structure Changes Through Computation Path Abstraction

A sparse expert model goes through multiple distinct stages during execution, including:

• Input embedding and positional encoding
• Multi-layer stacked attention and expert feed-forward networks
• Index selection and key-value cache management in sparse attention
• Cross-modal interaction between the vision encoder and language backbone
• Output layer and sampling logic

The team abstracted these stages into several key computation paths and confirmed their executability on the target hardware individually. Paths with existing mature implementations were reused directly, while new paths were migrated through collaboration between the operator library and compiler.

2.3 Handling Ultra-Large-Scale Weights Through Online Conversion

Due to the enormous total weight size, pre-converting all expert weights to deployment format and persisting them would incur extremely high offline processing costs while consuming large amounts of disk and memory resources. Therefore, the team adopted an online conversion approach: converting expert weights on demand during inference loading and execution, and reusing temporary workspaces to avoid generating complete offline copies.

III. How the Software Stack Supports Rapid Migration of New Models

The foundation for rapid adaptation lies in the hierarchical design of the software stack. A complete software stack typically includes the following layers:

• Programming model and runtime interfaces
• AI framework adaptation layer
• Operator library
• Domain-specific compiler
• Device drivers and multi-card communication library

In this practice, the team built on this layered structure to complete model migration layer by layer, from framework integration and operator generation to device execution.

3.1 Framework Layer Inheriting Upstream Implementation

At the framework layer, the team chose to inherit the upstream model implementation on top of an existing inference serving framework, rather than writing server-side logic from scratch. The advantage is that model structure code, weight loading logic, and inference scheduling strategies can maximize reuse of existing community work, reducing maintenance costs.

The main work at the framework layer included:

• Mapping model configurations and weights to formats recognizable by the target hardware
• Connecting the inference request processing flow with the device execution queue
• Handling dynamic input sizes, particularly the concatenation of image and text inputs

3.2 Operator Library and Compiler Collaborating to Fill Critical Computation Gaps

When new computation patterns appear in the model, the operator library needs to provide corresponding foundational implementations. In this adaptation, the team used the operator library to fill in the index selection and computation logic required for sparse attention, and used the compiler to translate custom computation expressions in the model into code runnable on the target hardware.

Notably, the compiler’s role is not just code generation but also scheduling optimization for actual computation patterns in the model. For example, the computation density at different positions in sparse attention varies significantly, and the compiler needs to select appropriate parallel strategies based on index distribution.

3.3 Unified Management of Device Execution and Multi-Card Communication

For ultra-large-scale models, a single card’s memory cannot hold all weights, necessitating multi-card partitioning. The software stack needs to provide device execution management and multi-card communication capabilities so that the upper framework does not need to be aware of the underlying inter-card topology. In this adaptation, the software stack uniformly covered framework execution, operator compilation, device execution, and multi-card communication, allowing model adaptation to reuse mature interfaces while performing targeted optimization for new computation paths.

IV. Integration of Hybrid Attention and Vision Computation

An important characteristic of trillion-parameter models is the hybrid attention mechanism. In addition to regular dense attention layers, some layers use sparse attention to reduce computation and memory access overhead in long-context scenarios. In this adaptation, the team needed to handle both types of attention and complete the vision pathway integration.

4.1 Index Selection and Cache Management for Sparse Attention

Sparse attention does not directly compute correlations between all positions. Instead, it first selects positions to participate in computation through an index selection mechanism, then performs attention computation only on those positions. This process includes the following key steps:

1. Index selection: Based on current input and historical information, select a subset of positions with higher scores from all candidates.
2. Key-value cache layout adjustment: The key-value cache for sparse attention needs to be reorganized according to selected indices, rather than simply appended in sequence order.
3. Prefill and decode path separation: During the prefill stage, the input sequence enters the model all at once with higher computation density; during the decode stage, only a single newly generated position is processed each time, with a completely different computation pattern.
4. State management: Cross-layer residual and sparse attention states need to be correctly passed between layers.

The team adapted each of these steps to ensure the entire execution pipeline runs correctly on the target hardware. Among these, the sparse attention operator was integrated through the operator library, while index selection and cache management logic were tightly coupled with framework-layer scheduling logic.

4.2 Fusion of Vision Attention and Language Backbone

Multimodal models typically first convert images into feature sequences through a vision encoder, then feed them together with text sequences into the language backbone. The attention computation inside the vision encoder may differ from the language side in dimensions, masking methods, and positional encoding. Adaptation requires ensuring:

• The feature shape output by the vision encoder aligns with the language backbone input
• Attention masks for image tokens and text tokens are correctly concatenated
• Vision attention computation does not interfere with the language side’s state management

The team integrated vision attention as an independent computation path and scheduled it uniformly with the language backbone’s attention path under the same execution framework. This ensures computational correctness while facilitating subsequent performance optimization specifically for the vision component.

V. Online Processing Strategy for Low-Precision Expert Weights

The storage bottleneck of ultra-large-scale sparse models is mainly concentrated in expert weights. Even with low-precision formats, the total byte count of expert weights remains enormous. How to efficiently load and execute expert computation under limited resources is one of the key issues in this adaptation.

5.1 Online Conversion Avoiding Complete Offline Copies

A common approach is to pre-convert all expert weights to the required deployment precision format and save them as complete offline files. However, when the number of experts is large and total weight size is enormous, this approach brings two problems:

• Offline conversion takes too long, affecting the model launch timeline
• Complete copies consume large amounts of disk and memory resources, increasing deployment costs

In this work, the team adopted an online conversion approach: converting expert weights from their original format to the required execution precision on demand during inference, with conversion results written to a temporary workspace. This eliminates the need to pre-generate complete offline low-precision weight files, significantly reducing dependence on storage resources.

5.2 Cross-Layer Reuse of Temporary Workspaces

During layer-by-layer execution, expert weight conversion for different layers proceeds serially. The team reused the same temporary workspace across layers to store converted weight data. Specifically, once a layer’s expert computation is complete, its occupied workspace can be reused by the next layer. This approach effectively controls peak memory usage.

Implementing this strategy requires close coordination between the framework layer and the device execution management layer: the framework layer is responsible for marking workspace lifetimes, and the device layer handles reclamation and reallocation based on those markers. Through this collaboration, the model can complete forward inference of trillion-parameter scale under limited memory conditions.

5.3 Implications for Deployment Practice

The online conversion and workspace reuse strategy is not only applicable to this adaptation scenario but also has general reference value. For any team that needs to deploy ultra-large-scale sparse models on limited hardware resources, the following points are worth noting:

• Prioritize evaluating the actual usage patterns of weights in the model, avoiding over-allocating resources for infrequently activated experts
• Pipeline weight conversion with computation so that conversion overhead overlaps with computation overhead as much as possible
• Design reusable temporary buffer management mechanisms to reduce peak memory usage

VI. Summary and Future Directions

This adaptation achieved Day-0 support for a trillion-parameter sparse model on domestic accelerator cards, establishing model loading, service startup, and basic text and image inference pipelines. From an engineering perspective, this work validated the following lessons:

Software stack compatibility is the foundation for rapid adaptation. When the programming model, runtime interfaces, and AI framework adaptation layer can quickly accommodate new models, the team can focus on filling in critical operators and computation paths rather than getting bogged down in layer-by-layer adaptation details.

Computation path abstraction helps clarify complex models. Breaking the model into key paths such as embedding, attention, expert networks, and vision encoding allows confirming the feasibility and optimization potential of each path separately, reducing overall adaptation complexity.

Online conversion strategy is critical for ultra-large-scale weights. Avoiding complete offline low-precision copies, and controlling resource usage through on-demand conversion and workspace reuse, is an effective means of achieving rapid deployment.

Future work can go deeper in the following areas:

1. Further optimize the memory access patterns of sparse attention on target hardware to reduce latency in long-context scenarios.
2. Explore pipeline overlap between expert weight conversion and expert computation to reduce the impact of conversion on end-to-end inference throughput.
3. Tune mixed scheduling strategies for text and image requests based on real business workloads.
4. Improve dynamic batching, queue management, and resource isolation capabilities under multi-user concurrency at the serving layer.

As large-scale sparse models and multimodal models continue to evolve, the engineering complexity of inference deployment will continue to rise. For cloud inference services and private deployments, the ability to quickly complete adaptation after a new model release will directly affect the practical availability of technical solutions. By accumulating general migration tools, reusing mature execution paths, and establishing standardized adaptation processes, teams can continuously shorten the time window from model release to usable service in subsequent model iterations.