Adapting Trillion-Parameter Sparse LLMs for Inference on Domestic Accelerators

1. Background: New Inference Challenges with Trillion-Parameter Sparse Models

In recent years, the parameter scale of large language models has continued to grow, evolving from hundreds of billions to the trillion-parameter level. Unlike earlier models dominated by dense parameters, trillion-scale models typically adopt a sparse Mixture-of-Experts (MoE) architecture. While the total parameter count is enormous, only a fraction of the experts are activated during a single forward pass, meaning the computational volume does not scale linearly with total parameters. Nevertheless, inference deployment still faces three typical challenges.

First, model loading is difficult. If trillion-level parameters are saved at standard precision, the weight files alone can reach several terabytes, far exceeding the VRAM capacity of a single accelerator. Even with quantization strategies, multi-card collaboration, weight sharding, and tiered loading are typically required.

Second, service startup time is prolonged. The combination of large-scale weight reading, distributed communication initialization, and computation graph construction can easily lead to cold starts taking minutes or longer, which is detrimental to rapid iteration and elastic scaling.

Third, the multimodal pipeline is complex. When a model supports both text and image inputs, the inference pipeline must simultaneously handle text encoding, image encoding, multimodal fusion, and autoregressive decoding. A performance bottleneck in any of these stages will impact overall availability.

Therefore, determining whether an acceleration platform can support ultra-large models requires looking beyond theoretical computing power. It also requires evaluating its engineering maturity in model loading, service startup, distributed inference, and multimodal pipelines.

2. Adaptation Goals and Overall Approach

To address these challenges, the goals of this adaptation work can be summarized in three points:

1. Fast Loading: Map trillion-level sparse model weights into multi-card VRAM using tiered and sharded approaches to reduce cold start time.
2. Service Availability: Launch the inference service, enabling the model to provide stable basic inference capabilities for both text and images.
3. Pipeline Integration: Validate the complete pipeline from input preprocessing and encoding to decoding output, ensuring correct basic functionality.

Centered on these goals, the overall approach begins with model structure analysis to determine weight sharding and communication topology. This is followed by loading and startup, and finally, multimodal inputs are integrated for end-to-end validation. This “analyze structure first, connect service next, validate pipeline last” methodology helps quickly locate issues within complex systems.

3. Model Structure Understanding: Why “Trillion Parameters” Does Not Equal “Trillion Computations”

The first step in adaptation is understanding the model structure. Taking the sparse MoE architecture as an example, the model typically contains the following modules:

• Shared Attention Layers: Responsible for text sequence modeling, with relatively stable parameters.
• Routing Module: Selects which experts to activate based on input features.
• Expert Feed-Forward Networks: Numerous in quantity, accounting for a high proportion of total parameters, but only a subset is activated per forward pass.
• Multimodal Encoders: When image input is supported, visual embedding and cross-modal alignment modules are also required.

The parameter count of such models is primarily concentrated in the expert feed-forward networks. For instance, a model with hundreds of experts, each with a substantial parameter count, might have total parameters in the trillions, but actual inference activates only a few experts per token. Therefore, adaptation needs to focus on solving two main problems:

1. How to efficiently store and manage all expert weights.
2. How to quickly dispatch and compute the activated experts during inference.

In multi-card environments, expert parallelism or expert sharding strategies are typically employed, distributing different experts across different accelerator cards. The routing results determine which cards the data is sent to. This reduces single-card VRAM pressure and avoids frequent full-weight transfers.

4. VRAM and Communication Design: From Single-Card to Multi-Card Collaboration

Trillion-level models cannot rely on a single card to keep all weights resident in memory. Therefore, a reasonable VRAM and communication scheme must be designed. Common practices include weight sharding and tiered loading.

4.1 Weight Sharding

The goal of weight sharding is to ensure each accelerator card only stores a portion of the parameters while guaranteeing that data required for forward computation can be obtained via inter-card communication. Sharding granularity can be divided by layer, by expert, or by tensor dimension. For models with a massive number of experts, sharding by expert dimension is usually more straightforward: each card is responsible for the parameters and computations of a subset of experts, and the routing module sends data to the corresponding cards.

A schematic expert sharding logic is as follows:

# Schematic: Sharding expert parameters to different devices by expert index
def shard_experts(expert_list, num_devices):
    shards = [[] for _ in range(num_devices)]
    for idx, expert in enumerate(expert_list):
        device_id = idx % num_devices
        shards[device_id].append(expert)
    return shards

The advantage of this sharding method is its simplicity and ease of integration with the routing module. The drawback is the additional communication overhead required for data distribution and result collection.

4.2 Tiered Loading

For the startup phase, reading all weights into VRAM at once is time-consuming. A tiered loading strategy allows the system to load the first model layer and essential structures first, quickly establishing a service entry point, and then asynchronously loading the remaining layers. This enables the service to respond to health checks early in the startup process, improving the deployment experience.

Schematic tiered loading flow:

1. Read model configuration and build computation graph structure
2. Load embedding layers, first-layer attention, and routing module
3. Start service process and open health check endpoint
4. Asynchronously load expert shard weights to each device
5. Switch to inference-ready state after all loading is complete

This strategy requires the inference framework to support weight lazy loading or asynchronous loading capabilities. For ultra-large models, it can significantly reduce the waiting time for “service unavailable.”

5. Service Deployment: From Weight Files to Callable Interfaces

After weight loading is complete, the next step is to encapsulate the model into a service capable of providing inference capabilities externally. Service deployment typically includes the following components:

• Inference Engine: Handles model forward computation and decoding scheduling.
• Request Queue: Buffers concurrent requests to prevent overloading.
• Batching Scheduler: Merges multiple requests into batches to improve throughput.
• I/O Conversion: Interfaces with both text and image input modes.

For models supporting both text and image inputs, the service layer must distinguish between request types. Text requests typically undergo tokenization and embedding before entering autoregressive decoding. Image requests must first pass through a visual encoder to extract features, which are then concatenated or fused with the text sequence before entering the unified decoding process.

A simplified request handling logic is as follows:

# Schematic: Determine request type and dispatch to different preprocessing paths
def handle_request(req):
    if req.type == "text":
        tokens = tokenize(req.text)
        inputs = embed_text(tokens)
    elif req.type == "image":
        image_features = encode_image(req.image)
        prompt_tokens = tokenize(req.prompt)
        inputs = fuse_multimodal(prompt_tokens, image_features)
    outputs = decode(inputs)
    return outputs

It is worth emphasizing that service availability depends not only on the inference engine but also heavily on request scheduling, exception handling, and monitoring mechanisms. In large model scenarios, a single request may have a long generation length. Therefore, reasonable timeout and cancellation strategies must be set to prevent long requests from occupying resources and blocking other requests.

6. Validation Methods for Text and Image Inference Pipelines

Whether the adaptation is successful is ultimately determined through end-to-end inference pipeline validation. The validation work can be divided into three levels.

6.1 Basic Pipeline Validation

First, validate whether text input can complete the entire process from tokenization to output. Focus areas include:

• Whether the input length meets expectations.
• Whether the output is complete and without truncation.
• Whether the generated content is semantically relevant to the input.

This step can rule out issues with model loading, service startup, and basic decoding.

6.2 Multimodal Pipeline Validation

Next, validate the image input path. A typical approach is to prepare an image with clear semantics, accompanied by a short text prompt, and observe whether the model output correctly describes the image content. For example, input an image containing a specific object, give the prompt “Please describe the object in the image,” and check if the output contains the corresponding information.

The difficulties in multimodal validation lie in: whether the image encoding results are correctly fused; whether the cross-modal positional relationship is accurate; and whether the output contains hallucinations. The basic pipeline validation phase can initially focus on “whether it can correctly identify the main object,” leaving finer-grained evaluations for later.

6.3 Distributed Consistency Validation

For multi-card deployment, it is also necessary to verify whether inter-card communication and weight sharding are correct. This can be done by comparing the output similarity of a small-scale single-card model and a multi-card model under the same inputs, confirming that the sharding and communication logic have not introduced errors. This comparative validation is especially important during the initial adaptation.

7. Key Points of Domestic Software Stack Integration

Adapting ultra-large models on domestic accelerator cards relies heavily on the collaborative capabilities of the software stack, which directly impacts deployment efficiency. A complete adaptation typically involves multiple layers: drivers, runtime, compilers, operator libraries, and inference frameworks. The following points are worth noting.

First is operator coverage. Trillion-level models often use newer operators, such as sparse routing, expert aggregation, and high-precision attention. If the underlying operator library has insufficient coverage, it must be supplemented through operator combination or custom implementation, which increases adaptation costs.

Second is communication efficiency. In expert parallelism scenarios, inter-card communication is frequent. If the communication library is not tightly integrated with hardware characteristics, it will become a clear bottleneck. Therefore, the communication topology must be set reasonably to keep data transfers within a local scope as much as possible.

Third is debugging tools. Once numerical anomalies or performance drops occur in distributed ultra-large model inference, locating the issue is difficult. Comprehensive logging, profiling tools, and layer-by-layer output comparison capabilities can significantly shorten problem localization time.

Finally, framework compatibility. Mainstream inference frameworks support ultra-large models to varying degrees. During adaptation, the model export format, sharding strategy, and loading method must be adjusted based on framework characteristics. Maintaining compatibility with general frameworks helps with future migration and reuse across different environments.

8. Practical Significance and Future Directions

Completing the inference adaptation of a trillion-parameter sparse large model on domestic accelerator cards holds significance in at least three aspects.

First, it validates the platform’s scalability. Supporting trillion-level models means the accelerator card can handle extreme-scale scenarios in VRAM management, communication, and operators, providing a reference for subsequent, larger models or more complex tasks.

Second, it accumulates rapid adaptation experience. Through standardized processes for tiered sharding, service deployment, and pipeline validation, adaptation of similar models can be completed in a shorter time, improving response speed.

Third, it lays the foundation for multimodal applications. Connecting basic text and image inference pipelines equips the platform with the ability to host multimodal intelligent applications, paving the way for richer interaction forms in the future.

Subsequent work can delve deeper into the following directions:

• Conduct precision evaluation to verify that the model’s output quality on public tasks is consistent with reference implementations.
• Optimize expert scheduling strategies to reduce routing and communication overhead and improve throughput.
• Explore inference stability and VRAM optimization under longer context lengths.
• Enhance service layer capabilities to support streaming output, dynamic batching, and concurrency scheduling optimization.

Overall, ultra-large model inference adaptation is not a breakthrough in a single环节, but the result of the coordinated advancement of model structure understanding, VRAM communication design, service deployment, and validation systems. Only by forming a standardized adaptation methodology can we maintain the ability to keep pace quickly in the context of continuous model iteration. For developers and enterprise users, paying attention to a platform’s support capabilities and engineering efficiency for ultra-large models will help make more robust decisions in technology selection and deployment practices.