Deploying Ultra-Large Sparse Models on Domestic Accelerator Cards: Adaptation and Inference Practices

1. Background: Engineering Challenges of Ultra-Large Model Deployment

In recent years, the parameter counts of large language models have continued to grow, rapidly moving from the hundreds of billions to the trillions, and in some research settings sparse models with trillions of parameters have already appeared. Compared with traditional dense models, the weight matrices of these models are highly sparse and the proportion of activated parameters is very low, which makes inference on a single card or a small number of accelerator cards feasible. At the same time, however, model loading, weight sharding, service startup, and multimodal input handling all face new engineering challenges.

From a developer’s perspective, deploying an ultra-large sparse model is not simply a matter of “reading the weights into VRAM.” The core issues include:

• The total model weight may reach the terabyte scale, far exceeding the memory capacity of a single card;
• Sparse parameters need to be sharded by layer or by expert while ensuring that routing results can be addressed correctly during inference;
• The inference service must handle both text and image inputs, involving preprocessing, encoding, and multimodal feature fusion pipelines;
• First-load and service startup times need to be minimized to meet business launch schedules.

Therefore, the goal of Day-0 adaptation work is usually not to pursue peak throughput, but to first run the complete “load—start—infer—return results” loop on the target hardware, providing a usable baseline for subsequent tuning.

2. Adaptation Goals and Basic Approach

2.1 The Role of Day-0 Adaptation

Day-0 adaptation refers to completing basic usability validation on a specified hardware platform as soon as a new model is released. It does not require optimal performance, but it must demonstrate that the hardware can support the model’s key inference paths. Specifically for ultra-large sparse models, Day-0 adaptation typically includes the following validation goals:

1. Model weights can be correctly loaded onto accelerator cards or a cluster of accelerator cards;
2. The inference service can start normally and expose interfaces externally;
3. The text inference path can return reasonable results;
4. If the model supports image input, the basic image inference path can also be connected;
5. No out-of-memory errors, process crashes, or obvious precision anomalies occur throughout the process.

2.2 Reducing Single-Card Pressure Through Sparse Structure

The reason ultra-large models can run on relatively small-scale hardware is that they adopt the idea of conditional computation. The model does not activate all parameters for every input; instead, through gating or routing mechanisms, it activates only a subset of experts or parameter blocks. For example, the total parameter count may reach trillions, but a single forward pass actually uses only a small fraction of them.

The direct benefit of this design is that a single card only needs to keep the routing network, embedding layers, and necessary metadata resident, while expert weights can be loaded on demand or reside in the memory of multiple cards. The prerequisite, however, is that the deployment framework supports layer-by-layer loading and release of sparse weights. If the framework still reads all weights at once following the dense-model approach, deployment will be difficult to complete.

2.3 Hardware and Software Stack Coordination

Adapting large models on accelerator cards requires not only sufficient memory bandwidth and compute from the hardware itself, but also a complete software stack. This includes:

• Low-level drivers and runtimes;
• Operator libraries optimized for accelerator cards;
• Compilers/runtimes that support dynamic shapes, multi-card communication, and sparse computation;
• An adaptation layer that interfaces with the upper-level inference framework.

During the Day-0 phase, it is often necessary to jointly debug the driver, operator library, and inference framework to locate compatibility issues. Common problems include: missing operators causing computation graph compilation failures; abnormal efficiency of sparse matrix multiplication under non-contiguous memory layouts; and incorrect results caused by desynchronization between multi-card communication and routing computation.

3. Model Loading and Memory Planning

3.1 Weight File Preprocessing

Ultra-large models are usually released as sharded files, for example splitting each layer or each expert module into an independent checkpoint. The following preprocessing steps are required before deployment:

• Verify the shape, precision, and checksum of each shard;
• Convert floating-point weights to a precision suitable for accelerator inference, such as FP16 or BF16;
• Build an index table for sparse weights, recording expert IDs, layer IDs, and offset addresses.

Loading the original weights directly often fails due to insufficient memory. A more reliable approach is to use streaming reads with on-the-fly conversion during loading: read one shard, convert one shard, write one shard to memory, and then release the temporary buffer in host memory. This way, even if the total weights exceed memory capacity, a complete copy does not need to be kept in host memory at the same time.

3.2 Multi-Card Sharding and Routing Mapping

When a single card cannot hold all experts, a sharding strategy along the expert dimension can be used. For example, different experts are placed on different accelerator cards, with each card additionally storing a copy of the routing table. During inference, the router first computes which experts should be activated for the input, then sends the input tensor to the device hosting the corresponding expert via inter-card communication.

In the Day-0 phase, this pattern is usually implemented in the simplest way possible. For example, a fixed mapping between experts and cards is used without introducing complex load-balancing strategies, ensuring that the pipeline can be verified end to end. The pseudocode is roughly as follows:

# For illustration only: dispatch inputs to the corresponding device by expert ID
def dispatch_tokens(hidden_states, expert_ids):
    device_map = build_static_expert_map()
    per_device_inputs = {}
    for expert_id in expert_ids:
        device = device_map[expert_id]
        mask = (expert_ids == expert_id)
        selected = hidden_states[mask]
        per_device_inputs.setdefault(device, []).append((expert_id, selected))
    return per_device_inputs

The actual implementation also needs to handle tensor consistency across devices, the choice of communication primitives, and whether backpropagation is enabled. When only performing inference validation, the logic can be greatly simplified.

3.3 Memory Usage Estimation

Deployment engineers need to estimate memory usage before startup. The main components include:

• The routing network and resident parameters;
• Currently loaded expert weights;
• Input and intermediate activation tensors;
• Communication buffers;
• Fixed overhead of the framework runtime.

It is recommended to reserve at least 15% to 20% of memory headroom during the Day-0 phase to cope with fragmentation and temporary peaks. If memory is still insufficient, try shortening the input sequence, reducing concurrent requests, or using a more aggressive weight offloading strategy.

4. Service Startup and Inference Pipeline Bring-Up

4.1 Key Steps in Service Startup

After model loading is complete, the model needs to be wrapped as an externally accessible service. This process generally includes:

• Initializing the inference engine and loading model configuration and weights;
• Building the computation graph and completing warmup inference;
• Starting HTTP or RPC interfaces and registering health checks;
• Setting request queues, timeouts, and concurrency limits.

In the Day-0 phase, synchronous interfaces are usually preferred to reduce the uncertainty introduced by asynchronous scheduling. The server executes inference immediately upon receiving a request and returns the result, which helps quickly determine whether the pipeline is correct. The warmup step is also important, because the first inference often triggers lazy initialization, operator compilation, and cache establishment, causing abnormally high response times. Warmup allows this cost to be paid upfront.

4.2 Text Inference Path

Text inference is the foundation for verifying whether the service is usable. A typical flow is as follows:

1. Tokenize the text and generate a token sequence;
2. Convert tokens to embedding vectors;
3. Run through multiple layers of sparse Transformer computation;
4. Output logits and sample from the vocabulary;
5. Decode into text and return.

For sparse models, the correctness of routing computation especially needs to be verified. If the gating results deviate from the standard implementation, the final output quality may degrade or even become completely unusable. Therefore, during the Day-0 phase, a small set of standard samples can be used for comparison testing, comparing the output token distribution against a reference implementation.

4.3 Image Input and Multimodal Extension

If the model supports image input, the visual encoding pipeline also needs to be connected. Common approaches include:

• Receiving image files or Base64 encoding;
• Resizing, normalizing, and splitting into image patches;
• Extracting features using a visual encoder;
• Concatenating visual features with text embeddings or fusing them via cross-attention;
• Entering the subsequent generation flow, similar to the text-only path.

In the Day-0 phase, the image pipeline usually only needs to verify three things: “can accept input, can encode, can participate in generation.” Image data typically consumes more memory than text, so it is recommended to start testing with low-resolution inputs and gradually increase image size after confirming correctness.

5. Basic Inference Validation and Common Troubleshooting

5.1 Output Correctness Validation

After the inference pipeline runs successfully, it is necessary to confirm that the output results are semantically reasonable. For Day-0 validation of ultra-large models, the following methods can be used:

• Prepare a set of test samples covering different task types, such as question answering, summarization, and code completion;
• Compare model outputs with reference implementation outputs and record token-level agreement rates;
• For image input scenarios, check whether descriptions are basically relevant to the image content;
• Observe the logits value range to avoid NaN or Inf.

When outputs are clearly abnormal, first check whether the issue is caused by differences in data preprocessing. For example, inconsistent text tokenization, incorrect image normalization parameters, or missing special tokens can all cause the model to “appear to run” while producing completely unusable results.

5.2 Common Engineering Issues

Common issues during adaptation include:

• Insufficient memory: Can be resolved by reducing batch size, shortening sequence length, or enabling weight offloading;
• Missing operators: Need to be supplemented or replaced with general implementations, prioritizing correctness over performance;
• Multi-card communication timeouts: Can increase the communication timeout threshold or adjust the expert sharding granularity;
• Excessive first-token latency: In the Day-0 phase this may be caused by insufficient warmup; increase the number of warmup samples at startup;
• Repeated or truncated output: Check generation configuration, such as temperature, top-p, stop tokens, and maximum length settings.

6. Evolution from Day-0 to Production Readiness

Day-0 adaptation is only the starting point. To make the service truly production-ready, continuous optimization is needed in the following areas:

6.1 Inference Performance Tuning

This includes operator fusion, KV cache optimization, continuous batching, dynamic shape support, and expert-parallel communication optimization. By reducing memory accesses and improving compute efficiency, end-to-end latency can be significantly reduced and throughput per unit time increased.

6.2 Service Elasticity and Fault Tolerance

Production environments need to consider capabilities such as model hot updates, request rate limiting, exception recovery, and canary releases. Ultra-large model services are usually deployed with multiple instances and traffic is scheduled through a unified access layer. When an instance becomes abnormal, it should be automatically removed and restarted.

6.3 Cost and Resource Utilization Optimization

Although sparse models have enormous total parameter counts, their actual computation may be lower than that of dense models with the same parameter count. Therefore, resource estimation should not simply extrapolate hardware requirements from parameter scale. A more reasonable approach is to perform capacity planning based on the actual number of activated parameters, batch size, and generation length, while also considering load characteristics when deciding whether multi-card deployment is needed.

6.4 Observability

It is recommended to integrate basic monitoring during the Day-0 phase, including memory usage, request latency, throughput, error rate, and inter-card communication time. These data are not only the basis for tuning but also provide reference for subsequent scaling.

---

Conclusion

Deploying and adapting ultra-large sparse models is a systems engineering effort that requires attention to model structure, memory planning, service workflows, and multimodal input pipelines simultaneously. The core of Day-0 adaptation is not to pursue high performance, but to bring up the full pipeline from model loading to inference return on the target accelerator cards in the shortest possible time, providing a stable baseline for subsequent tuning. For developers, understanding the sparse characteristics of the model, designing a reasonable multi-card sharding strategy, and doing a good job of output correctness validation and common troubleshooting are important prerequisites for completing this kind of work successfully.