Deploying Trillion-Parameter Sparse LLMs on Domestic GPUs: A Practical Adaptation and Optimization Guide

Trillion-Parameter Sparse LLMs Meet Domestic GPUs: Where to Begin Adaptation

As large language models cross the trillion-parameter threshold, the Mixture of Experts (MoE) architecture has become the mainstream choice due to its “sparse activation” characteristic. Recently, a domestic AI chip manufacturer announced that its integrated training and inference compute card and self-developed software stack achieved Day-0 adaptation for an open-source MoE model with over three trillion parameters on the day of its release. This successfully connected the foundational pipeline from model loading to multimodal (text and image) inference. This progress reflects the systemic engineering challenges that the domestic AI computing ecosystem must overcome when dealing with “ultra-large models + sparse computing.” This article breaks down the general adaptation pathways, core difficulties, and optimization strategies from a purely technical perspective.

1. The Three Major Hurdles of Deploying Trillion-Parameter Models

Taking a 3-trillion-parameter MoE model as an example, its deployment dilemmas center on three main areas:

1. The Memory Wall. Even using BF16 precision, the theoretical memory requirement for 3 trillion parameters exceeds 6TB (this accounts only for BF16 weights; KV Cache and activations require additional evaluation). Since single-card memory typically ranges from 64GB to 192GB, reliance on a combined strategy of Tensor Parallelism (TP), Pipeline Parallelism (PP), and Expert Parallelism (EP) is mandatory. TP splits single-layer weights by column or row across multiple cards (suitable for dense layers like Attention/FFN), while EP distributes different experts across GPUs along the expert dimension (dim=0). Their responsibilities are distinct and should not be confused.

2. The Communication Wall. The dynamic token routing in MoE architectures generates massive All-to-All communication volumes among experts. Cross-node network bandwidth often becomes the absolute bottleneck for inference latency.

3. The Scheduling Wall. Sparse activation invalidates traditional static computation graph compilation strategies, necessitating runtime dynamic scheduling and pooled memory management.

These challenges require hardware vendors to innovate simultaneously in hardware architecture and software abstraction layers.

2. Breakdown of the Four-Stage Day-0 Adaptation Pipeline

Day-0 adaptation means the hardware and software stack can seamlessly support a model on the day of its official open-source release. This requires deep collaboration with the community in advance and a mature, full-stack toolchain. The core pipeline includes:

1. Model Loading: More Than Just “Reading Files”

The original weights of a 3-trillion-parameter model are typically sharded into 2GB to 8GB files. The first hurdle of adaptation involves:

• Custom Deserializer: The standard load_state_dict is inefficient when dealing with thousands of shards and cannot leverage High Bandwidth Memory (HBM) characteristics. Vendors typically provide dedicated tools supporting parallel reading, asynchronous DMA copies, and direct memory mapping.
• Weight Reordering: To meet TP splitting requirements, layout reordering must be completed during the loading phase. For example, Attention QKV projection weights must be converted to column-parallel (split by feature dimension), while FFN weights can use row-parallel (split by hidden layer); EP requires reorganization along the expert dimension after loading.

Below is illustrative pseudo-code (with comments explaining applicable scenarios):

import glob, torch
def load_sharded_model(path, world_size, tp_degree, ep_degree):
    """
    Load sharded weights and reorder based on parallelism strategies.
    tp_degree: Tensor parallelism degree (applicable for column/row splitting in dense layers)
    ep_degree: Expert parallelism degree (splitting along the expert dimension dim=0)
    """
    shard_files = sorted(glob.glob(f"{path}/*.bin"))
    shards = [torch.load(f, map_location='cpu') for f in shard_files]
    # Merge all shards into a complete weight dictionary
    merged = {}
    for key in shards[0].keys():
        merged[key] = torch.cat([s[key] for s in shards], dim=0)
    for key, tensor in merged.items():
        if "experts." in key:
            # Expert weight shape [num_experts, hidden, ffn_size]
            # Expert Parallelism: Split along num_experts (dim=0) by ep_degree
            tensor = tensor.chunk(ep_degree, dim=0)  # Each GPU holds a subset of experts
        elif "attention." in key:
            # Tensor Parallelism: FFN weights split column-parallel (dim=1)
            hidden, ffn_size = tensor.shape  # Explicit variable definition
            tensor = tensor.chunk(tp_degree, dim=1)
    # Asynchronous transfer to GPU memory and establish virtual-to-physical address mapping
    return tensor

2. Service Startup: How to Compress Cold Start to Minutes?

For a 3-trillion-parameter model, service startup time can reach the scale of ten minutes depending on model size and cluster configuration (estimated value, not empirically tested). Key optimization methods include:

• Operator Fusion and Kernel Pre-compilation: Fusing “RMSNorm + SwiGLU + Matrix Multiplication” into a single GPU kernel reduces kernel launch overhead. Pre-compile FlashAttention variants for different sequence lengths into a cache directory.
• Hierarchical Memory Reservation: Adopt a two-level memory pool consisting of resident weight layers and dynamic activation layers. Lock all weights at startup and establish fixed virtual-to-physical address mappings to prevent fragmentation.

Recommended Practices Checklist:

1. Complete HBM pre-allocation and address locking for all weights before startup.
2. Pre-compile the Top-20 high-frequency operators into independent cache modules.

3. Inference Pipeline Validation: Accuracy Alignment is the Hard Metric

Conclusion 1: Dynamic routing Top-K must strictly align with the original implementation. The Top-K expert indices of the MoE gating network must ensure numerical stability of the topk operation and noise randomness consistency with the original implementation; otherwise, generation quality will drift. It is recommended to compare the gating output distribution histograms one by one.

Conclusion 2: Multimodal alignment requires dedicated kernel support. Image input involves ViT patch embedding and positional encoding. Dedicated im2col transformation kernels should be provided to improve feature extraction throughput, and the output accuracy deviation of the visual encoder should be verified layer by layer to be within ±1e-3.

Recommended Practices Checklist:

1. Use the same random seed to reproduce gating sampling and compare Top-K index distributions.
2. Perform pixel-level accuracy verification on at least 100 samples for the visual modality.

4. Software Stack Foundation: Instruction Set Mapping Determines Adaptation Speed

The maturity of a self-developed instruction set software stack is key to Day-0 efficiency. An excellent stack should provide high-level APIs close to industry-standard GPU programming models, allowing model authors to migrate code with minimal changes. Meanwhile, the compiler must possess dynamic shape automatic tiling and scheduling capabilities to avoid warp divergence.

3. From “Running” to “Running Well”: Three Major Optimization Directions

1. How to Co-optimize Memory and Communication?

When adopting a hybrid strategy of “Data Parallelism + Expert Parallelism,” DMA engines and multi-stream technology can be utilized to achieve complete pipelining of computation and communication: while the local node computes the current layer’s Attention, it prefetches the remote expert parameters required for the next layer. In inference scenarios, apply the PageAttention concept to manage KV Cache in pages. This requires the system to support on-demand virtual address page tables to avoid the waste caused by pre-allocating large chunks of KV Cache.

Recommended Practice: Enable dual-stream (Compute Stream + Comm Stream) parallelism, and set the KV Cache paging granularity to 256 tokens/page.

2. How to Dynamically Adjust Scheduling Strategies?

MoE models suffer from load imbalance, requiring the introduction of a priority-based task queue. When a specific expert backlog exceeds a threshold (e.g., queue depth > 512), a load balancing algorithm is triggered to reroute some tokens to idle expert replicas (Expert Replication) and dynamically adjust the batch size.

Recommended Practice: Set the expert load alert threshold to 1.5 times the average load, triggering automatic replication.

3. How to Control Quantization Accuracy?

Even 4-bit quantization can significantly reduce memory and bandwidth pressure. The adaptation layer needs to support mixed-precision schemes like FP8, INT8, and INT4. Note: Embedding layers and gating networks are extremely sensitive to quantization; it is recommended to retain BF16 precision. Quantization calibration datasets should closely match the training distribution to minimize accuracy loss.

Recommended Practice (Quantization Selection Process):

Layer Level	Recommended Precision	Description
Embedding / Gate	BF16	Sensitive to quantization, avoids accuracy collapse
Attention QKV	FP8	Controllable dynamic range, minimal loss
FFN Weights	INT4	Large volume, significant benefits
KV Cache	FP8	Noticeable improvement in inference throughput

4. Adaptation Considerations and Risk Warnings

When advancing adaptation in practice, three risks must be noted:

First, Memory Overflow (OOM). Large-scale models under peak load are prone to triggering OOM. It is recommended to reserve at least a 15% memory safety margin and configure monitoring alerts (e.g., warning if memory usage > 85%, mandatory throttling if > 95%).

Second, Driver and ISA Version Compatibility. Before adaptation, confirm the minimum driver and instruction set version requirements. Verify using tools similar to driver_version --check to avoid runtime exceptions.

Third, Data Security. Model weights and inference data transmitted across multiple nodes must be encrypted. It is recommended to use TLS 1.3 or SM4 link encryption and enable mutual authentication to prevent Man-in-the-Middle (MITM) attacks.

5. Industry Implications: Capability Boundaries and Selection References

This event conveys three signals: First, domestic hardware has demonstrated the ability to keep pace with cutting-edge model iterations in specific scenarios, and the universality of underlying software stacks has improved significantly. Second, the focus of ecosystem competition has shifted from single-card computing power to the linear scaling ratio and fault recovery capabilities of thousand-card clusters. Third, “high-quality adaptation” has become a new KPI for infrastructure selection, with enterprises focusing more on model coverage speed, inference throughput, and cost per token.

Common Mistakes in Deploying LLMs on Domestic GPUs

1. Confusing the splitting dimensions of Tensor Parallelism and Expert Parallelism, leading to memory waste or communication spikes.
2. Skipping accuracy alignment and going directly to production, causing uncontrollable generation quality.
3. Ignoring KV Cache page management, resulting in memory overflow during long-sequence inference.

How to Choose MoE Quantization Precision

The core principle is “do not quantize sensitive layers, aggressively quantize massive layers.” Conduct embedding calibration experiments first, then gradually expand to the entire model to avoid sudden accuracy collapse from a one-step approach.

For developers, it is crucial to deeply understand the mapping relationship between model architecture and hardware characteristics, and to master the combination logic of various parallel strategies. Current Day-0 adaptation mostly solves the “from nothing to something” phase. In the future, there remains substantial room for system-level optimization in areas like long-sequence inference, sparse attention optimization, and vector database integration. The combination of ultra-large parameter models and domestic GPUs is a long journey from “usable” to “highly efficient,” and the practices of every participant are paving the way for this marathon.