Deploying Trillion-Scale MoE Models Locally: Memory Estimation and Heterogeneous Inference Practices

As large language model parameter scales approach the trillion-level, the Mixture of Experts (MoE) architecture has become a mainstream choice in the industry. This is largely due to its characteristic of having a massive total parameter count while maintaining a small number of activated parameters per inference step. However, the enormous total parameter volume imposes extremely stringent hardware challenges for local, private deployments. Using a typical trillion-parameter MoE model as an example, this article deeply analyzes its memory occupation mechanisms and communication bottlenecks, and explores practical implementation strategies for heterogeneous inference in resource-constrained scenarios.

1. Underlying Memory and Storage Requirements of Trillion-Scale MoE Parameters

When evaluating the deployment cost of an ultra-large MoE model, the primary task is to clarify the relationship between “total parameters” and “activated parameters.” Taking a typical industry MoE model with a total of 2.8 trillion parameters as an example, its architecture usually contains hundreds to nearly a thousand independent expert networks. During inference, each token activates only a minuscule fraction of them (e.g., 16 experts, which is an activation rate of less than 2%).

1. Activation Ratio Does Not Equal Memory Occupation Ratio

A common misconception is: “Since less than 2% of the experts are activated each time, does the required memory only need to be 2% of the total parameters?” The answer is no.

The inference logic of the MoE architecture requires the model to be capable of routing the current token to any 16 out of 896 experts at any given moment based on the routing matrix. This means that before inference begins, all 2.8 trillion parameters must be fully loaded into addressable storage. The activation ratio only affects the floating-point operations (compute consumption) of a single forward pass, while the lower limit of memory occupation is entirely determined by the total parameter count.

2. Bare Data Storage Limits Under Low-Bit Quantization

To reduce the massive static memory footprint, large models often introduce quantization-aware training during the SFT (Supervised Fine-Tuning) phase. Assuming a model adopts an MXFP4 weight and MXFP8 activation scheme, we can rigorously calculate its memory lower limit:

• Weight Bit-width: In the MXFP4 format, each element occupies 4 bits. However, in the Open Compute Project (OCP) specification, every 32 4-bit elements require an additional 8-bit scale factor. Therefore, the effective bit-width is 4 + (8/32) = 4.25 bits.
• Bare Data Volume: 2.8 trillion parameters × 4.25 bits ÷ 8 bits/Byte ≈ 1487.5 GB.

It is important to note that this is purely the lower limit for weight data. In actual deployment, you must also account for non-4-bit quantized tensors, framework metadata, memory alignment overhead, runtime workspaces, and dynamically growing KV caches. Consequently, the actual memory demand will far exceed 1500GB.

3. Comparison with Conventional 100-Billion-Scale Models

To intuitively grasp this scale, we can compare it with common 100-billion-scale models in the industry (such as a 671B parameter model). In FP8 format, its main parameters plus MTP (Multi-Token Prediction) parameters require about 685GB of memory. Even using AWQ 4-bit quantization, it requires approximately 335GB for weights plus 37GB for activations, totaling nearly 400GB just to barely run on 8 high-end GPUs (e.g., 80GB VRAM per card, 640GB total).

The scale of a 2.8 trillion parameter model is more than four times that of a 100-billion parameter model, meaning that traditional single-machine multi-GPU deployment solutions are entirely obsolete.

2. Super-Node Architecture: Breaking Through MoE Communication Latency Bottlenecks

Faced with a memory demand exceeding 1500GB, splitting the model across dozens or even hundreds of accelerators is inevitable. However, this introduces fatal communication latency issues.

1. Extreme Sensitivity of All-to-All Communication

During the forward pass of an MoE architecture, each token goes through the following steps:

1. The network computes routing probabilities to determine which 16 experts the current token should go to.
2. The system initiates All-to-All communication, transferring the token’s activation values across cards to the accelerators hosting the target experts.
3. After the experts complete their computations, the results are transferred back to the original card via All-to-All communication.

With 896 experts distributed across different compute nodes, the network latency from this cross-node transfer will directly consume the compute dividends of the accelerators. The stronger the compute power, the larger the proportion of communication wait time, creating a severe “bucket effect.”

2. Super-Node Networking Solution

To solve this issue, it is recommended to deploy a “super-node” architecture consisting of 64 or more accelerators when deploying such ultra-large MoE models. By utilizing NVLink or equivalent high-speed interconnect buses to enlarge the communication domain, all expert routing transfers can be confined within a single super-node. This avoids cross-switch network jitter, ensuring high bandwidth and low latency.

3. Tier-1 Practice: Low-Cost “Just Make It Run” Solution Based on CPU-GPU Heterogeneous Architecture

For teams without super-node clusters and only a few consumer-grade or single-way enterprise GPUs, if the goal is simply to “get the model running without caring about throughput speed,” heterogeneous inference is the best choice.

1. Core Decoupling Philosophy of Heterogeneous Inference

The core of heterogeneous inference frameworks lies in the physical isolation of “compute-intensive” and “storage-intensive” tasks:

• GPU Handles Attention: Every layer of an MoE has a shared attention component (such as the KDA architecture and Gated MLA components). This part is highly compute-intensive and demands extreme bandwidth, making it best suited to remain on the GPU.
• CPU Handles Sparse Experts: The 896 experts, which occupy the vast majority of storage but only activate 16 at a time, are entirely offloaded to the CPU’s standard memory (DRAM). Large-memory servers (e.g., TB-level memory configurations) are used for feed computations.

2. Hardware Configuration and Performance Expectations

Using open-source heterogeneous inference optimization frameworks, it is possible to run 100-billion or trillion-scale models on a dual-socket high-performance server CPU paired with a single high-end consumer GPU (such as the RTX 4090 series).

Below is a schematic heterogeneous routing allocation code snippet demonstrating how to offload different operators:

# Pseudocode: MoE routing allocation strategy based on a heterogeneous computing framework
class HeterogeneousMoEConfig:
    def __init__(self):
        # Compute-intensive operators: bound to GPU
        self.gpu_layers = ["kda_attention", "gated_mla", "router"]
        # Storage-intensive operators: 896 experts reside entirely in CPU memory
        self.cpu_layers = ["expert_0", "expert_1", ..., "expert_895"]
    def forward(self, hidden_states):
        # 1. GPU computes attention and routing weights
        hidden_states = self.run_on_gpu(self.gpu_layers, hidden_states)
        topk_experts = self.get_topk_router(hidden_states, k=16)
        # 2. Extract and compute sparse experts in CPU memory
        # Utilize CPU multi-core parallelism to process the few activated experts
        expert_output = self.run_on_cpu_experts(topk_experts, hidden_states)
        # 3. Aggregate results back to GPU
        return self.run_on_gpu("aggregate", expert_output)

Under this scheme, although it cannot achieve production-grade high throughput, the Prefill phase may reach several hundred Tokens/s, and the Decode phase can maintain a level of a little over ten Tokens/s. This holds extremely high practical value for internal algorithm validation, Prompt effect debugging, or small-scale offline data processing, with hardware costs being only a fraction of a traditional cluster.

4. Cloud Elastic Deployment and Engineering Recommendations

For enterprise teams that need to balance performance and cost, building a massive hardware cluster locally requires not only a huge initial investment but also carries heavy burdens of later maintenance and idle rates. In this case, relying on general cloud platform infrastructure is a more rational choice.

When deploying ultra-large MoE models in the cloud, the following engineering practices are recommended:

1. On-Demand Compute Resource Allocation: Deploy inference services on cloud container platforms. Utilize the cloud’s high-performance networks (such as RDMA-supported environments) to temporarily form virtual super-nodes, enabling elastic scaling based on traffic peaks and valleys.
2. Weight Storage Separation: Store the massive model weights (nearly 1500GB) in cloud object storage. Mount or pre-warm them directly to compute nodes via high-speed intranet channels to avoid permanently occupying the expensive local disks of GPU nodes.
3. Monitoring and Logging Loop: Due to the complexity of MoE routing mechanisms, integrate cloud logging services and monitoring alerts to track All-to-All communication latency ratios, CPU memory throughput bottlenecks, and KV Cache growth levels per inference in real-time, continuously optimizing scheduling strategies.

Conclusion: The local deployment of ultra-large MoE models is not merely a stacking of hardware funds, but an extreme test of system architecture, communication topology, and heterogeneous compute scheduling. From super-node clusters pursuing ultimate throughput to CPU-GPU heterogeneous solutions pursuing ultimate cost-effectiveness, developers must rationally choose the engineering path that best matches the latency and throughput requirements of their actual business scenarios.