Local Deployment of Trillion-Parameter MoE Models: Memory Assessment and Heterogeneous Inference

As large language models advance toward trillion-scale parameter counts, the Mixture of Experts (MoE) architecture has become the industry standard due to its sparse activation characteristics. Yet the enormous total parameter count presents unprecedented memory pressure for local deployment. This article uses a MoE model with 2.8 trillion total parameters as a case study to analyze deployment thresholds, hardware cost estimates, and a viable low-cost heterogeneous inference strategy.

1. Rethinking Memory Requirements for Trillion-Scale MoE Architectures

For a MoE model with 2.8 trillion total parameters, the architecture consists of 896 expert networks, with only 16 experts activated per token, yielding an activation ratio of approximately 1.79%. The backbone network integrates Delta Attention and Attention Residuals mechanisms, complemented by a Gated MLA component to improve attention computation efficiency.

1.1 Memory Footprint Is Determined by Total Parameters, Not the Activation Ratio

Many developers mistakenly assume that a low activation ratio of 1.79% can significantly reduce memory requirements. In reality, the activation ratio only affects computational cost (FLOPs) during inference. The lower bound of memory usage is determined by the total parameter count. During MoE inference, the system must load the weights of all 896 experts into addressable storage. At runtime, the dynamic routing mechanism selects and invokes the 16 activated experts for computation.

1.2 Quantization-Aware Training and Actual Storage Size

This model is tightly coupled with Quantization-Aware Training (QAT) from the supervised fine-tuning (SFT) stage onward, using MXFP4 weights with MXFP8 activations. Converting 2.8 trillion parameters to 4-bit precision produces roughly 1400GB of raw data. However, according to the OCP specification, the MXFP4 format requires one 8-bit scaling factor for every 32 4-bit elements. After adding scaling factors, the actual weight size becomes approximately 2.8T x 4.25 bit / 8, or about 1487.5GB. This accounts only for weights and excludes non-4-bit tensors, routing metadata, memory alignment padding, runtime workspace, and the dynamically allocated KV cache.

2. Hardware Cost Estimates and Supernode Communication Bottlenecks

2.1 Memory Baseline in Comparative Context

To better understand the scale, consider the widely used 671B-parameter MoE models. At FP8 precision, the main parameters plus MTP parameters occupy approximately 685GB of memory. Even with AWQ 4-bit quantization, roughly 335GB of weights plus 37GB of activations are required, totaling nearly 400GB just to run on 8 high-performance GPUs with 80GB of memory each.

The 2.8-trillion-parameter model discussed here is more than four times larger than that baseline. This means a single 8-GPU server, typically offering 640GB to 960GB of total memory, cannot accommodate the nearly 1500GB of weights in a full deployment.

2.2 Supernode Architecture and All-to-All Communication Challenges

For the 1500GB memory requirement, official guidance typically recommends deploying on a supernode cluster with more than 64 accelerators. Beyond memory distribution, the core challenge lies in MoE communication mechanics.

Large-scale MoE inference is extremely sensitive to All-to-All communication bandwidth. With 896 experts distributed across multiple GPUs, each token, after routing computation, must have its activations transferred to the physical card hosting the target expert. After computation, results are transferred back to the originating card. This cross-card data movement occurs at a very high frequency, and communication latency can directly erode GPU compute advantages. Therefore, high-speed interconnect technologies such as NVLink must be used to expand the communication domain as much as possible, ensuring that data movement stays within a single supernodes communication domain and avoiding microsecond-level latency penalties from cross-node networks such as InfiniBand or RoCE.

3. Low-Cost Heterogeneous Inference: CPU-GPU Collaboration

If an organization cannot afford the high hardware cost of a 64-card supernode but still needs to run the model locally, even at the expense of inference speed, a CPU-GPU heterogeneous inference approach is feasible. The core principle is computation-density-based hierarchical offloading.

3.1 Architecture Decomposition and Workload Partitioning

Within each MoE layer, there are two types of computational modules:

• Shared attention components: Including Delta Attention and Gated MLA, these are compute-intensive tasks with high bandwidth demands. They remain on the GPU, leveraging high-bandwidth memory (HBM) for fast computation.
• Sparse expert networks: The 896 experts account for the vast majority of storage, approximately 1400GB or more, but only 16 are activated per inference step. These are memory-bound tasks. They are fully offloaded to CPU memory (DRAM), relying on large-memory servers for massive storage capacity.

3.2 Performance Expectations and Tuning Strategies

Using open-source heterogeneous inference frameworks, the model can run on standard servers equipped with large memory capacity. For example, when running smaller quantized models in the 60-billion-parameter range on mainstream dual-socket servers with a single consumer-grade GPU, the prefill stage can reach several hundred tokens per second, while the decode stage runs at around a dozen tokens per second.

For the 2.8-trillion-parameter model, however, decode speed may drop to single digits because CPU memory bandwidth, typically several hundred GB/s, is far lower than GPU HBM bandwidth of several TB/s. To mitigate this, developers often introduce an expert selection strategy: by predicting routing distributions, high-probability expert data can be prefetched into GPU memory ahead of time, masking some of the PCIe data transfer latency from CPU to GPU.

The pseudocode below illustrates the data offloading and scheduling logic in a heterogeneous inference architecture:

class HeterogeneousMoELayer:
    def __init__(self, config):
        # Compute-intensive attention layer stays on the GPU
        self.attention = AttentionKDA(config).to('gpu')
        # Large expert network is offloaded to CPU memory
        self.experts = MoEExperts(config, count=896).to('cpu')
    def forward(self, tokens):
        # Step 1: Attention computation on the GPU
        attn_output = self.attention(tokens)
        # Step 2: Router computation on the GPU
        router_logits = self.router(attn_output)
        selected_experts = topk(router_logits, k=16)
        # Step 3: Expert computation scheduled on the CPU
        # with asynchronous transfer to the GPU
        expert_outputs = []
        for token, expert_idx in zip(attn_output, selected_experts):
            # Retrieve the corresponding expert weights from CPU memory
            # Production frameworks use async prefetch or pipelining
            # to hide data transfer latency
            out = self.experts[expert_idx](token)
            expert_outputs.append(out)
        # Step 4: Result aggregation and residual connection on the GPU
        return self.aggregate(expert_outputs) + attn_output

4. Summary and Deployment Recommendations

Local deployment of trillion-parameter MoE models is an extreme balancing act between memory capacity and bandwidth. For production-grade high throughput, a supernode cluster built on high-speed interconnect is the only viable solution. For functional validation or latency-insensitive offline batch processing, a CPU-GPU heterogeneous offloading approach offers a highly cost-effective path to deployment. When planning deployments, organizations should first evaluate concurrency requirements and latency tolerance, then select an appropriate hardware configuration and quantization precision strategy.