Local Deployment Practices and Cost Analysis for a Trillion-Parameter MoE Large Model

As open-source large models scale toward trillion-level parameter counts, local deployment is no longer simply a matter of downloading and running a checkpoint. It has become a complex systems engineering effort involving memory limits, interconnect bandwidth, and heterogeneous computing. This article uses a typical 2.8-trillion-parameter Mixture-of-Experts (MoE) model as a reference to examine the underlying resource requirements and practical deployment options at different levels.

1. The Memory Wall: Extreme Storage Pressure from 2.8 Trillion Parameters

To understand deployment cost, it is first necessary to clarify the relationship between parameter count and actual storage footprint. For an MoE model with 2.8 trillion total parameters, memory usage cannot be calculated by simply multiplying the parameter count by the byte size of a single precision format.

1.1 Parameter Distribution Characteristics of MoE Architectures

This model uses 896 expert networks, with only 16 experts activated per token during inference, giving an activation ratio of roughly 1.79%. However, this does not mean that only a small fraction of parameters needs to be loaded. In MoE deployment, memory usage is strictly determined by the total parameter count, while the activation ratio only affects compute consumption during inference. In practice, all 896 expert weights must be fully loaded into the storage medium, and the inference engine then routes activations to the corresponding experts for computation based on dynamic routing.

1.2 Quantization-Aware Storage Lower Bound

To reduce the storage barrier, the model adopts quantization-aware training during the SFT (Supervised Fine-Tuning) stage, using MXFP4 weights with MXFP8 activations. Converting 2.8 trillion parameters to 4-bit gives a theoretical lower bound of approximately 1,400 GB for the raw data.

However, that is only the ideal case. Under the OCP specification, every 32 4-bit elements in MXFP4 format require an additional 8-bit scale factor. The calculation is as follows:

• Combined bit width of weights and auxiliary data: approximately 4.25 bits (4-bit weights plus 0.25-bit scale)
• Actual storage required: 2.8T × 4.25 bits ÷ 8 ≈ 1,487.5 GB

This figure of nearly 1,500 GB covers only the raw weight data. During actual operation, additional space is required for non-4-bit tensors, routing metadata, memory alignment padding, runtime workspace, and dynamically growing KV cache. For comparison, a commonly seen 671B-parameter model typically requires around 400 GB of storage in FP8 or 4-bit quantized form to run on eight high-performance accelerators. At 2.8T parameters, this model is more than four times larger, which directly determines the hardware baseline for full-scale deployment.

2. The Communication Domain: Why a Supernode Architecture Is Mandatory

The official recommendation for deploying such models calls for a supernode consisting of at least 64 accelerators. The reason is that for ultra-large-scale MoE inference, beyond the memory wall, there is also a severe communication wall.

2.1 All-to-All Communication Latency

The 896 experts are distributed across multiple accelerator cards, often spanning dozens of devices. For every generated token, after attention computation is completed, the system must transfer activations through All-to-All communication to the physical cards where the target experts reside, perform the expert computation, and then transfer the results back. This high-frequency cross-card data movement can easily cause communication latency to consume the available compute, leading to a situation where computation is fast but data transfer is slow.

2.2 Building a High-Speed Interconnect Domain

To mitigate this problem, the single communication domain must be expanded. A supernode built with NVLink or equivalent high-speed interconnect technology can reduce cross-card communication latency to an extremely low level, ensuring that the activation transfers described above do not leave a single supernode communication domain. This is the fundamental reason why ordinary server clusters cannot handle such workloads and why a supernode architecture is required.

3. Deployment Tier 1: Heterogeneous Computing — A Minimal Viable Approach

When budgets are limited and inference speed is not a priority, with single-digit tokens per second being acceptable, and the only goal is to make the model produce tokens, a CPU-GPU heterogeneous inference approach can be used.

3.1 Core Idea of Heterogeneous Inference

Each MoE layer contains a shared attention component that is compute-intensive but relatively small in parameter count, making it suitable for processing in high-bandwidth GPU memory. The 896 sparse experts, which account for the vast majority of storage but only a small activated subset per step, can be offloaded entirely to CPU system memory and computed on large-memory servers.

3.2 Technical Practice and Expected Performance

In common open-source framework implementations, the typical approach is to pair high-performance CPUs, such as dual-socket multi-core server processors, with consumer-grade or mid-range accelerators. For models in the hundreds-of-GB range using Q4 quantization, optimized heterogeneous inference frameworks can achieve over two hundred tokens per second in the prefill phase and about 14 tokens per second in the decode phase. However, such figures are usually accompanied by optimization conditions such as constrained expert selection strategies.

For a 2.8-trillion-parameter model with a footprint approaching 1,500 GB, the heterogeneous approach requires a server with 1.5 TB to 2 TB of system memory. The following is a simplified example of the resource allocation logic for heterogeneous inference:

# Pseudocode: resource allocation logic for heterogeneous inference
class HeterogeneousInferenceEngine:
    def __init__(self, model_config):
        self.gpu_memory_pool = AcceleratorMemory(size="80GB")
        self.cpu_memory_pool = HostMemory(size="2048GB")
        # Load shared attention and gating networks into GPU memory
        self.gpu_layers = load_layers(
            model_config.shared_attention, 
            device="gpu", 
            dtype="fp8"
        )
        # Offload all 896 sparse experts to CPU memory for on-demand activation
        self.cpu_experts = load_experts(
            model_config.moe_experts,
            device="cpu",
            dtype="mxfp4"
        )
    def forward(self, token):
        # GPU handles the compute-intensive attention calculation
        attn_output = self.gpu_layers.compute(token)
        # Router determines the 16 active experts, delegated to CPU
        active_experts = router(token)
        return self.cpu_experts.compute(attn_output, active_experts)

4. Deployment Tier 2: High-Throughput Deployment on a Supernode Cluster

When an organization needs to handle high-concurrency requests and pursue throughput ranging from tens to over a hundred tokens per second, the single-node heterogeneous approach is no longer sufficient, and a supernode cluster deployment becomes necessary.

4.1 Resource Pooling and Tensor Parallelism

In a supernode composed of 64 accelerators, the total memory pool can reach several terabytes, which is sufficient to shard the 1,500 GB of weights evenly and keep them resident in accelerator memory. Through a combination of tensor parallelism (TP) and expert parallelism (EP), the 896 experts are distributed across different nodes, with the high-speed interconnect handling communication between shards.

4.2 Elastic Cloud-Native Deployment and Cost Optimization

For clusters of this scale, many organizations cannot easily absorb the upfront hardware procurement cost. Cloud-based container platforms and heterogeneous compute instances enable elastic scheduling of compute resources. During low-traffic periods, some compute nodes can be released, while the supernode cluster can be scaled up quickly during peak periods. Combined with cloud-native object storage for unified weight file management, this approach can significantly reduce cold-start time and overall TCO.

5. Cost Estimates and Conclusion

Based on the analysis above, the cost of locally deploying a 2.8-trillion-parameter MoE model is concentrated in two tiers:

Minimal Heterogeneous Tier:
The hardware cost mainly consists of one multi-socket server with more than 2 TB of system memory plus one or two consumer-grade accelerators. Total hardware investment can be kept in the range of tens of thousands of RMB. The trade-off is extremely slow decoding speed, making it suitable only for individual research or offline batch processing.

Full-Scale Supernode Tier:
For industrial-grade usability, at minimum a supernode cluster with 64 high-end accelerator cards, such as 80 GB variants, is required, with total accelerator memory exceeding 5 TB to accommodate weights and KV cache. The cost of the accelerators alone is in the tens of millions of RMB. When high-speed switches, storage servers, and data center operations are included, the initial investment is extremely high.

In summary, locally deploying a trillion-parameter MoE model is not merely a matter of technical tuning. It is an extreme challenge involving system architecture, memory bandwidth, and interconnect communication. For most organizations, leveraging large-scale heterogeneous compute pools and specialized inference acceleration services in the cloud, with on-demand usage, is likely the most practical way to balance business responsiveness and cost efficiency.