Local Deployment of Very Large-Scale MoE Models: Memory Estimation and Tiered Cost Analysis

As open-source large models scale toward hundreds of billions or even trillions of parameters, the Mixture-of-Experts (MoE) architecture has become a mainstream approach for balancing compute cost and model capability. However, for a very large MoE model with roughly 2.8 trillion total parameters, local deployment creates significant hardware pressure and network communication challenges. This article starts with memory footprint estimation and then analyzes local deployment options and hardware costs across three tiers based on different business requirements.

1. Architectural Characteristics and Baseline Memory Estimation

Before discussing specific deployment plans, it is necessary to clarify the memory characteristics of the MoE architecture. Take a model with 2.8 trillion total parameters, 896 experts, and 16 activated experts per token as an example. The expert activation ratio is about 1.79%. This ratio affects inference compute consumption, but it does not reduce memory usage. In distributed deployment, all 896 expert weights must be fully loaded into the storage medium. At inference time, the routing mechanism retrieves the relevant experts for computation.

If the model uses quantization-aware training and adopts a mixed-precision scheme with 4-bit weights and 8-bit activations, the lower bound of raw data memory is about 1400 GB. According to the Open Compute Project specification, 4-bit quantization requires an additional 8-bit scaling factor for every 32 elements, bringing the actual weight memory to about 1487.5 GB. When non-4-bit tensors, metadata alignment, and runtime workspace are included, the lower bound for full deployment is extremely high.

By comparison, a common 671B-parameter MoE model requires about 400 GB of memory under 4-bit quantization, including activations, and can barely run on eight 80 GB accelerator cards. A 2.8-trillion-parameter model is more than four times larger, which means a single-node 8-card setup cannot load the weights at all. A multi-node cluster architecture becomes mandatory.

2. Tiered Deployment Plans and Cost Planning

Deployment requirements can be divided into three tiers based on different business scenarios, each with a different hardware strategy.

2.1 Basic Deployment: Run the Model Without Strict Speed Requirements

For scenarios where the goal is only to validate model capability and there is no strict latency requirement, the main objective is to reduce memory hardware cost.

• Technical approach: Use weight offloading. Store most less frequently used expert weights in large host memory or NVMe SSDs, while keeping only resident experts and the experts needed by the current hot token routing path in GPU memory.
• Hardware configuration: A full GPU cluster is not required. A small number of accelerator cards, such as four 80 GB cards, can handle core computation and scheduling. The host side should be equipped with 2 TB to 4 TB of high-speed DDR5 memory and multiple NVMe arrays as an extended memory pool.
• Cost estimate: The main cost is concentrated in host memory and storage. Hardware investment can be kept in the range of several hundred thousand RMB. The trade-off is that data movement over the PCIe bus introduces very high latency. Output speed may drop to only 1–2 TPS, making this option suitable mainly for offline batch processing.

2.2 Single-User Deployment: Target Output Speed of 25 TPS

25 TPS is a baseline for smooth single-user interaction. It requires loading the model weights while maintaining a relatively fluid computing pipeline.

• Technical approach: Load the full weights into accelerator memory and eliminate slow PCIe offloading. Use a hybrid strategy of tensor parallelism and expert parallelism to distribute the 896 experts evenly across multiple accelerator cards.
• Hardware configuration: To accommodate about 1.5 TB of weights plus KV cache, at least twenty 80 GB accelerator cards are required. A typical compute unit is one node with eight cards, so the deployment needs three nodes connected through a high-speed network.
• Cost estimate: The accelerator cards alone will cost more than 1.5 million RMB. Including server motherboards, high-wattage power supplies, and cooling equipment, the total investment is about 2 million RMB. This setup can guarantee per-node throughput, but leaves very little compute headroom.

2.3 High-Concurrency Deployment: 10 Concurrent Users with at Least 25 TPS per User

To sustain 25 TPS per user under 10 concurrent requests, the system must provide strong parallel computing capability and low-latency communication. This places stringent requirements on the underlying interconnect network.

• Core challenge: All-to-all communication in MoE architectures is extremely frequent. For every generated token, activations must be sent to the cards hosting the target experts after routing, and results must be sent back after computation. If communication crosses nodes, network latency directly consumes compute capacity.
• Hardware configuration: The official recommendation for this type of deployment is a supernode consisting of at least 64 accelerators. This means building an 8-node cluster. A supernode expands the communication domain through a high-speed direct interconnect protocol so that all-to-all communication stays within a single super communication domain as much as possible, or even within a single rack.
• Cost estimate: The hardware procurement cost for 64 accelerator cards is about 6 million RMB. With high-bandwidth switches, liquid-cooled data center retrofits, and other infrastructure, the total delivery cost can easily exceed 10 million RMB.

3. Cluster Operations and Cloud-Native Service Management

When a local deployment reaches 64 cards or more, traditional manual operations can no longer support stable service delivery. Organizations building such large inference clusters must introduce a cloud-native management system.

First, a container orchestration platform should be used to enable elastic scheduling and fault isolation of compute resources. If an accelerator card fails and causes expert routing errors, the scheduling system should quickly shift traffic to healthy nodes. Second, given the large volume of expert weight files, object storage should be used for unified management and version control, enabling concurrent pulling during node startup. Finally, a full monitoring and alerting system must be deployed to collect core metrics such as GPU utilization, memory fragmentation, cross-node communication latency, and KV cache hit rate in real time, with log services used for root cause analysis.

4. Summary and Selection Guidance

Local deployment of very large MoE models is a test of both hardware resources and engineering capability. For teams that only need to validate the pipeline, memory offloading is the most cost-effective choice. For single-user development scenarios, fully loading weights into memory with about 20 cards marks the threshold for smooth performance. For high-concurrency commercial use, a 64-card supernode cluster with a high-speed interconnect network is the minimum baseline. Organizations should carefully evaluate concurrency and latency requirements and weigh the sunk cost of self-built hardware against cloud-based elastic compute before making a decision.