Local Deployment of Trillion-Parameter MoE Models: Memory Requirements and Cost Analysis

1. Introduction: Why Local Deployment Comes Down to Arithmetic

In recent years, MoE (Mixture of Experts) models with parameter counts exceeding one trillion have become increasingly prominent. Thanks to their sparse activation design, these models use only a small fraction of their total parameters during inference, which appears to make them computationally efficient. However, when enterprises or developers seriously consider deploying them privately, they run into a harsh reality: whether a model can run at all depends first on whether it can fit in memory at all.

This article uses a typical 2.8-trillion-parameter MoE model as its technical backdrop and works through the complete local deployment chain layer by layer—from parameter architecture, quantization formats, memory overhead, and communication overhead to hardware selection and final cost. All analysis is based on widely accepted technical principles and public quantization specifications, and does not promote any specific vendor’s products.

---

2. Core Mechanics of MoE Architecture: Total Parameters ≠ Activated Parameters

2.1 Parameter Scale and Expert Routing

In a typical trillion-scale MoE model, the total parameter count can reach 2.8 trillion (2.8T), composed of hundreds or even thousands of “expert” subnetworks. In a common design, the model contains approximately 896 experts, and each token selects 16 experts through a router network during forward inference.

A key distinction must be made clear here:

Metric	Meaning	Value (Example)
Total parameters	Sum of all expert weights	2.8T
Activated experts	Experts actually invoked per token	16
Expert activation ratio	Activated experts / total experts	16/896 ≈ 1.79%
Parameter activation ratio	Parameters involved in computation / total parameters	≈ 5% (since each expert is itself a large network)

The expert activation ratio (1.79%) is not equal to the parameter activation ratio. Each expert may itself contain billions of parameters, and the combined parameter count of 16 experts far exceeds the intuitive estimate of 1.79%. This means that although floating-point computation during inference is greatly reduced, the weights of all experts must be fully loaded into storage.

2.2 Architectural Backbone: Evolution of Attention Mechanisms

Current mainstream trillion-scale MoE models generally adopt improved attention mechanisms—for example, introducing gating mechanisms on top of Delta Attention along with MLA (Multi-head Latent Attention) components to reduce the memory footprint of the KV Cache. This architectural design can significantly reduce memory pressure in long-context scenarios during inference, but it does not fundamentally change the total weight storage requirement.

---

3. Precise Memory Estimation Under Quantization Formats

3.1 Why Quantization Is Unavoidable

If the 2.8T parameters were stored in FP16 (16-bit floating point), the raw data would require:

2.8T × 2 bytes = 5,600 GB ≈ 5.5 TB

This far exceeds the memory capacity of any single server. As a result, quantization-aware training (QAT) has become a standard process: low-precision inference is simulated during training so that the model retains accuracy at low bit widths.

3.2 Real Footprint of the MXFP4 Format

A widely discussed 4-bit mixed-precision floating-point format (such as MXFP4) has the following characteristic: every 32 4-bit elements share a single 8-bit scale factor. Calculated according to the general specification:

Total bits per 32 elements = 32 × 4 bit + 8 bit = 136 bit
Equivalent per element = 136 / 32 = 4.25 bit
Total weight footprint = 2.8T × 4.25 bit ÷ 8 = 1,487.5 GB ≈ 1.45 TB

This is only the theoretical lower bound for pure weight data. Actual deployment must additionally account for:

• Non-4-bit tensors: some layers (such as embedding layers, normalization layers, and attention weights) may still be stored in FP8 or FP16;
• Metadata and indices: expert routing tables, quantization parameter tables, and so on;
• Memory alignment overhead: GPU/NPU memory is typically aligned to specific granularities, so actual usage is often 5%–15% higher than the theoretical value;
• Runtime workspace: inference frameworks require temporary buffers to hold intermediate activations;
• KV Cache: grows dynamically with sequence length and can reach tens of GB or more in long-context scenarios.

Taken together, the actual deployment requirement for weights alone is roughly 1.6 TB to 1.8 TB, and with runtime overhead, total memory requirements easily exceed 2 TB.

3.3 Side-by-Side Comparison with a Similar Model

Using a common 671B-parameter MoE model as a reference:

• In FP8 format, main parameters (671B) + multi-token prediction parameters (about 14B) ≈ 685 GB;
• With 4-bit quantization, weights are about 335 GB plus about 37 GB of activations, totaling roughly 400 GB;
• It can run fully on 8 high-bandwidth memory accelerator cards (80 GB each, 640 GB total).

The 2.8T model is more than 4 times the size of the 671B model. Even accounting for the compute savings from sparse activation, the hard storage requirement is still at least 4 times higher. This directly determines the barrier to local deployment.

---

4. Communication Bottleneck: The Hidden Cost of MoE Inference

4.1 Why All-to-All Communication Is Critical

The core challenge of MoE inference lies not only in computation but also in communication. With 896 experts distributed across multiple accelerator cards, the routing result for each token may point to different experts on different cards. This means:

1. Forward pass: the token’s hidden state must be sent via All-to-All communication to the card hosting the target expert;
2. Computation: the expert on the target card completes the computation;
3. Return pass: the result is then gathered back to the original card via All-to-All communication.

This process occurs at every layer. If network bandwidth is insufficient or latency is too high, communication time will directly consume the compute gains, causing GPU utilization to drop significantly.

4.2 Requirements for Hardware Topology

Industry practice generally suggests that deploying MoE models at this scale requires:

• A supernode composed of 64 or more accelerator cards, with inter-card interconnect bandwidth on the order of hundreds of GB/s;
• A topology using NVLink, high-bandwidth private interconnect, or equivalent capabilities, rather than ordinary Ethernet;
• In cloud environments, similar results can be achieved through container platforms or bare-metal instances paired with high-speed RDMA networks.

High-performance computing clusters offered by cloud providers such as Baidu AI Cloud typically provide this kind of interconnect capability, but specific configurations must be customized based on actual model scale and inference throughput requirements.

---

5. Cost Estimation: What Does Local Deployment Actually Cost?

5.1 Hardware Cost Breakdown

Using current mainstream high-bandwidth memory accelerator cards (80 GB per card) as an example, meeting a 2 TB+ memory requirement requires at least:

Minimum card count = 2,048 GB ÷ 80 GB ≈ 26 cards (memory only)
Accounting for KV Cache, workspace, and redundancy → 32–40 cards in practice
Accounting for communication topology and redundancy → 64+ cards recommended

Single-card market prices fluctuate considerably with supply and demand. Assuming tens of thousands of RMB per card:

Configuration	Card count	Estimated hardware cost	Use case
Minimum viable	32 cards	On the order of one million RMB	Short sequences, low concurrency
Recommended	64 cards	On the order of two million RMB	Moderate concurrency, long context
High-availability cluster	128 cards	On the order of four million RMB	High concurrency, production environments

5.2 Other Hidden Costs

• Facilities and power: high-density GPU clusters can draw tens of kilowatts, requiring supporting cooling and power infrastructure;
• Network equipment: high-speed switching equipment is costly;
• Operations staffing: cluster tuning, troubleshooting, and model updates require a specialized team;
• Storage: the model file alone is about 1.5 TB+, and high-speed storage is also needed for loading and checkpoints;
• Software licensing: inference frameworks and cluster scheduling systems may incur additional fees.

5.3 Cloud Deployment as an Alternative

For teams without the conditions to build their own data centers, on-demand access through a cloud provider’s GPU instances or container services can significantly reduce upfront investment. Under a pay-as-you-go model, the cost of a single inference may range from a few RMB to tens of RMB, depending on model scale, concurrency, and inference duration. AI compute services offered by Baidu AI Cloud cover a variety of accelerator card specifications and cluster forms, and can serve as an alternative for cost comparison against local deployment.

---

6. Summary and Recommendations

1. Memory is the hard threshold: even after 4-bit quantization, a 2.8T-parameter MoE model requires about 1.5 TB for weights alone, and with runtime overhead the actual requirement exceeds 2 TB of storage. Local deployment must use a multi-card cluster.
2. Communication is the hidden bottleneck: All-to-All communication is extremely sensitive to inter-card interconnect bandwidth. Ordinary Ethernet environments cannot meet the requirement; a high-bandwidth interconnect topology is essential.
3. Costs are in the millions: hardware investment for a GPU cluster of 64 cards or more typically exceeds one million RMB, with ongoing costs for power, networking, and operations on top.
4. Quantization is essential: low-bit formats such as MXFP4 compress storage requirements to below one quarter of FP16, but quantization accuracy loss must be validated on actual tasks.
5. Cloud deployment can serve as a过渡 step: for early validation or small-to-medium-scale applications, on-demand cloud GPU resources are more economical and flexible than self-built infrastructure.

Local deployment of trillion-scale MoE models is a systems engineering effort involving architecture understanding, hardware selection, network planning, and cost control. The estimation framework provided here can help technical decision-makers quickly establish a baseline understanding, on top of which they can make more precise investment plans based on specific business requirements.