Local Deployment of Ultra-Large MoE Models: Memory Estimation and Hardware Cost Analysis

In recent years, the Mixture-of-Experts (MoE) architecture has demonstrated outstanding scalability in the field of ultra-large language models. By sparsifying parameters, MoE models can significantly increase total parameter capacity and knowledge density without proportionally increasing per-token inference compute. However, local deployment of such models imposes extremely strict requirements on GPU memory capacity and cross-node communication bandwidth. This article uses a typical 2.8-trillion-parameter MoE model as an example to examine the memory estimation logic and hardware cost assessment involved in deployment.

1. Memory Footprint Logic for Trillion-Scale MoE Architectures

To evaluate deployment cost, the first task is to accurately estimate the minimum GPU memory requirement. For an MoE model with 2.8 trillion total parameters, the architecture typically includes hundreds or even close to a thousand expert sub-networks. In one representative architecture, the system contains 896 independent experts, but only 16 experts are activated per token during inference, resulting in an expert activation ratio of approximately 1.79%.

A core concept needs to be clarified here: the expert activation ratio only affects floating-point operations (FLOPs) consumption during inference. It does not affect the baseline GPU memory footprint. In real-world MoE deployment, routing decisions are made dynamically at runtime. The computing system must keep the weights of all 896 experts fully resident in memory so that it can instantly invoke the target experts for any token according to the routing table. Therefore, memory estimation must be based on total parameters (2.8 trillion), not activated parameters.

2. Mixed-Precision Quantization and a Detailed Lower-Bound Memory Estimate

To compress the enormous parameter volume, the industry commonly introduces quantization-aware training during the supervised fine-tuning (SFT) stage. Current state-of-the-art low-bit deployment schemes typically store weights in MXFP4 format while handling activations in MXFP8.

Converting 2.8 trillion parameters to 4-bit storage, which equals 0.5 bytes per parameter, gives a theoretical lower bound of approximately 1,400 GB for raw data. However, under practical hardware specifications such as the OCP specification, an MXFP4 data block is not composed solely of 4-bit elements. Every 32 4-bit elements must be paired with one 8-bit scaling factor to maintain numerical stability.

Based on this specification, a rigorous calculation is as follows:

• Parameter elements: 2.8T × 4 bit = 11.2 Tb
• Scaling factors: 2.8T × (1/32) × 8 bit = 0.7 Tb
• Combined bit consumption: 11.2 + 0.7 = 11.9 Tb
• Converted to bytes: 11.9 Tb ÷ 8 ≈ 1,487.5 GB

This 1,487.5 GB is only the static lower bound for weight data. In a real inference scenario, the system must also accommodate non-4-bit tensors, tensor metadata, memory alignment overhead, runtime computation workspaces, and dynamically growing KV cache. As a result, real-world GPU memory demand will clearly exceed this figure.

3. Reference Comparison and Scale Awareness

To understand this scale more intuitively, it helps to look at the widely deployed 671B-class large MoE models. In FP8 format, the main parameters combined with a multi-token prediction (MTP) module produce an overall weight size of roughly 685 GB. With an AWQ 4-bit quantization scheme, these models require approximately 335 GB for weight memory plus 37 GB for activation memory, totaling around 400 GB. In engineering practice, such models typically require eight 80 GB high-end accelerators, providing 640 GB of total GPU memory, to barely achieve full-capability deployment.

By comparison, a 2.8-trillion-parameter model is more than four times larger. This means that at the same quantization precision, the hardware scale required for deployment grows linearly or even more sharply, directly determining the minimum hardware threshold for full-capability deployment.

4. Supernode Architecture and Cross-Node Communication Challenges

When a single machine with multiple GPUs cannot hold all experts, the model must be deployed across nodes. In this scenario, communication latency becomes the system’s biggest bottleneck. Large-scale MoE inference is extremely sensitive to all-to-all interconnect bandwidth. With 896 experts distributed across different accelerators in multiple servers, every token must complete forward-layer computation, pass through the router, and then move activations across nodes to the accelerator hosting the target expert. After computation, the results must be transferred back to the original node.

This frequent cross-node data movement consumes compute resources directly. If communication bandwidth is insufficient, compute units remain idle waiting for data for extended periods, causing utilization to drop sharply.

To address this issue, official industry recommendations call for deployment on supernode clusters consisting of 64 or more accelerators. A supernode uses NVLink or equivalent high-speed interconnect technology to merge multiple independent nodes into one large unified communication domain. Under this architecture, large-scale expert activation transfers can be contained within a single supernode, avoiding the high latency introduced by traditional network switches and thereby preserving effective compute utilization.

5. Hardware Configuration Strategy and Cloud-Native Deployment Recommendations

For local or private-cloud deployment of ultra-large MoE models, enterprises and developers should plan hardware configuration and costs across the following dimensions:

1. GPU memory capacity planning: Based on the estimate above, supporting nearly 1,500 GB of weight memory while reserving sufficient KV cache space requires an accelerator cluster with more than 2 TB of total GPU memory. This means dozens of 80 GB-class accelerators are the starting point.
2. High-speed interconnect network: Traditional Ethernet or PCIe interconnects should be avoided. The cluster must support NVLink or an equivalent high-speed cache-coherent interconnect architecture to ensure cross-GPU and cross-node bandwidth at the hundreds of GB/s level, minimizing the communication penalty of MoE routing.
3. Distributed scheduling system: The deployment platform needs strong containerized orchestration and heterogeneous computing scheduling capabilities. With a cloud-native container platform and efficient distributed storage services, model parameters can be loaded quickly and load balancing across nodes can be achieved.
4. Elasticity and observability: Since running at full capacity is expensive, the system should integrate comprehensive logging and monitoring with alerting mechanisms. During low-traffic periods, elastic scaling can reduce capacity. For non-critical workloads, expert offloading techniques can temporarily store less frequently used experts in system memory or SSDs to optimize overall operational costs.

Conclusion: Local deployment of a trillion-scale MoE model is an extremely complex system engineering undertaking. Hardware procurement alone involves dozens of high-end accelerators and expensive high-speed networking equipment. When making decisions, enterprises must evaluate not only one-time hardware investment but also communication optimization under supernode architecture, distributed scheduling capabilities, and long-term operational costs. For teams without a large-scale computing infrastructure, leveraging mature public cloud high-performance computing instances may be a more economical and practical deployment path.