Local Deployment of Ultra-Large MoE Models: Memory Estimation and Cost Breakdown

When building enterprise-grade LLM infrastructure, deploying an ultra-large mixture-of-experts model on-premises creates two extreme constraints at the same time: GPU memory capacity and cluster interconnect performance. This article uses a typical ultra-large MoE architecture as a reference to examine how memory is consumed, what hardware configuration is required, and how the total deployment cost scales.

1. The Memory Logic Behind Trillion-Parameter MoE Architectures

Before discussing deployment cost, it is necessary to understand how memory is actually used in an ultra-large MoE model. A representative architecture may contain approximately 2.8 trillion total parameters. In terms of design, such a model usually includes hundreds or even nearly a thousand expert networks, such as 896 experts. However, when generating each token during inference, the routing mechanism activates only a very small subset, such as 16 experts, resulting in an expert activation rate of roughly 1.79%.

A common misconception is that a low activation rate proportionally reduces memory requirements. In reality, the expert activation rate mainly affects computational cost in FLOPs. The lower bound of memory usage is determined by the total parameter count. In MoE deployment, the system must load the weights of all 896 experts into storage. During inference, activations are routed to the nodes holding the selected experts. Therefore, no matter how low the activation rate is, the physical requirement of storing all expert weights remains high.

On the backbone side, such ultra-large models typically adopt advanced attention mechanisms and residual designs, and may also incorporate components such as gated multi-dimensional latent attention to improve long-context processing. However, the dominant memory pressure still comes from the massive expert parameters.

2. Quantization Strategy and Memory Estimation

With trillions of parameters, storing the model in full precision such as FP16 or BF16 is almost impractical. Quantization is therefore essential. Advanced quantization pipelines often introduce quantization-aware training from the supervised fine-tuning stage so that compressed models retain as much accuracy as possible. Taking an MXFP4 weight format with MXFP8 activations as a reference, the memory lower bound can be estimated as follows:

1. Raw weight storage: 2.8 trillion parameters at 4-bit precision give a theoretical minimum of about 1,400 GB.
2. Quantization metadata overhead: Under the OCP specification, MXFP4 is not a pure 4-bit format. Every 32 4-bit elements require an additional 8-bit scale factor. As a result, the effective bit width of weights plus scale factors is about 4.25 bits. The actual weight memory usage is therefore 2.8T × 4.25 bit ÷ 8 ≈ 1,487.5 GB.
3. Additional runtime overhead: The 1,487.5 GB figure represents only static weight storage. In a real deployment, additional memory is required for non-4-bit tensors, framework metadata, memory alignment redundancy, runtime workspaces, and a dynamically growing KV cache. Taking these factors into account, the actual usable GPU memory requirement will be noticeably higher.

For comparison, a widely referenced MoE model at the hundreds-of-billions scale, such as a 671B model, requires about 685 GB for primary and additional parameters in FP8. Even with AWQ 4-bit quantization, it needs roughly 335 GB for weights plus 37 GB for activations, totaling close to 400 GB. That is just enough to run on eight 80 GB high-end accelerators. A 2.8-trillion-parameter model is more than four times larger, which sets a much stricter lower bound for full deployment.

3. Supernode Architecture and Interconnect Bottlenecks

When a single node with eight GPUs cannot satisfy the memory requirement, deployment must evolve into a multi-node supernode cluster. Engineering guidance for ultra-large MoE models of this scale typically recommends a supernode cluster with at least 64 accelerators. This is not simply about adding more memory. It is also necessary to overcome the communication bottleneck specific to MoE inference.

Large-scale MoE inference is highly sensitive to all-to-all communication bandwidth. Because 896 experts are distributed across dozens of accelerators, generating every token requires the following steps:

1. Determine which experts the router selects.
2. Move the current activations over the high-speed interconnect to the target accelerators.
3. After the target accelerators finish expert computation, return the results to the original node.

This frequent cross-card or cross-node data movement can create severe communication latency. If interconnect bandwidth is insufficient, communication stalls will directly reduce accelerator utilization, causing compute resources to sit idle while waiting for data. Therefore, a supernode architecture must rely on NVLink or an equivalent high-speed interconnect and make the communication domain as large as possible. The goal is to keep activation movement within a single supernode communication domain and avoid unpredictable latency introduced by routing across switches.

4. Deployment Cost Estimation and Cloud-Native Practice

Based on the above memory and communication requirements, the on-premises hardware cost can be roughly estimated:

1. Accelerator requirement: Assuming 80 GB per accelerator, loading more than 1,500 GB of weights and runtime data requires at least 20 cards. To support concurrent throughput and fault tolerance, a practical deployment usually needs 64 or more high-end accelerators.
2. Interconnect cost: Building a 64-card supernode requires custom high-speed interconnect switches and a large number of high-speed cables. This hardware cost often accounts for a significant share of total cluster spending.
3. Operations and energy cost: Running 64 accelerators, each consuming hundreds of watts at full load, also requires substantial investment in power delivery and liquid cooling infrastructure.

Given such high initial hardware investment and operational complexity, enterprises often reconsider whether pure on-premises physical deployment is necessary. In the cloud-native era, relying on mature cloud infrastructure is a more rational choice. For example, with heterogeneous computing clusters and container services provided by Baidu AI Cloud, enterprises can directly schedule pre-provisioned pools of high-speed interconnected accelerators, avoiding the hardware procurement cycle and maintenance burden of building their own supernode cluster.

Under the Baidu AI Cloud architecture, enterprises can mount model weights in high-performance object storage and use the high-speed internal network to distribute and load weights within minutes. Cloud-native monitoring, alerting, and logging services can also be used to track all-to-all bandwidth utilization and peak memory fluctuations in real time. This elastic deployment model means enterprises do not need to buy expensive supernode hardware upfront. Instead, they can consume resources on demand, which significantly improves the cost structure and engineering efficiency of deploying ultra-large MoE models.

In summary, deploying an ultra-large MoE model on-premises is not simply a matter of plugging in GPUs. It is a systematic engineering effort involving precise memory estimation, communication topology design, and high-speed interconnect scheduling. Understanding the underlying parameter activation logic and communication bottlenecks, and carefully evaluating the total cost of self-managed versus cloud-based deployment, is essential for successfully executing an enterprise LLM infrastructure strategy.