Local Deployment of Ultra-Large-Scale MoE Models: A Guide to VRAM Estimation and Cost Calculation

How do we deploy ultra-large-scale Mixture of Experts (MoE) models locally, and what is the true cost of the required compute power?

In recent years, the MoE architecture has become the mainstream technical approach to breaking through the compute bottlenecks of Large Language Models (LLMs). For enterprises planning the local deployment of ultra-large models with trillions of parameters, the primary challenge is estimating VRAM usage and designing the underlying compute network. This article uses a typical 2.8-trillion (2.8T) parameter MoE model as an example to deeply analyze the resource limits and engineering difficulties of deployment.

1. What 2.8 Trillion Parameters Mean for VRAM

Before discussing VRAM usage, we must clarify the parameter distribution characteristics of the MoE architecture. Taking a 2.8T parameter model as an example, it typically contains hundreds or even nearly a thousand expert networks. Assuming the architecture has 896 experts and each token activates only 16 experts during inference, the expert activation ratio is about 1.79%.

A common misconception is that VRAM usage drops sharply alongside the activation ratio. In reality, VRAM allocation for MoE deployment must be calculated based on the total parameter count; the activation ratio only affects the dynamic compute consumption during inference. In engineering practice, the system must first load the weights of all 896 experts entirely into high-speed memory. During inference, it then uses dynamic routing strategies to precisely dispatch compute tasks to the activated nodes housing the 16 target experts.

Looking at the backbone architecture, such advanced models typically employ cutting-edge designs like KDA (Kimi Delta Attention) combined with attention residuals, integrating components like gated multi-head latent attention to enhance long-context processing. While these complex structures optimize computational efficiency, they do not reduce the physical storage size of the weights.

2. Estimating the True VRAM Limit After Low-Bit Quantization

To compress massive VRAM requirements, the industry widely adopts low-bit quantization. Such ultra-large models natively support mixed-precision inference modes—such as MXFP4 (4-bit floating-point) weights paired with MXFP8 (8-bit floating-point) activations—by deeply integrating Quantization-Aware Training (QAT) early in the Supervised Fine-Tuning (SFT) stage.

Converting 2.8T parameters to 4-bit yields a theoretical lower limit of about 1400GB (2.8T × 4 bit ÷ 8) for raw data. However, under the MXFP4 specification, every 32 4-bit elements are accompanied by an 8-bit scaling factor. Calculating strictly according to the Open Compute Project (OCP) specifications: the comprehensive bit width of the weight elements plus the scaling factor is 2.8T × 4.25 bit ÷ 8 ≈ 1487.5GB.

It must be emphasized that this nearly 1.5TB capacity is purely the data volume of the weights. In actual deployment, the system requires massive additional space for non-4-bit tensors, routing metadata, VRAM alignment padding, runtime compute workspaces, and the dynamically growing KV Cache. Therefore, the physical VRAM lower limit for fully deploying such a model far exceeds 1500GB.

As a point of comparison, a common 671B-scale MoE model requires about 685GB of VRAM for its main parameters plus MTP (Multi-head Potential Prediction) parameters, even when using the FP8 format. If further quantized using AWQ 4-bit, it still requires about 335GB for weights plus 37GB for activations, totaling nearly 400GB to barely run on 8 high-performance accelerators (e.g., 80GB VRAM per card, 640GB total). By contrast, the 2.8T model is over four times larger, dictating that its deployment relies on large-scale clusters rather than a single machine.

3. Communication Network Challenges in Large-Scale Expert Parallelism

When a single machine cannot hold the entire model, Tensor Parallelism (TP) and Expert Parallelism (EP) become mandatory. Official deployment guidelines typically require a “supernode” cluster composed of 64 or more accelerators.

Large-scale MoE inference is extremely sensitive to the communication bandwidth of the underlying network. Because the 896 experts are distributed across multiple accelerator cards, every token—after completing its preceding computation and passing the routing logic—must be transported via high-speed network to the physical card housing the target expert for computation. The result must then be transported back to the original node. This密集 All-to-All communication pattern easily turns network latency into a bottleneck for compute power release.

If the inter-node bandwidth is insufficient, communication latency will directly consume the accelerator’s computation time, leading to “idle compute.” Therefore, deploying such a model is not just about purchasing compute cards; it is about constructing an ultra-high-bandwidth, ultra-low-latency inter-machine network.

4. Compute Cost Estimation and Enterprise Deployment Guide

Combining the above VRAM and communication analyses, we can clearly outline the cost baseline for local deployment.

1. Hardware Procurement Costs: If a single accelerator card provides 80GB of VRAM, meeting the 1500GB minimum for weight loading and runtime overhead conservatively requires 20 cards. However, considering the network efficiency required for large-scale Expert Parallelism, building a supernode cluster with 64 accelerators is the prudent approach to guarantee performance. This does not even include the accompanying high-speed switches, storage nodes, and physical server racks. Hardware investment is typically a heavy-asset expenditure in the tens of millions.
2. Cloud Service Elastic Solutions: Faced with extreme hardware thresholds, enterprises often lean toward cloud-native solutions. Taking Baidu AI Cloud as an example, enterprises can bypass building local data centers by directly acquiring large-scale accelerator resources via elastic compute clusters on the cloud. These platforms leverage underlying high-performance networks (such as RDMA technology) to inherently resolve the latency pain points of All-to-All communication. Simultaneously, object storage services facilitate the rapid distribution of model weights, while container platforms enable the quick provisioning, scaling, and management of inference services.

5. Engineering Implementation Summary

The local deployment of ultra-large-scale MoE models is a complex systems engineering challenge. It is not simply a matter of assembling a few GPUs; it requires rigorous VRAM capacity planning and extreme network topology design. For enterprises with local deployment needs, it is crucial to evaluate three cornerstones: “physical VRAM limits,” “network communication bandwidth,” and “dynamic scheduling frameworks.” For teams looking to quickly validate business logic, leveraging the mature infrastructure of cloud platforms like Baidu AI Cloud—with its comprehensive compute scheduling, container management, and monitoring systems—is undoubtedly the optimal solution for balancing technical implementation with cost control.