Demystifying VRAM and Costs for Local Deployment of Ultra-Large MoE Models: A Trillion-Parameter Case Study

I. Why Total Parameters Dictate Deployment Thresholds Over Activated Parameters

When discussing the deployment costs of ultra-large models, a common misconception is conflating “parameters activated per token” with “total parameters that need to be loaded.” For dense models, these two figures are roughly identical. However, for MoE architectures, the difference is massive.

Consider a model with 2.8 trillion total parameters. Under an MoE architecture, the model contains 896 experts, and each token selects 16 experts for computation based on the routing mechanism. Dividing 16 by 896 gives an expert activation ratio of roughly 1.79%. While this number is frequently used to describe the model’s “sparse activation level,” it does not mean you only need to load 1.79% of the parameters.

This is because routing during inference is dynamic. Different tokens, batches, and requests will hit different experts. To ensure accurate forward computation for any given input, the deployment environment must keep all expert weights in accessible storage, rather than loading just a select few. In other words:

• Total parameters determine the lower bound for VRAM and storage.
• Activated parameters determine the compute requirements for a single step.
• The number of experts, routing strategy, and request distribution collectively determine communication and scheduling costs.

Therefore, for a 2.8T MoE model, regardless of how few experts are activated per token, VRAM planning must be estimated based on the full 2.8 trillion parameters. This is the foundation for all subsequent cost derivations.

II. Quantization Reduces VRAM, But It’s Not as Simple as “Divide by 4”

If original weights are stored in FP8 or BF16, a 2.8T model will directly reach the tens of terabytes scale, making it nearly impossible to host on conventional local hardware. Consequently, deployment plans typically introduce low-bit quantization. However, actual VRAM consumption is not simply “parameter count multiplied by quantization bit width.”

Taking 4-bit weights as an example, the naive calculation is:

2.8T parameters × 4 bit ÷ 8 = 1400GB

This seems like an acceptable lower bound. However, block quantization formats like MXFP4 used in practice are not purely 4-bit storage. MXFP4 typically groups 32 4-bit elements together, with an 8-bit scale attached to each group to restore the numerical range. Thus, the equivalent bit width per element is not 4 bits, but:

32 × 4 bit + 8 bit = 136 bit
136 bit ÷ 32 = 4.25 bit

Calculating with this more realistic figure:

2.8T parameters × 4.25 bit ÷ 8 ≈ 1487.5GB

This means the weights alone approach 1.5TB. And this is just the theoretical raw data.

In a real deployment, several additional components must be factored in:

1. Non-4-bit Tensors: Not all parameters are suitable for 4-bit quantization. Embeddings, normalization layers, and routing networks may retain higher precision.
2. Metadata and Alignment Overhead: Weight sharding, page alignment, checksums, and loading indices all increase actual memory usage.
3. Runtime Workspace: Intermediate activations during forward computation, operator temporary buffers, and gradient states are reduced during inference but do not disappear entirely.
4. KV Cache: With long contexts or high concurrent requests, the KV cache can become an independent and significant source of VRAM consumption.
5. Multi-Replica and Pipeline Buffers: If pipeline parallelism is employed, buffers between stages will also consume extra space.

Therefore, deployment planning cannot just look at “how many GB is 4-bit.” It must include weights, quantization metadata, runtime states, and communication buffers in the budget.

III. A 671B Dense MoE Model as a Reference

To more intuitively understand the deployment pressure of a 2.8T parameter model, we can use an approximately 671B parameter MoE model as a reference.

In FP8 precision, the main parameters plus the auxiliary prediction module parameters total about 685GB for this model. If 4-bit weight quantization is applied, the weights take up roughly 335GB, while activation and quantization-related overheads require an additional 37GB, totaling around 400GB.

This means that even for a 671B parameter model, to fully load and maintain a usable inference space, 8 x 80GB VRAM accelerator cards (640GB total) are just barely sufficient, and this is before reserving room for the KV cache and runtime overhead. If concurrency increases or the context lengthens, the headroom will be exhausted rapidly.

The total parameter count of a 2.8T model is more than four times that of a 671B model. Using the same quantization strategy, the weights alone would exceed 1.4TB, and overall VRAM requirements would easily surpass 1.6TB or more. This explains why a single-node multi-GPU setup is fundamentally unfeasible for a 2.8T parameter model.

IV. Why Multi-Card Supernodes Are Required Instead of Simply Stacking VRAM

When model weights reach the 1.5TB level, even if you purchase enough GPUs, fitting them into a single machine faces physical limits. More critically, MoE inference is not just about “fitting the model”; it involves massive inter-card communication.

In an MoE architecture with 896 experts, experts are distributed across multiple cards. After a token completes routing, its intermediate activations must be sent to the cards hosting the target experts, and the results must be retrieved after computation. This process can be simplified as:

Input token
  → Routing decision
  → Cross-card dispatch to 16 experts
  → Individual expert computation
  → Cross-card result retrieval
  → Merge output

If each card only hosts a subset of experts, processing a single token can generate multiple cross-card communication paths. As request volume grows, communication pressure increases non-linearly.

The core constraint here is not single-card compute power, but all-to-all communication bandwidth. Large-scale MoE inference is extremely sensitive to this type of communication. If the communication domain is small and cross-node hops are numerous, latency will double or worse. At this point, even if each GPU has high FLOPs, actual utilization may remain extremely low.

Therefore, deployment plans for such models typically recommend using supernodes composed of 64 or more accelerators, relying on high-bandwidth interconnects to consolidate the communication domain into a single high-speed region. The goal of NVLink, InfiniBand, or equivalent high-speed interconnects is consistent: prevent expert communication from occurring across node hops, thereby reducing latency and congestion.

In other words, for local deployment of ultra-large MoE models, the real threshold is often not “having enough VRAM,” but “having a sufficiently large high-speed communication domain.”

V. Costs Go Beyond GPU Prices

If we calculate based on 64 x 80GB VRAM accelerator cards, the GPU procurement cost alone reaches a significant order of magnitude. Based on common market prices for individual cards:

64 × Single card price

This single line item can easily reach the level of millions of dollars (or equivalent local currency). But this is far from the total cost. An actual deployment typically also includes:

1. Chassis and Interconnect Equipment: High-speed switches, cables, optical modules, and cooling systems.
2. Memory and Storage: Loading weights requires high-speed NVMe or persistent storage, and model version management will also occupy substantial space.
3. Power and Datacenter Upgrades: Multi-card systems can consume tens of kilowatts, imposing extra requirements on power supply, cooling, and rack weight capacity.
4. Backup and Redundancy: Production environments usually need to reserve capacity for faulty card replacement or degraded operation.
5. Operations and Maintenance Personnel: Deploying ultra-large models involves distributed frameworks, monitoring, logging, alerting, version rollbacks, and performance tuning, resulting in high operational complexity.

Therefore, when budgeting, you cannot estimate total cost simply by multiplying “the price of one card” by “the number of cards.” A reasonable calculation should be:

Total Cost = Compute Hardware Cost
           + High-Speed Interconnect Cost
           + Storage & Memory Cost
           + Datacenter & Power Cost
           + Deployment & Tuning Cost
           + Long-Term O&M Cost

VI. Deployment Recommendations for Different Objectives

Local deployment of ultra-large MoE models is not simply a matter of “getting it to run.” The optimal solution varies depending on your objectives:

1. Technical Validation and Small-Scale Testing

If the goal is to validate the inference pipeline, evaluate routing behavior, or conduct limited capability testing, it is not recommended to deploy the full-precision weights immediately. You can use low-bit quantization, reduce expert replicas, or shorten the context length to get the model running on a smaller cluster first.

2. Offline Batch Processing

If requests do not require ultra-low latency, you can adopt a throughput-priority strategy: increase batch size, lower concurrent response requirements, and trade time for space to alleviate communication and VRAM pressure. In this scenario, hardware configurations can be slightly lower than the theoretical peak requirement.

3. Online High-Concurrency Inference

Online scenarios are sensitive to single-request latency. You must ensure the communication domain is large enough, the KV cache is sufficient, and scheduling is stable. The deployment architecture here typically needs to reserve at least 20% to 30% of VRAM and compute headroom to avoid queuing or failures during traffic spikes.

4. Hybrid Deployment Strategies

For budget-constrained scenarios with continuous inference demands, cold weights can be stored on high-speed persistent devices, reducing resident VRAM through dynamic loading strategies. However, this approach sacrifices first-token latency and demands extremely high storage bandwidth, requiring a detailed cost-benefit evaluation.

VII. The Gap Between “Can It Run” and “Is It Usable”

When evaluating the deployment of ultra-large MoE models, many teams instinctively ask, “How many cards do we need?” But the actual problem should be broken down into three more detailed dimensions:

• Capacity: Is there enough VRAM to hold all weights and runtime states?
• Bandwidth: Can expert communication be completed within the latency budget?
• Stability: Under sustained high load, will VRAM fragmentation, communication congestion, or node drops occur?

Capacity issues can be solved by stacking cards. Bandwidth issues require high-speed interconnects and a rational topology. Stability issues rely on engineering deployment and O&M systems. All three are indispensable.

For an MoE model with 2.8 trillion parameters, 896 experts, and 16 activated experts per token, the core conclusion can be summarized as follows:

Quantization merely lowers the barrier to entry. What truly determines the feasibility of local deployment is the scale of the high-speed communication domain and engineering capabilities.

When conducting budget planning, it is recommended to calculate progressively based on the following priorities:

1. Peak VRAM usage on the single-token path.
2. KV cache and I/O pressure under multi-concurrency.
3. All-to-all communication frequency and communication domain scale.
4. Fault recovery and monitoring capabilities under full load.
5. Total Cost of Ownership (TCO) for long-term operation, rather than one-time procurement costs.

Drawing conclusions this way will be much closer to real-world deployment scenarios than simply calculating “how many GB can 2.8 trillion parameters be compressed to,” and it will better help decision-makers avoid making misjudgments based on the wrong metrics.