How Much VRAM and Compute Does It Cost to Deploy a Trillion-Parameter MoE Model Locally?

1. The Starting Point: Total Parameters Set the VRAM Floor

Any discussion of deploying extremely large sparse models locally must begin by distinguishing two concepts that are often conflated: total parameters and activated parameters.

Consider a Mixture of Experts (MoE) model whose total parameter count reaches the trillion level. Such an architecture typically contains hundreds of expert modules, but each token activates only a small subset of them during inference. Suppose a model has 2.8 trillion total parameters and 896 experts, with 16 experts activated per token. The expert activation ratio would be approximately 1.79%. It is important to stress that this is the activation ratio of experts, not strictly the activation ratio of parameters — different experts may have different parameter counts, and shared trunk parameters do not participate in routing decisions at all.

Why does this distinction matter? Because it determines two completely different resource dimensions:

• VRAM usage is determined by total parameters: all experts must reside in the storage hierarchy, and inference loads a subset of them according to routing results. You cannot load only 16 experts into VRAM, because the next token may be routed to a completely different combination of experts.
• Compute consumption is determined by activated parameters: only the activated weights participate in matrix multiplication, so FLOPs are far lower than those of a dense model with the same total parameter count.

In other words, MoE architectures trade storage for compute. The VRAM floor is set by total parameters, while the compute ceiling is set by activated parameters. The largest part of deployment cost falls on VRAM first.

2. How Quantization Formats Affect Actual VRAM Usage

2.1 From FP8 to 4-bit: The Raw Data Floor

Using the same 2.8 trillion-parameter model as an example, storing weights in FP8 format requires approximately:

2.8 × 10^12 parameters × 1 Byte = 2800 GB

That is already a striking number. With 4-bit quantization, the theoretical floor becomes:

2.8 × 10^12 × 0.5 Byte = 1400 GB

However, 1400 GB is only an ideal value. In real deployments, quantization formats usually carry additional scale factors, zero points, and metadata.

2.2 The Real Overhead of MXFP4

Take the MXFP4 format defined in the OCP (Open Compute Project) specification as an example: every 32 4-bit elements share one 8-bit scale factor. This means the average bit width per weight element is not 4 bits, but:

4 bit + (8 bit / 32) = 4.25 bit

Based on this, the actual weight storage requirement for a 2.8 trillion-parameter model is:

2.8 × 10^12 × 4.25 bit ÷ 8 ≈ 1487.5 GB

This does not yet include non-4-bit tensors such as embedding layers, normalization layers, and routing networks, nor metadata, memory alignment padding, runtime workspaces, or KV cache. Adding these together, the VRAM floor for a full deployment is typically above 1.6 TB.

2.3 Comparison with a Mid-Scale MoE Model

To establish an intuitive reference point, consider a mid-scale MoE model with about 671B total parameters, plus roughly 14B multi-token prediction (MTP) parameters, for a total of about 685B. In FP8 format, the weights require approximately 685 GB. Even with 4-bit weight quantization and 8-bit activations, the weights take about 335 GB and activations about 37 GB, totaling roughly 400 GB — just enough to run on eight 80 GB accelerators with 640 GB total VRAM.

A trillion-level model is more than four times larger. This means that even with the same 4-bit quantization strategy, VRAM demand scales by more than four times. The floor for a full deployment is therefore determined by this fact: it is not a matter of a few accelerator cards, but dozens of cards.

3. The Communication Bottleneck: A Hidden Cost Multiplier

VRAM is only the first hurdle. What truly complicates large-scale MoE deployment is all-to-all communication.

During MoE inference, after each token passes through the routing network, its activations must be sent to the accelerator card hosting the target experts, and the results must be moved back after computation. With 896 experts distributed across multiple cards, every token goes through a “dispatch — compute — collect” communication cycle.

This has two direct consequences:

1. Communication latency directly eats into compute efficiency: if inter-card bandwidth is insufficient, accelerators spend a large portion of their time waiting, and actual utilization may fall far below theoretical peak.
2. Supernodes become a practical necessity: for deployments of this scale, official guidance typically recommends using a supernode of 64 or more accelerators connected through high-bandwidth interconnects such as NVLink or equivalent die-to-die links to alleviate all-to-all pressure.

In other words, communication bandwidth is a more hidden cost than VRAM. You could use a larger number of low-bandwidth cards to satisfy the VRAM requirement, but the communication bottleneck would degrade inference speed to an unacceptable level.

4. A Cost Estimation Framework

Combining the analysis above, a rough cost estimation framework can be summarized as follows:

Cost Item	Estimation Basis	Reference Scale
VRAM / storage	Total parameters × quantization bit width + runtime overhead	Above 1.6 TB
Number of accelerators	VRAM requirement ÷ VRAM per card	64 or more
Interconnect bandwidth	All-to-all communication demand	Supernode-level interconnect
Power and cooling	Total power consumption × runtime	Tens of kilowatts
Operational cost	Cluster management, monitoring, fault recovery	Ongoing investment

Taking 64 cards with 80 GB VRAM each as an example, hardware procurement cost alone could reach tens of millions of RMB. If using on-demand cloud instances, the hourly cost would also reach thousands of RMB. This does not even include storage, networking, power, and labor costs.

5. A Key Checklist Before Deployment

Before deciding on local deployment, it is advisable to confirm the following items one by one:

1. Is the quantization plan complete: what formats are used for weights, activations, and KV cache respectively? Does it include quantization-aware training?
2. Does the VRAM budget leave enough headroom: beyond weights, has at least 20% extra space been reserved for runtime workspaces, temporary buffers, and fragmentation?
3. Does the interconnect topology meet requirements: can inter-card bandwidth support all-to-all communication? Is supernode-level interconnect required?
4. Does the inference framework support it: does the selected inference engine support the MoE architecture’s routing strategy and quantization format?
5. Is the cost model sustainable: is the total cost of ownership, including hardware procurement, power, operations, and model updates, within budget?

6. Summary

Local deployment of a trillion-parameter MoE model is fundamentally a problem of balancing storage, communication, and compute. The VRAM floor is determined by total parameters and quantization format, typically above 1.6 TB. The communication bottleneck dictates that you must adopt a supernode architecture with high-bandwidth interconnect. Although compute consumption is reduced by sparse activation, communication latency may cause actual throughput to fall far below theoretical values.

For most teams, deploying such a model locally at full capacity requires not only hardware investment in the tens of millions of RMB, but also professional cluster operations capability. Before making a decision, it is advisable to first validate the inference framework, quantization strategy, and communication topology with a mid-scale model, then expand gradually.