Local Deployment of Trillion-Parameter MoE Models: A Full Breakdown of VRAM Costs and Hardware Requirements

1. Why the “Activation Ratio” Does Not Equal “VRAM Savings”

When discussing local deployment of large-scale MoE (Mixture of Experts) models, the first concept that is easily misread is the expert activation ratio.

Take an MoE model with total parameters on the order of a trillion: suppose it has roughly 2.8 trillion total parameters, uses an MoE architecture with 896 experts, and routes each token to 16 of them. The expert activation ratio is then approximately 16 ÷ 896 ≈ 1.79%.

This number easily creates the illusion that “since only 1.79% is activated, VRAM usage should also be 1.79%.” That is wrong.

Key conclusion: MoE VRAM usage is calculated based on total parameters. The activation ratio only affects compute consumption (FLOPs), not weight storage requirements.

The reason lies in how inference systems are engineered:

1. All experts must reside in storage at all times. Any of the 896 experts could be hit by a given token’s routing, so all expert weights must be accessible. You cannot load only 16 experts, because the next token might route to a completely different set of 16.
2. Routing is dynamic and token-by-token. During inference, the gating network (Router) output is used to dispatch activations to the devices hosting the target experts for computation, then collect the results back.
3. The activation ratio determines compute volume. Each forward pass only performs matrix multiplications for the activated experts, so compute consumption is proportional to activated parameters. But the physical storage of weights is proportional to total parameters.

This distinction is the foundation for estimating deployment costs. Estimating VRAM based on the activation ratio will lead to severely underestimated conclusions.

2. How Quantization Formats Affect the VRAM Floor

Once it is clear that “VRAM is calculated based on total parameters,” the next step is to determine how many bits each parameter occupies.

2.1 Quantization Awareness Introduced During Training

Some advanced large-scale MoE models introduce quantization-aware training (QAT) starting from the supervised fine-tuning (SFT) stage, adapting weights and activations to low-bit formats separately. A common combination is: 4-bit floating-point format for weights, 8-bit floating-point format for activations.

This “low-bit weights, slightly higher-bit activations” pairing is a common trade-off in the industry: weights occupy the vast majority of storage, so compressing them to 4-bit yields the greatest benefit; activations participate in dynamic computation, so keeping them at 8-bit reduces accumulated numerical error.

2.2 Don’t Overlook the Extra Overhead from Scale Factors

A key detail of 4-bit quantization is block scaling. Take the common MXFP4 format as an example: every 32 4-bit elements share a single 8-bit scale factor.

This means the actual bit overhead per parameter is not 4 bits, but rather:

Per 32 elements: 32 × 4 bit + 1 × 8 bit = 136 bit
Average per element: 136 ÷ 32 = 4.25 bit

Based on this, the lower bound for raw weight data is estimated as:

2.8 trillion × 4.25 bit ÷ 8 = approximately 1487.5 GB

Note that this is only the lower bound for raw weight data itself, and does not yet account for:

• Non-4-bit tensors (e.g., some normalization layers or embedding layers may retain higher precision)
• Metadata and tensor descriptors
• Memory alignment padding
• Runtime workspace
• KV cache (non-negligible in long-context scenarios)

Actual deployable VRAM requirements typically need to be raised by a considerable margin above the raw data floor.

2.3 A Mid-Scale Model for Reference

To build intuition, compare a model with roughly 671B total parameters (including approximately 14B multi-token prediction parameters, totaling about 685B):

Precision Format	Weight Usage	Activation Usage	Total
FP8	~685 GB	—	~685 GB
4-bit weight quantization	~335 GB	~37 GB	~400 GB

In other words, a model with roughly 685B parameters still requires about 400GB of VRAM after 4-bit quantization to run on 8× 80GB accelerators (640GB total).

A 2.8-trillion-parameter model is more than 4× that size. Following the same 4-bit quantization approach, the raw weight data alone approaches 1.5TB, and the actual deployment floor is even higher. This is the hard barrier of “full-capability deployment.”

3. Multi-GPU Supernodes: Communication Is the Real Hidden Bottleneck

With VRAM calculated, the second barrier is interconnect bandwidth.

3.1 The Cost of All-to-All Communication

Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth. The reasons are as follows:

1. The 896 experts are distributed across multiple accelerators;
2. After routing, each token’s activations need to be sent to the card hosting the target expert;
3. After expert computation completes, the results need to be moved back;
4. The next layer repeats this process.

This is the classic Expert Parallelism communication pattern. If communication latency is not controlled, it directly eats into compute — accelerators sit idle while waiting for data movement, and utilization drops sharply.

3.2 Why Supernodes Are Needed

To mitigate this problem, the hardware level requires:

• High-bandwidth inter-card interconnect: throughput far exceeding ordinary Ethernet;
• Low-latency all-to-all topology: reducing hop count and congestion;
• Sufficiently large supernode scale: keeping expert distribution as concentrated as possible within the high-bandwidth domain.

Therefore, for deploying such trillion-scale MoE models, a more reasonable hardware form is a supernode composed of dozens or more accelerators. Too small a scale forces experts to be distributed across nodes, causing communication overhead to rise sharply; a sufficiently large scale is needed to confine communication within the high-bandwidth domain.

A simplified deployment estimation flow can be written as:

def estimate_deployment(total_params_trillion,
                        bits_per_weight=4.25,
                        kv_cache_gb=100,
                        runtime_overhead_ratio=0.15):
    # Lower bound for raw weight data
    weight_gb = total_params_trillion * 1e12 * bits_per_weight / 8 / 1e9
    # Add runtime overhead and KV cache
    total_gb = weight_gb * (1 + runtime_overhead_ratio) + kv_cache_gb
    # Estimate card count based on 80GB per card
    cards = total_gb / 80
    return weight_gb, total_gb, cards
weight, total, cards = estimate_deployment(2.8)
print(f"Raw weight data: {weight:.0f} GB")
print(f"Total with overhead: {total:.0f} GB")
print(f"Estimated 80GB cards: {cards:.0f}")

This code is only an order-of-magnitude estimate. In practice, the combined partitioning strategy of expert parallelism, tensor parallelism, and pipeline parallelism must also be considered.

4. A Complete Breakdown of the Cost Structure

Integrating the analysis above, the cost of locally deploying a trillion-scale MoE model can be broken down into the following components:

4.1 VRAM Cost (Weight Storage)

• Raw weight data: approximately 1.5TB (4-bit + scale)
• Plus non-quantized tensors, metadata, alignment: raised further
• Plus runtime workspace and KV cache: raised further still
• Conclusion: VRAM requirements are in the TB range, beyond what a single machine can host

4.2 Interconnect Cost (Communication Hardware)

• Requires high-bandwidth, low-latency inter-card interconnect
• Requires supernodes composed of dozens of cards
• Interconnect hardware accounts for a considerable share of overall cost, and is often underestimated

4.3 Compute Cost (Number of Accelerators)

• Card count is determined jointly by the VRAM floor and the parallelism strategy
• The activation ratio affects per-inference compute consumption, not the minimum card count
• Actual card count typically needs redundancy to handle peaks and fault tolerance

4.4 Operational Cost

• Power and cooling: dozens of high-power accelerators running continuously
• Data center and power distribution modifications
• Operations and failure recovery

5. Practical Recommendations for Different Scenarios

Not every team needs a “full-capability deployment.” Requirements can be layered as follows:

Scenario 1: Research and Validation

• Goal: validate algorithms, get the pipeline working
• Recommendation: use a small-scale distilled version or API calls; local deployment of the full model is unnecessary

Scenario 2: Mid-Scale Private Deployment

• Goal: data stays on-premises, moderate throughput requirements
• Recommendation: choose an MoE model with a moderate parameter count, pair it with 4-bit quantization, and use an 8-card node

Scenario 3: Full-Capability Production Deployment

• Goal: complete capabilities, high concurrency
• Recommendation: form a supernode from dozens or more accelerators, prioritize all-to-all communication bandwidth, and jointly tune VRAM and communication

6. Summary

Back to the original question — how much does it cost to deploy a trillion-scale MoE model locally?

The key chain of reasoning is:

1. VRAM is calculated based on total parameters, not the activation ratio. The activation ratio only determines compute consumption.
2. After 4-bit quantization, raw weight data is still in the TB range. For 2.8 trillion parameters in MXFP4 format, the raw data floor is approximately 1.5TB.
3. Actual requirements stack on top: non-quantized tensors, metadata, alignment, workspace, and KV cache.
4. The bottleneck in multi-GPU deployment is all-to-all communication, requiring supernodes of dozens of cards to confine communication within the high-bandwidth domain.
5. Cost is determined jointly by VRAM, interconnect, compute, and operations, with interconnect and VRAM being the most easily underestimated.

For the vast majority of teams, before deploying a trillion-scale MoE model locally at full capability, a more practical path is to first clarify the real requirements of their scenario and assess whether mid-scale models, quantization and compression, or elastic cloud resources can achieve the goal. The value of local deployment lies in data controllability and long-term cost amortization, but only when the hardware barriers and engineering capabilities are already in place.