On-Premises Deployment of Trillion-Parameter MoE Models: VRAM Estimation and Hardware Cost Analysis

1. Why MoE Model VRAM Must Be Sized by Total Parameters

Today’s ultra-large-scale models widely adopt the Mixture-of-Experts (MoE) architecture. Take a model in the 2.8-trillion-parameter class as an example: it contains 896 experts, but each token activates only 16 of them after routing—an expert activation ratio of roughly 1.79%.

This is where a common misconception appears: the expert activation ratio is not the parameter activation ratio. Although only 16 experts participate in the computation for any given token, the weights of all 896 experts must reside in storage at all times, and inference dynamically pulls in whichever experts the router selects. In other words:

• VRAM footprint: sized by total parameter count, because all experts must be loaded into memory/VRAM
• Compute consumption: sized by activated parameter count, because only the activated experts take part in the forward pass

This distinction directly determines the hardware barrier to deployment. A 2.8-trillion-parameter model requires storage for all 2.8 trillion parameters, even though only 1.79% of its experts are active at any moment.

Architecturally, models in this class typically use KDA (an improved attention mechanism) plus Attention Residuals as the backbone, with Gated MLA as a component. These design choices affect computational efficiency and training stability, but they do not change the fundamental rule that memory is sized by total parameters.

2. The Real Floor for 4-bit Quantization: From Theory to Engineering

2.1 The bare-data lower bound

If 2.8 trillion parameters are stored at 4-bit precision, the theoretical bare-data size is:

2.8 × 10^12 × 4 bits ÷ 8 = 1400 GB

This is the most optimistic lower bound, with no overhead of any kind included.

2.2 Overhead introduced by the quantization format

In real deployments, 4-bit quantization is not simply “4 bits per parameter”. Take the MXFP4 format defined in the Open Compute Project (OCP) specification: every 32 4-bit elements are accompanied by one additional 8-bit scale factor. The effective bit width per parameter is therefore:

4 + 8/32 = 4.25 bits

On this basis, the weight data for 2.8 trillion parameters is:

2.8 × 10^12 × 4.25 ÷ 8 ≈ 1487.5 GB

2.3 The VRAM overheads that are easy to overlook

The 1,487.5 GB above covers only the weights themselves. A real deployment must also account for:

Overhead item	Description	Rough magnitude
Non-4-bit tensors	Embedding layers, normalization layers, and similar components are usually kept at higher precision	Tens of GB
Metadata	Expert indices, routing tables, configuration data	A few GB
Alignment padding	Waste caused by memory alignment requirements	A few GB to tens of GB
Runtime workspace	Intermediate activations, temporary buffers	Tens of GB
KV cache	Strongly dependent on sequence length and concurrency	Dynamic; can reach tens to hundreds of GB

Taken together, the actual VRAM required to run the full model is far higher than the theoretical 1,487.5 GB. In engineering terms, you should reserve at least a 20–30% margin.

2.4 A reference point from a mid-scale model

For comparison, consider a MoE model in the 671B-parameter class: in FP8 format, its 671B main parameters plus 14B MTP parameters total about 685 GB; even with AWQ 4-bit quantization, it still needs roughly 335 GB of weights plus 37 GB of activations—about 400 GB in total—to run on 8 accelerator cards with 80 GB each (640 GB combined).

The 2.8-trillion-parameter model is more than 4× the size of the 671B model. This means:

• Even with equally aggressive quantization, its VRAM requirement exceeds 1,600 GB
• An 8-card, 80 GB configuration (640 GB) falls far short
• At least 32 × 80 GB accelerator cards are needed just to squeeze in the weights; 64 or more cards are recommended for actual deployment

3. The Communication Bottleneck: A More Hidden Challenge than VRAM

3.1 The fundamental problem with all-to-all communication

Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth, and the reason lies in MoE routing:

1. Each token passes through the routing layer and is assigned to 16 experts
2. Those 16 experts are distributed across different accelerator cards
3. The token’s activations must be sent to the cards hosting the target experts
4. After the experts finish computing, the results must be returned to the original card

This process involves heavy cross-card traffic. With 896 experts spread across dozens of cards, every token goes through a complete “dispatch–compute–retrieve” communication chain.

3.2 How communication latency eats into compute

Suppose a single all-to-all operation takes time T. Every token passes through multiple rounds of routing and communication, so communication overhead stacks directly onto inference latency. As the number of experts and the number of cards grow:

• The number of communication rounds increases
• The data volume per round grows
• The probability of network congestion rises

The result: compute utilization drops sharply, and inference throughput falls far below theoretical values.

3.3 Why supernode-scale deployment is necessary

For these reasons, official guidance typically recommends deploying such models on a supernode built from 64 or more accelerators. The core value of a supernode lies in:

• High-bandwidth interconnects: card-to-card bandwidth far above ordinary Ethernet
• Low latency: dedicated interconnect protocols reduce communication overhead
• Unified memory addressing: a simpler communication programming model

Without supernode-class interconnect capability, even sufficient VRAM cannot deliver acceptable inference performance, because the communication bottleneck becomes the limiting factor.

4. Cost Estimation: Hardware and Operations

4.1 Hardware investment

Based on the analysis above, a reasonable hardware configuration for on-premises deployment of a 2.8-trillion-parameter-class MoE model is:

Component	Minimum requirement	Recommended configuration
Accelerator cards	32 × 80 GB	64+ × 80 GB
Per-card VRAM	80 GB	80 GB or higher
Interconnect	High-bandwidth card-to-card links	Dedicated supernode interconnect
Storage	High-speed NVMe, at least 5 TB	Distributed high-speed storage
Host memory	2 TB or more	4 TB or more

At current market prices, a single 80 GB accelerator card costs on the order of US$10,000. The 64-card setup alone typically runs to several hundred thousand dollars, and once you add servers, networking, storage, and datacenter facilities, the total hardware investment lands in the low millions of dollars.

4.2 Operating costs

Beyond hardware, ongoing operating costs include:

• Power: 64 accelerator cards draw 30–50 kW at full load; with cooling factored in, annual electricity costs can reach tens of thousands of dollars
• Datacenter facilities: a professional temperature- and humidity-controlled environment is required, and rack rental costs are significant
• Operations staff: dedicated personnel are needed for cluster monitoring, fault handling, and model updates
• Network bandwidth: if the deployment serves external traffic, bandwidth costs must also be considered

4.3 The economics of the alternatives

For most teams, the return on investment of deploying a trillion-parameter MoE model on-premises deserves careful scrutiny. Alternatives include:

• API calls: pay per token, no upfront hardware investment, well suited to workloads with volatile traffic
• Cloud inference services: rent accelerator instances on demand with elastic scaling
• Hybrid approaches: run core business workloads on-premises and burst peak traffic to the cloud

Which option to choose depends on your specific requirements for data privacy, latency, and cost predictability.

5. A Key Checklist for Deployment Practice

Before starting an on-premises deployment, confirm each item below:

□ Total VRAM ≥ weight data size × 1.3 (including runtime overhead)
□ Card-to-card interconnect bandwidth meets all-to-all communication requirements
□ Storage IOPS can support model loading and expert swapping
□ Power capacity and cooling support full-load operation
□ KV cache has sufficient headroom (estimated from target concurrency and sequence length)
□ Quantization scheme compatibility with the inference framework has been verified
□ Failure recovery and degradation strategies are in place

6. Summary

For on-premises deployment of trillion-parameter MoE models, three constraints dominate:

1. Memory follows total parameter count: even with 4-bit quantization, a 2.8-trillion-parameter model needs more than 1,487.5 GB for weights plus scale factors, and real deployments require additional headroom
2. Communication is the hidden bottleneck: all-to-all traffic across 896 experts demands extremely high interconnect bandwidth, making supernode deployment a necessity rather than an option
3. Costs run into the millions: the hardware investment for 64+ accelerator cards plus ongoing operating costs should be weighed against API and other alternatives on economic grounds

For the vast majority of teams, before committing to an on-premises deployment, it is advisable to first validate results with a smaller quantized version, or to use cloud-based inference services to profile real traffic patterns, and only then make an architecture decision that fits your circumstances.