The True Cost of Deploying Trillion-Parameter Models Locally: Memory and Bandwidth Analysis

1. Why “Activation Ratio” Cannot Be Used Directly to Estimate VRAM

When discussing a Mixture of Experts (MoE) model with a total parameter count reaching the trillion level, the first cognitive trap we encounter is mistaking the expert activation ratio for the parameter activation ratio, thereby underestimating the VRAM requirement.

Take a MoE model with 2.8 trillion total parameters as an example. Its architecture includes 896 experts, and each token activates 16 of them during inference, resulting in an expert activation ratio of about 1.79%. This small number easily creates the illusion that “only 1.79% of the parameters need to be loaded.” The reality, however, is:

• VRAM usage is calculated based on total parameters: All 896 experts must reside in the memory hierarchy and are dynamically invoked based on routing results during inference.
• The activation ratio only affects compute consumption: It determines the number of parameters actually involved in the matrix operations during each forward pass, not the number of parameters that need to be constantly resident in memory.

In other words, the deployment cost of an MoE model is determined by the “total warehouse area,” not by the “number of workstations operating simultaneously.” This is the first principle for estimating the local deployment cost of any trillion-parameter MoE model.

2. How Quantization Formats Affect the Actual Storage Floor

To alleviate VRAM pressure, such models typically introduce quantization-aware training from the supervised fine-tuning stage. Using MXFP4 weights paired with MXFP8 activations as an example, we need to calculate the actual footprint layer by layer.

2.1 Raw Data Floor

Converting 2.8 trillion parameters to 4-bit yields a theoretical raw data volume of:

2.8 × 10^12 × 4 bit ÷ 8 = 1400 GB

2.2 Actual Footprint Including Scaling Factors

MXFP4 is not purely 4-bit storage. According to the OCP specification, every 32 4-bit elements carry an additional 8-bit scale. Therefore, the average bit width per weight element is:

4 + 8 ÷ 32 = 4.25 bit

Recalculating:

2.8 × 10^12 × 4.25 bit ÷ 8 ≈ 1487.5 GB

This 1487.5 GB is still only the lower limit for the weights themselves. In actual deployment, you must also account for:

• Non-4-bit tensors (such as normalization layers and embedding layers, which typically maintain higher precision)
• Model metadata and tensor alignment padding
• Runtime workspaces (temporary buffers, communication buffers)
• KV Cache (which grows linearly with concurrent requests and context length)

Therefore, the total VRAM required for real deployment will be significantly higher than the theoretical value of 1487.5 GB.

3. Horizontal Reference: Mid-Sized MoE Deployment Baseline

To establish an intuitive sense of scale, we can refer to deployment data from a mid-sized MoE model.

A model with approximately 671B main parameters plus 14B multi-token prediction parameters requires about 685 GB in FP8 format. Even with 4-bit weight quantization, the weights take up about 335 GB, and adding roughly 37 GB for activations brings the total to about 400 GB needed to run on 8 80GB accelerators (640 GB total).

The scale of a 2.8 trillion parameter model is more than 4 times that of the above. This means:

Model Scale	Quantization Format	Weight Footprint (Approx.)	Minimum Accelerators Needed (80GB)
Mid-sized MoE	4-bit	~400 GB	8
Trillion-level MoE	MXFP4	~1500 GB+	32+ (weights only)
Trillion-level MoE	MXFP4 + Runtime overhead	2000 GB+	64+ (Recommended)

4. Communication Bottlenecks: An Underestimated Deployment Threshold

Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth, an implicit threshold that many teams easily overlook when estimating costs.

4.1 Communication Pattern Analysis

When 896 experts are distributed across multiple accelerators, the processing flow for each token is as follows:

1. Routing decision: Determine which 16 experts the token should be assigned to.
2. Activation dispatch: Transfer the token’s activations to the accelerator hosting the target expert.
3. Expert computation: Complete the forward pass on the target accelerator.
4. Result return: Transfer the computation results back to the original accelerator.

This process is repeated for every MoE layer. The more layers there are, the higher the communication frequency.

4.2 How Communication Latency Eats Up Compute

Assume the latency of a single all-to-all communication is T_comm, and the expert computation time for a single layer is T_comp. When T_comm and T_comp are on the same order of magnitude, accelerator utilization will drop sharply. In extreme cases, compute utilization may fall below 30%.

This is why official recommendations suggest supernodes composed of 64 or more accelerators—not to simply stack compute power, but to reduce all-to-all communication latency to an acceptable range through high-bandwidth interconnects (such as high-speed inter-chip interconnects).

5. Local Deployment Cost Estimation Framework

Synthesizing the above analysis, we can outline an actionable estimation framework:

5.1 VRAM Requirements

Total VRAM = Weights + KV Cache + Runtime Workspace + Communication Buffers
           ≈ 1500 GB + (Concurrency × Context Length × Layers × Hidden Dimension × Precision) + 200~500 GB

5.2 Number of Accelerators

Minimum Cards = Total VRAM ÷ VRAM per Card
Recommended Cards = Minimum Cards × 1.5 (Allowing for communication and peak overhead)

Using an 80GB card as an example, the weights alone require at least 19 cards. Factoring in runtime overhead and communication margins, 64 or more cards are actually recommended.

5.3 Cost Composition

Cost Item	Description
Hardware Procurement	Accelerators, supernode interconnects, host machines, storage
Power and Cooling	Power consumption for trillion-level model inference can reach tens of kilowatts
Operations & Maintenance	Distributed inference framework tuning, troubleshooting
Opportunity Cost	Self-building cycles are typically measured in months

6. Practical Recommendations

1. Calculate communication costs before memory costs: Many teams fail to achieve expected throughput despite having sufficient VRAM, fundamentally due to inadequate interconnect bandwidth.
2. Quantization is not a panacea: MXFP4 has already compressed weights close to the theoretical limit; further compression space is limited and will lead to accuracy loss.
3. Prioritize hybrid deployment strategies: Keep frequently invoked experts resident in VRAM, place low-frequency experts in high-speed storage tiers, and swap them in on demand.
4. Plan capacity properly: The KV Cache grows linearly with concurrency; production environments must reserve capacity based on peak concurrency.
5. Evaluate cloud solutions as a benchmark: When self-hosting costs are high, using elastic accelerator instances in general-purpose cloud services for a proof-of-concept deployment can significantly reduce upfront investment risk.

Deploying a trillion-parameter model locally is not simply a matter of “buying cards and plugging them in.” It is a systems engineering effort involving memory calculation, quantization formats, interconnect topologies, and communication scheduling. Hopefully, the estimation framework in this article will help readers establish clear cost expectations before getting started.