Deploying Trillion-Parameter MoE Models Locally: A Complete Derivation of VRAM and Hardware Costs

1. The Starting Point: Parameter Count Does Not Equal VRAM Usage, but Storage Must Scale with Total Parameters

When discussing the local deployment cost of trillion-parameter MoE models, the first misconception to clarify is “because active parameters are few, VRAM requirements are low.” This is only half true—the activation ratio affects compute consumption, while the lower limit of VRAM capacity is determined by total parameters.

Take a MoE architecture with 2.8 trillion total parameters as an example: out of 896 experts, 16 are activated per token, meaning the expert activation ratio is about 1.79%. During inference, the routing network selects the top-16 experts based on the token’s hidden state. The selected experts participate in the forward pass, while the rest remain idle. However, “idle” does not mean “non-existent”—the weights of all 896 experts must reside in the storage medium, waiting to be routed. Therefore, the first golden rule of VRAM planning is:

MoE Deployment VRAM Lower Limit = Total Parameters × Bits Per Parameter ÷ 8 + Non-Weight Overhead

The activation ratio only dictates compute load and memory bandwidth pressure, not the capacity floor. Confusing the two can lead to budget estimation errors by several multiples.

2. Precise Calculation of Weight Storage Limits: From “Raw Data” to “Actual Footprint”

2.1 Quantization Format Determines Bytes Per Parameter

Assume the model introduces quantization-aware training during the supervised fine-tuning stage, using a 4-bit floating-point format for weights and an 8-bit floating-point format for activations. Calculated at a pure 4-bit rate, the raw data lower limit for 2.8 trillion parameters is:

2.8 × 10^12 × 4 bit ÷ 8 = 1400 GB

This is the most idealized number and is practically impossible to achieve in actual deployment. The reason lies in the overhead of block quantization formats: in common 4-bit block quantization specs, every 32 4-bit elements are accompanied by an 8-bit scale factor. The effective bits per parameter calculate to:

4 bit + (8 bit ÷ 32) = 4.25 bit

Thus, the actual footprint of weight elements plus scale factors is:

2.8 × 10^12 × 4.25 bit ÷ 8 ≈ 1487.5 GB

2.2 Four Easily Overlooked Overheads

The 1487.5GB figure is still only the theoretical value for the main weights. A complete VRAM budget must also account for:

• Non-4-bit Tensors: Embedding layers, output heads, normalization layer parameters, and routing network weights are typically not 4-bit quantized. They are stored in BF16 or FP8, which can amount to tens of GBs in a trillion-parameter model.
• Metadata and Alignment Padding: Tensor descriptors, shard indices, and memory alignment holes generally require a 3% to 8% reservation of the total weight size.
• Runtime Workspace: Operator temporary buffers, communication buffers, and graph capture memory pools are strongly correlated with batch size and sequence length.
• KV Cache: Grows extremely fast in long-context scenarios and is the most unpredictable part of inference VRAM.

A pragmatic engineering estimation formula can be written as:

Total VRAM ≈ Weights (including scales) × 1.10 + KV Cache + Workspace + Framework Reservation

Based on this, the weight-side lower limit for a full-capacity deployment of a 2.8 trillion parameter model is around the 1.6TB mark. Adding the KV cache and workspace, the effective VRAM requirement for a single machine falls in the 1.8TB to 2.2TB range, depending on context length and concurrent batch size.

2.3 Horizontal Reference: Medium-Scale MoE Deployment Baseline

Using a common medium-scale MoE as a reference can calibrate our sense of magnitude. For a MoE model with around 670B total parameters plus roughly 14B auxiliary parameters for multi-token prediction, the main and auxiliary parameters combined take about 685GB in FP8 format. If 4-bit weight quantization and low-bit activation are used, the weights take about 335GB and peak activation about 37GB, totaling around 400GB, which can run on 8 80GB accelerators (640GB total).

In contrast, the scale of a 2.8 trillion parameter model is over four times that of this reference model, pushing the weight lower limit directly above 1.4TB. This explains why the barrier to entry for a full deployment isn’t “a few cards” but rather “a super-node.”

3. Compute and Communication: The Real Bottleneck of MoE Inference is Not VRAM

3.1 Activation Ratio Determines Compute, Not Capacity

Activating only 1.79% of experts per token means the forward compute for a single token is far less than that of a dense model with the same total parameters. This is the economic source of MoE: trading storage for compute. However, the cost is irregular memory access—different tokens route to different experts, making weight read patterns highly discrete. Consequently, memory bandwidth utilization often fails to reach the levels of dense models.

3.2 All-to-All Communication is the Hidden Tax of Scaled Deployment

When 896 experts are distributed across multiple accelerators, the inference path for each token is:

1. The routing network computes the expert selection.
2. Token activations are sent to the target devices via all-to-all communication.
3. The experts perform the computation.
4. The results are moved back to the original device via all-to-all communication.

This round-trip communication volume grows linearly with expert parallelism degree, and communication latency directly erodes compute utilization. When experts are scattered across dozens of cards, the bandwidth demand for all-to-all communication can exceed the single-card interconnect bandwidth by several times, becoming the decisive factor for end-to-end throughput. This is the fundamental reason why official recommendations suggest using a super-node composed of 64 or more accelerators: high-bandwidth, low-latency intra-node interconnect (rather than standard Ethernet) is a prerequisite for maintaining MoE inference efficiency.

3.3 Trade-offs in Deployment Topology

Three common parallel strategy combinations in practice:

• Expert Parallelism (EP): Experts are sharded across different devices. High communication volume but low per-card VRAM pressure, suitable for high-bandwidth interconnects within a super-node.
• Tensor Parallelism (TP): Splits within a single layer. Communication-intensive but low latency, usually restricted to a single 8-card machine.
• Pipeline Parallelism (PP): Splits by layer. Low communication volume but introduces pipeline bubbles, suitable for cross-node scaling.

The typical configuration for trillion-scale MoE is “intra-node TP + inter-node EP,” with parallel dimensions planned separately for attention layers and MoE layers. Attention layers have fewer parameters and are communication-sensitive, favoring TP; MoE layers have massive parameters and are naturally shardable, favoring EP.

4. Cost Estimation Framework: From VRAM Demand to Budget Range

4.1 Three Tiers of Hardware Costs

The total cost of ownership for local deployment can be broken down into:

Tier	Components	Estimation Focus
One-time Hardware	Accelerators, interconnect switches, storage, power, and racks	Estimate card count based on (Effective VRAM Demand ÷ Per-card VRAM) × 1.3 redundancy factor
Operating Costs	Power, cooling, operations and maintenance (O&M) personnel	Trillion-scale models under full load typically draw tens of kilowatts; electricity costs cannot be ignored
Hidden Costs	Model loading time, fault recovery, version management	Loading 1.5TB of weights from storage to VRAM takes tens of minutes

4.2 A Reusable Estimation Process

# Pseudo-code: Full deployment VRAM and card count estimation
total_params = 2.8e12
bits_per_param = 4.25          # 4-bit weights + amortized block scale factor
weight_gb = total_params * bits_per_param / 8 / 1e9
overhead_ratio = 0.10          # Non-quantized tensors, metadata, alignment
kv_cache_gb = estimate_kv(context_len=32768, batch=8, layers=..., heads=...)
workspace_gb = 80              # Communication buffers and operator workspace, varies with parallel strategy
total_vram_gb = weight_gb * (1 + overhead_ratio) + kv_cache_gb + workspace_gb
cards_needed = ceil(total_vram_gb / per_card_vram_gb * 1.3)  # 30% redundancy

Following this process, the weight side is roughly 1.64TB, with KV cache and workspace accounting for several hundred GBs. Under the 80GB-per-card spec, the required number of cards falls in the 32 to 48 range. Considering communication topologies and redundancy, the actual deployment scale is typically aligned to a 64-card super-node. This is the arithmetic logic behind the “official recommendation of 64 or more cards.”

4.3 Realistic Perception of Cost Magnitude

For a local full-capacity deployment of a trillion-scale model, hardware investment is in the tens of millions of RMB range, and annual operating costs (power, cooling, O&M) are in the millions of RMB range. For most teams, the three more pragmatic paths are:

• Reduced Precision Deployment: Further compress weight bits, trading precision for VRAM, but requiring quantization recalibration.
• Partial Expert Residency: Place cold experts in host memory or high-speed storage, swapping them in on-demand, trading latency for capacity.
• On-Demand Rental Instead of Self-Building: Use cloud GPU instances or containerized inference services billed by the hour to avoid one-time hardware investments and depreciation risks.

5. Decision Checklist: Six Questions to Answer Before Deployment

1. What is the target context length? KV cache under long contexts might exceed the weights themselves, directly determining the card count.
2. What is the peak concurrent batch size? Workspace and KV cache grow linearly with batch size.
3. What is the acceptable per-token latency? The all-to-all communication topology dictates the latency floor.
4. Does precision allow going below 4-bit? For every 1-bit reduction, weight VRAM decreases by about 12%.
5. Is intra-node interconnect bandwidth sufficient? The higher the EP degree, the more stringent the interconnect requirements.
6. Are model loading and fault recovery times acceptable? Loading TB-scale weights is a minute-level operation requiring preloading and hot-standby strategies.

Conclusion

The local deployment of trillion-parameter MoE models is essentially an engineering arithmetic problem of “trading storage for compute.” The VRAM lower limit is locked by total parameters and quantization format, compute efficiency is determined by the activation ratio, and end-to-end throughput is ultimately dictated by the all-to-all communication topology. Calculating these three layers separately and stacking them incrementally yields a credible budget range—rather than being misled by figures like “only 1.79% parameters activated” and underestimating the true barrier to full-capacity deployment.