Trillion-Parameter MoE Model Local Deployment: How Much VRAM and Hardware Do You Actually Need?

The open-source large model wave is pushing parameter scales from hundreds of billions to trillions. For teams with strict data compliance requirements, ultra-low latency needs, or long-term massive inference demands, local deployment remains an unavoidable path. However, the first practical challenge facing trillion-parameter models is not “how to deploy,” but “how much memory, how many GPUs, and what budget is required?”

This article uses a typical next-generation trillion-parameter open-source MoE model as an example. Based on publicly available information, it has approximately 2.8 trillion total parameters, utilizes a Mixture of Experts architecture, and natively outputs 4-bit weights with quantization-aware training integrated from the supervised fine-tuning (SFT) stage. We will walk through the methodology for estimating a local deployment budget.

Note: All speed, VRAM, and hardware configuration figures below are estimates based on public information prior to the official weight release and must be verified with benchmark tests once the weights are available. The methodology itself is universal and can be applied to any model by substituting the parameters to generate a corresponding budget table.

1. Common Misconception: MoE VRAM Must Be Calculated by Total Parameters

This model has 896 experts, but each token only activates 16 experts, meaning the expert activation ratio is about 1.79%. A common first reaction to this number is: “Since activation is under 2%, can VRAM also be calculated based on just 2%?” This is the most frequent pitfall in estimation.

The MoE routing logic can be simplified as follows:

for token in sequence:
    scores    = router(token)               # Score all 896 experts
    top_ids   = topk(scores, k=16)         # Select the top 16 experts
    token_out = sum(experts[i](token) for i in top_ids)  # Only these 16 experts compute

Which 16 experts a token hits is dynamically determined by the router during inference based on the token’s semantic content. You cannot predict which experts will be used, so all 896 experts must be loaded into VRAM (or an exchangable tiered storage area, which incurs extra bandwidth and latency costs). An analogy: a library has 896 books, and each reader borrows 16 at a time, but the bookshelf must hold all 896. The borrowing ratio affects the “workload per service,” not the size of the bookshelf.

Two more points need clarification:

• Expert activation ratio does not equal parameter activation ratio. Shared components like the attention backbone, embedding layers, and normalization layers are fully activated for every token. The 1.79% only describes the routing sparsity of the expert layers. Computational power estimates should follow the activated parameter count, not total parameters multiplied by 1.79%.
• The correct MoE principle is: “Compute follows activated parameters; storage follows total parameters.” This asymmetrical structure is the exact design intent of MoE—trading larger memory for several times lower compute overhead. It is also why its local deployment threshold is significantly higher than that of dense models.

Conclusion: The weight storage baseline for this model is dictated by its 2.8 trillion total parameters and cannot be discounted by the activation ratio.

2. The True VRAM Baseline: 4-bit Weights Are Not “Total Parameters Divided by Four”

This model executes quantization-aware training from the SFT stage, natively outputting Microscaling 4-bit floating-point format (MXFP4) weights paired with MXFP8 activations. Quantization precision is “baked in” during training, resulting in far less quality loss during inference compared to post-training weight quantization. This is a prerequisite for its deployability.

The raw data baseline is simple: 2.8T × 4 bit ÷ 8 ≈ 1400GB. However, Microscaling formats do not quantize elements independently; they use group quantization—every 32 4-bit elements share a single 8-bit group scale factor, adding an average overhead of 1/4 byte per parameter:

total_params   = 2.8e12          # Total parameters
bits_per_param = 4 + 8 / 32      # 4bit data + 8bit scale shared per 32 elements → 4.25bit
weight_GB      = total_params * bits_per_param / 8 / 1e9   # ≈ 1487.5 GB

The seemingly insignificant “one extra byte per 32 elements” alone contributes nearly 90GB of additional budget. This is still just the weight data itself; actual usage must also account for:

• Non-quantized tensors: Embedding layers, router scoring matrices, and normalization layers typically retain higher precision and do not benefit from 4-bit compression.
• Checkpoint metadata and tensor alignment padding.
• Runtime workspace: GPU kernel workspaces, activation buffers, and inter-GPU communication buffers.
• KV cache: This model uses a Delta attention variant with attention residual connections and gated latent attention components, which effectively suppresses KV growth for long contexts. However, the actual KV footprint per token still needs to be calculated once the official configuration is released, and it scales linearly with batch size and context length.

Rule of thumb: Above the weight baseline, reserving at least an additional 15%–30% margin is the bare minimum just “to run.” For a production service to run smoothly and support meaningful concurrency, planning a total VRAM budget of 1.5 to 1.7 times the weight memory is recommended.

3. Cross-Validation: Calibrating Estimates with a Smaller Open-Source Model

Relying solely on extrapolation can lead to overconfidence. We need to cross-validate using the industry’s most referenced small-scale model with the same architecture—a mainstream open-source 671B parameter MoE model (including the next-token prediction head, totaling about 685B effective parameters):

• Using FP8 (1 byte per parameter), weights are about 685GB.
• Using grouped 4-bit weight quantization, it requires about 335GB for weights plus 37GB for the activation workspace, totaling roughly 400GB.
• Actual deployment experience: It can run on 8 flagship inference GPUs with 80GB HBM each (640GB total). Weights and fixed overheads account for just over 60% of the total VRAM, leaving the rest for KV cache and concurrency.

This empirical data gives us a usable extrapolation factor: a 4-bit MoE model requires a total VRAM capacity roughly equivalent to “weights divided by 0.6” to reach a healthy production concurrency level. Applying 2.8T to the same framework:

Quantization Format	Weight VRAM (Est.)	Bare Minimum to Run	Recommended Configuration
BF16	~5.6TB	—	No deployment value currently
FP8	~2.8TB	3TB+	Not recommended unless precision benefits are explicit
MXFP4	~1487.5GB	~1.8TB	2.2TB–2.6TB

4. Hardware Configuration Options: Two Self-Purchase Paths and Cost Breakdown

Option A: Flagship inference GPUs with 80GB HBM per card.
It takes about 19 cards just to fit the weights. 24 cards (1920GB total) skirts the usable boundary, leaving only a ~20% margin and supporting very limited concurrency. 32 cards (2560GB total) is the recommended level, where weights occupy about 58%, consistent with the deployment posture of the 671B reference model. 32 cards are typically delivered as four 8-GPU servers, requiring a 100Gbps-level RDMA high-speed interconnect network to support expert parallelism. Based on recent market conditions, a single GPU is priced in the hundreds of thousands of dollars range, and a complete 8-GPU server can exceed a million dollars. The total self-purchase budget will roughly fall into the multi-million dollar range.

Option B: New generation large-memory inference GPUs (141GB/192GB per card).
Weights will occupy at least 8 cards. 12 cards (~2304GB) reaches the recommended level, while 16 cards (~3072GB) is highly comfortable and leaves room for higher future concurrency. Fewer cards mean reduced inter-GPU communication complexity, interconnect network, and switching equipment costs. Although the unit price per card is higher than the 80GB tier, the total investment is often more cost-effective, making it worthwhile to request parallel quotes for both configurations during procurement.

Beyond self-purchasing, do not overlook three easily missed cost lines:

• Host memory and storage: Fast loading of terabyte-scale weights requires host-side memory and high-speed NVMe support. A single 4-bit checkpoint is about 1.5TB, and operations require multiple replicas for primary-backup and rollback.
• Power and cooling: Dozens of flagship GPUs plus host and network equipment can easily push cluster rated power to tens of kilowatts. Electricity and datacenter hosting are continuous expenses.
• Engineering labor: Adapting new quantization formats, tuning multi-GPU parallel strategies, and running benchmarks will consume weeks of senior engineer working hours.

Option C: Cloud rental, converting CapEx into OpEx.
If utilization is low or demand is only short-to-medium term, building your own is not cost-effective. You can consider hourly elastic rentals on cloud GPU instances (such as Baidu AI Cloud), releasing them as needed. A rough break-even calculation: assuming the hourly quote for a single card is $P, a 16-card large-memory cluster running continuously year-round costs 16 × P × 8760 per year. Compare this against the depreciation (typically amortized over 3–5 years), power, and maintenance labor of a self-purchase plan. If annual usage falls below the break-even point, renting is better; if above it, buying is better.

5. Post-Release Verification Checklist

The entire point of pre-release estimation is to validate it post-release. Once official weights and technical reports are available, verify and adjust your budget table item by item using the following checklist:

1. Exact parameter count and per-layer quantization distribution: Whether embeddings, routing, and attention are included in the 4-bit scope and how group scales are arranged will directly refine the 1487.5GB figure.
2. Idle VRAM measurement: Actual usage after loading weights should closely match the estimate; large deviations indicate a misread of the quantization format.
3. Target concurrency benchmarks: Time to first token, decoding speed, and the concurrency inflection point where the KV cache saturates VRAM—these three metrics determine the actual throughput ceiling.
4. Long-context validation: The KV growth curve at 128K context length will test the actual compression efficacy of the gated latent attention components.
5. Factor backtesting: Use measured data to re-validate the “weights divided by 0.6” recommendation coefficient, then double-check the procurement list to prevent configuration mismatches.
6. Deviation attribution: If actuals severely mismatch estimates, first eliminate engineering variables like tensor parallelism and expert parallelism strategies, communication patterns, and batch sizes before adding more hardware.

6. Conclusion: Three Key Budget Takeaways

• VRAM: Storing 2.8T parameters in native 4-bit requires roughly 1487.5GB (including group scale overhead). Factoring in workspaces and KV cache, 1.8TB is the absolute floor, while 2.2TB or above is the recommended budget.
• Hardware: A deployable cluster requires 32 80GB-class GPUs or 12–16 large-memory cards. Depending on market conditions, self-purchase budgets fall in the multi-million dollar range, plus continuous costs for power, hosting, and maintenance labor.
• Path selection: For most small to medium teams, directly calling APIs or renting GPU instances on cloud platforms is far more economical than self-hosting. Full local parameter deployment is only justified when hard constraints like data sovereignty, ultra-low latency, or long-term stable high-volume usage exist.

To reiterate: All figures in this article are estimates based on public information and must be verified item by item with benchmark tests once official weights are released. The value of estimation lies not in absolute precision, but in allowing you to clarify your decision-making bottom line before the budget and procurement windows close.