Hardware Requirements and Cost Estimation for Deploying a Trillion-Parameter MoE Model On-Premises

Why Parameter Count Does Not Directly Equal VRAM Demand

Before discussing on-premises deployment, it is essential to understand a key premise: for a Mixture-of-Experts (MoE) model, the total parameter count does not linearly correspond to computational load during inference.

Take the target model discussed in this article as an example: it has roughly 2.8 trillion total parameters and uses an MoE architecture with 896 internal expert networks. During the forward pass, each token activates only 16 of those experts, putting the expert activation ratio at approximately:

16 ÷ 896 ≈ 1.79%

However, it must be emphasized that the expert activation ratio does not equal the parameter activation ratio. Each expert still contains multiple attention layers, feed-forward networks, and normalization structures, and expert parameter sizes may differ. The activation ratio mainly affects compute density and arithmetic intensity, whereas VRAM usage follows a different logic.

For MoE inference deployment, the lower bound on VRAM is dictated by the sum of all expert weights, not the small fraction activated per token. The reasoning is straightforward: an inference service must retain the ability to route any token to any expert at any time. The system cannot assume certain experts will never be reached. All experts must be loaded into storage before inference begins, and values are fetched on demand based on routing decisions.

This is a crucial and often overlooked constraint.

How Architectural Features Affect Storage and Communication

The model’s backbone uses a Delta Attention mechanism with residual connections between attention layers. A compressed latent attention (MLA) component serves as a key element, reducing key-value cache footprint while preserving long-context modeling capability.

From a deployment perspective, this architecture introduces three major concerns:

1. Attention cache pressure
   Compared to conventional multi-head attention, compressed latent attention significantly reduces the growth rate of the KV cache over long sequences. Still, this does not mean the cache is negligible. As concurrent requests or context length increase, the cache competes with weights for limited VRAM.

2. Routing and communication patterns
   With 896 experts distributed across multiple accelerators, every token requires routing and sending its intermediate activations to the device hosting the selected expert. Once computation finishes, results are returned. This “send—compute—fetch” cycle repeats at every Transformer layer.

3. All-to-All communication sensitivity
   When experts are spread across devices, the communication topology directly determines inference latency. If devices are connected by low-speed buses, routing overhead dominates and can exceed expert computation time. Therefore, large-scale MoE inference generally requires high-bandwidth, low-latency interconnect solutions.

Quantized Weight Formats and Memory Estimation

Assume the model has undergone quantization-aware training during the post-training phase. In this scenario, weights are stored in 4-bit floating-point format, and activations use 8-bit floating-point format. Memory cannot be estimated simply by multiplying parameter count by 4 bits.

In this 4-bit format, every 32 elements include an 8-bit scaling factor. Therefore, the effective storage per weight element is approximately:

4 bit + (8 bit ÷ 32) = 4.25 bit

For 2.8 trillion parameters, the raw weight footprint becomes roughly:

2.8 × 10^12 × 4.25 bit ÷ 8 bit/Byte ÷ 1024 ≈ 1487.5 GB

This is only a lower bound for storage. In practice, additional space must be reserved for:

• Non-quantized tensors, such as certain normalization parameters and embedding layers;
• Padding space resulting from data alignment;
• Runtime workspace;
• KV cache during inference;
• Communication buffers and temporary tensors.

As a result, the actual system footprint for a single set of weights will be noticeably higher than 1.5 TB. Reserving an extra 20% to 35% of headroom is a reasonable approach.

Comparison with a Common Sparse Model

To put this scale into perspective, consider a widely discussed 671B-parameter MoE model in the community.

Under FP8 format, the main model weights are about 671 GB. Including auxiliary parameters from the multi-token prediction module, total weight reaches approximately 685 GB. With 4-bit weight quantization, the weight portion drops to about 335 GB, and activation plus runtime space adds roughly 37 GB, for a total of about 400 GB. Such a model can be deployed on eight 80 GB accelerators.

The model discussed in this article has four times more total parameters. Extrapolating from parameter scale alone, its memory demand would also be at least four times higher. This explains why official recommendations point to high-performance compute nodes with dozens of accelerators, rather than stitching together a few regular servers.

Supernode Deployment and Communication Topology

A trillion-parameter model simply cannot fit on a single GPU or a single server. A supernode approach is required.

A supernode refers to a logical computing domain where dozens or more accelerators are connected through high-speed interconnect and treated as a unified resource pool. Its core value lies in:

• Distributing all expert weights across every device in the supernode;
• Confining the heavy All-to-All communication generated by token routing to a single high-speed domain;
• Reducing data movement across slower network segments;
• Lowering tail latency and improving overall throughput stability.

When planning a supernode, interconnect bandwidth design should prioritize the following metrics:

Per-GPU aggregate bandwidth (GB/s)
Cross-device communication latency (μs)
All-to-All effective throughput (GB/s)
In-node topology convergence ratio

If these metrics are insufficient, high per-GPU compute power will not help because expert routing communication becomes the bottleneck. The observed system behavior will be “unable to move data fast enough,” not “unable to compute fast enough.”

Recommended Hardware Configurations

Exact configurations backed by benchmark testing are not possible before the official weight release. However, based on parameter scale and general industry experience, the following tiered references can be established:

Entry-Level Configuration (Basic Weight Loading Only)

• At least 2 TB of usable VRAM;
• 16 to 32 high-end accelerators;
• High-bandwidth direct connectivity between cards;
• Suitable for single-request, low-concurrency scenarios;
• Requires heavy use of memory sharing or tiered weight loading techniques.

Mid-Range Configuration (Limited Concurrent Long-Context Inference)

• 4 to 8 TB of total VRAM;
• 64 or more accelerators;
• High-speed interconnect within a supernode;
• Supports several long-context requests concurrently;
• Requires careful partitioning of KV cache and weights.

High-Availability Configuration (Production-Grade Inference Service)

• Dual supernodes or a larger compute pool;
• Redundant devices and online failover mechanisms;
• Dedicated storage and model distribution channels;
• Real-time monitoring and communication bandwidth alerts.

How to Estimate On-Premises Deployment Costs

The total cost of local deployment is not limited to accelerator procurement. A complete cost model should include at least the following components:

1. Compute Hardware Cost

Assuming 64 high-end accelerators, with per-card pricing in the current mainstream market range, the accelerator cost alone would likely be:

64 × per-card price (hundreds of thousands to millions of RMB)

This single line item could reach tens of millions of RMB.

2. High-Speed Network Equipment

Supernodes require dedicated switches, optical modules, and high-speed cables. The larger the communication domain, the higher the network cost share, typically 10% to 25% of compute hardware cost.

3. Storage and Host Systems

Full model weights require high-speed persistent storage for loading and version switching. Recommended setup includes:

• All-flash storage arrays;
• Multiple high-memory host servers for model loading and data preprocessing;
• Backup copies for rapid rollback.

4. Racks, Power, and Cooling

Dozens of accelerators typically consume tens of kilowatts. Required infrastructure includes:

• Dedicated high-power racks;
• Liquid cooling or high-efficiency air cooling;
• Uninterruptible power supply and dual utility feeds.

5. Operations and Personnel

On-premises deployment also requires ongoing investment in model version management, monitoring and alerting, failure recovery, and security hardening.

All things considered, even with a cost-conscious hardware plan, the initial investment for a full production-grade local deployment is very likely to reach tens of millions of RMB, with substantial annual electricity and maintenance costs on top.

Key Checks Before Deployment

Once the official weights are released, do not deploy directly to production. Complete the following verification steps in order:

1. Measure actual VRAM consumption
   Test with different batch sizes and context lengths to chart the real memory footprint under quantized weights.

2. Benchmark single-GPU and multi-GPU inference latency
   Record the time share of routing phases versus expert computation phases.

3. Stress-test All-to-All bandwidth
   Simulate heavy concurrent routing and observe how communication latency scales.

4. Run long-context degradation tests
   Verify whether cache growth remains acceptable at 32K, 128K, or longer context windows.

5. Test failure recovery
   Simulate a single GPU failure and verify whether the supernode can fail over without interrupting service.

Conclusion

Deploying a trillion-parameter MoE model on-premises is not about solving whether the model can be loaded. The real challenge is balancing four constraints: storage, communication, compute, and cost.

Total weight size sets the lower bound on VRAM. Expert distribution dictates the importance of communication topology. The supernode design determines whether the system can operate reliably in a production environment. For most teams, estimating cost and evaluating alternative designs using the methods in this article before the weights are released is far more valuable than purchasing hardware early. After the official weights and technical report become available, validate these estimates against real measurements before making any final deployment decision.