Estimating VRAM and Hardware Costs for Local Deployment of Trillion-Parameter MoE Models

1. What Trillion-Parameter Scale Means for VRAM

Let’s first look at the parameter scale. With a total of 2.8 trillion parameters using an MoE architecture, assume a configuration of 896 experts where each token activates 16 of them. This results in an expert activation ratio of roughly 1.79%.

A common misconception needs to be clarified here: the expert activation ratio does not equate to a strict parameter activation ratio. VRAM consumption for an MoE model is calculated based on the total number of parameters. The activation ratio only affects compute consumption, not weight storage.

In other words, you must first load the weights of all 896 experts into the storage medium. During inference, the routing network dynamically selects the 16 activated experts to participate in computation. This means the lower bound of VRAM capacity is dictated by the total parameters, not the activated parameters.

If the architectural backbone uses KDA (an incremental attention mechanism) combined with Attention Residuals, and incorporates Gated MLA as a component, the overall structure remains a typical “attention backbone + MoE feed-forward layer” design. The core contradiction in MoE deployment is this: computation is sparse, but storage is dense.

Next, let’s consider quantization. Assuming the model underwent quantization-aware training (QAT) from the SFT stage, with weights using MXFP4 and activations using MXFP8. Converting 2.8 trillion parameters to 4-bit yields a raw data lower bound of approximately:

2.8e12 × 4 bit ÷ 8 = 1400 GB

However, MXFP4 is not purely 4-bit. According to the OCP specification, every 32 4-bit elements must be accompanied by an 8-bit scale factor. Therefore, the average bit width per weight element is actually:

4 + 8/32 = 4.25 bit

Recalculating with this:

2.8e12 × 4.25 bit ÷ 8 ≈ 1487.5 GB

This is merely the raw data lower bound for the weights themselves. It does not account for non-4-bit tensors, metadata, memory alignment, runtime workspaces, or the KV Cache. During actual deployment, these overheads stack up, further increasing the VRAM requirement.

For reference, consider an MoE model with approximately 671B total parameters: under FP8 format, the main parameters plus MTP parameters total around 685GB. Even with 4-bit weight quantization, the weights take up about 335GB, and adding roughly 37GB for activations, it takes a combined 400GB to run on 8 80GB accelerators (640GB total). The 2.8 trillion parameter model is more than four times that size. The lower VRAM bound for a full-precision deployment is thus determined: a single 8-GPU node is no longer sufficient; a multi-node super-node architecture is mandatory.

2. Three Tiers of Deployment Goals and Configuration Estimates

Tier 1: Just Running, Regardless of Speed

The goal at this tier is simply that “the model can load and produce results,” with no hard requirements for throughput or latency. The core constraint here is VRAM capacity.

Based on previous estimates, the raw weight data is roughly 1487.5GB. Adding the KV Cache, workspace, and framework overhead, a conservative estimate requires over 1600GB of available VRAM. If a single accelerator has 80GB of VRAM, you need at least 20 cards; considering actual utilization rates and memory fragmentation, planning for 24 cards is recommended. If more aggressive quantization or tiered offloading is used (placing some experts in host memory or even on NVMe drives), the VRAM threshold can be lowered, but inference speed will drop drastically, with Time-To-First-Token (TTFT) potentially reaching tens of seconds.

The cost structure for this tier is primarily driven by VRAM capacity costs. Assuming the use of 80GB accelerators, based on current market rates, multiply the acquisition or rental cost of a single card by 20 to 24. Add the server chassis, power supply, cooling, and networking, and you have the minimum threshold for investment. If renting cloud instances billed by the hour, running a full inference verification might only take a few hours, keeping costs relatively manageable.

Tier 2: Single-User Usage, Output Speed at 25 tokens/s

A speed of 25 tokens/s is an experience threshold close to human reading speed. To achieve this, the bottleneck shifts from “VRAM capacity” to “VRAM bandwidth + compute + communication.”

For an MoE model, the computation path for a single token is: the routing network selects 16 experts → distributes the activations to the cards hosting those experts → experts compute → results are gathered. If all experts reside on the same card, there is no communication overhead, but you won’t have enough VRAM. If experts are distributed across multiple cards, all-to-all communication latency becomes the dominant factor.

Assume a hybrid strategy of tensor parallelism + expert parallelism, deployed across several high-speed interconnected accelerators. For a single user requiring 25 tokens/s, the communication volume per forward pass can be estimated based on the activation size. With a hidden dimension in the thousands and 16 activated experts, the data transferred per all-to-all operation is in the hundreds of megabytes. To complete a full “distribute—compute—gather” cycle within tens of milliseconds, the interconnect bandwidth needs to be in the hundreds of GB/s range. This necessitates NVLink or an equivalent high-speed interconnect.

The hardware configuration for this tier roughly requires: at least 8 to 16 high-bandwidth accelerators forming a single communication domain, with total VRAM satisfying weight loading and interconnect bandwidth satisfying the all-to-all requirements. In terms of cost, beyond the accelerators themselves, the cost of high-speed switches and cables cannot be ignored. An 8-card super-node requires an investment ranging from tens of thousands to hundreds of thousands of dollars (or equivalent local currency). Renting a cloud-based super-node instance typically costs hundreds of dollars per hour.

Tier 3: 10 Concurrent Users, Single-User Output Above 25 tokens/s, Quantization Constraints

This is the most demanding scenario. Ten concurrent inference requests simultaneously, with each request maintaining an output speed of over 25 tokens/s. Additionally, the quantization level cannot drop below a certain threshold (e.g., you cannot drop to 2-bit just to save VRAM).

This scenario requires simultaneously satisfying three constraints:

1. VRAM Capacity: Weights + 10 KV Caches + Workspace. The KV Cache scales with sequence length and concurrency. Under 10 concurrent requests, the KV Cache alone can reach tens of GB or even over a hundred GB.
2. Compute Throughput: Total throughput must reach 10 × 25 = 250 tokens/s or higher. Considering the expert activation characteristics of MoE, actual compute requirements must be calculated based on activated parameters and routing overhead.
3. Communication Bandwidth: The activation distribution of 10 requests further amplifies all-to-all communication pressure. The communication domain needs to be larger, or a more efficient expert placement strategy must be adopted.

It makes sense that official recommendations suggest deploying such trillion-parameter MoE models on super-nodes composed of 64 or more accelerators. Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth. With 896 experts distributed across multiple cards, every token must route its activations to the cards hosting the target experts, compute, and route them back. If the communication domain isn’t large enough, the latency of cross-domain transfers will directly eat into compute, causing accelerator utilization to plummet.

Configuration recommendations for this tier:

• Accelerator Count: 64 or more, forming a single high-speed communication domain.
• Interconnect Method: NVLink or equivalent high-speed interconnect, ensuring all-to-all operations do not cross low-speed domains.
• Total VRAM: Based on 1500GB for weights + KV Cache + Workspace, it is recommended to reserve over 2TB.
• Quantization Level: No lower than an MXFP4/MXFP8 hybrid scheme to ensure a balance of precision and speed.

Cost-wise, acquiring a 64-card super-node is in the millions of dollars, with rental costs reaching thousands of dollars per hour. For most teams, this tier is better suited for on-demand cloud rental rather than building on-premises.

3. Cost Estimation Framework and Key Variables

Synthesizing the three tiers, the cost of locally deploying a trillion-parameter MoE model can be broken down into the following formula:

Total Cost = Accelerator Cost + Interconnect Cost + Host Cost + O&M Cost
Where:
Accelerator Cost = Single Card Cost × Number of Cards
Number of Cards = max(VRAM Requirement / Single Card VRAM, Compute Requirement / Single Card Compute, Minimum Communication Domain Size)

Key variables include:

• Quantization Format: 4-bit weights + 8-bit activations is the current mainstream quantization scheme for large models. However, the actual bit width and overhead vary between implementations, directly impacting the VRAM lower bound.
• KV Cache Strategy: Whether PagedAttention is used, or whether the KV Cache is quantized, has a massive impact on VRAM consumption in concurrent scenarios.
• Expert Placement Strategy: Whether experts are uniformly distributed or distributed based on popularity affects communication patterns and load balancing.
• Interconnect Topology: Fully connected, fat-tree, or ring topologies determine the actual effective bandwidth of all-to-all operations.

Before the weights are officially released, developers are advised to conduct benchmark tests using a smaller-scale MoE model. Verify the actual performance of your inference framework regarding all-to-all communication, expert routing, and KV Cache management, and then extrapolate proportionally to the trillion-parameter scenario. Estimations derived this way will be far more reliable than pure theoretical calculations.

4. Summary

Local deployment of a trillion-parameter MoE model is fundamentally a tri-constraint problem involving VRAM capacity + interconnect bandwidth + compute throughput. The three tiers correspond to different dominant constraints:

• Just running → Dominated by VRAM capacity; card count configured based on the weight lower bound.
• Single user at 25 tokens/s → Dominated by interconnect bandwidth; requires a high-speed communication domain.
• 10 concurrent users at 25 tokens/s → All three dominate simultaneously; requires a super-node of 64 or more accelerators.

Prior to the official release of weights and technical reports, all figures should be treated as engineering estimates. Once benchmark data is available, verifying VRAM consumption, communication latency, and actual throughput item by item will yield a truly reliable deployment plan.