Cost and VRAM Estimation for Local Deployment of a 2.8T-Parameter MoE Model

1. Setting the Problem Straight: Why “Fewer Activated Parameters” Does Not Mean “Lower VRAM Usage”

When evaluating MoE model deployment costs, many teams start by looking at how many parameters are activated per token, then assume they only need to provision VRAM based on activated parameters. This intuition roughly holds for dense models, but it almost completely breaks down under MoE architectures.

Consider a model with 2.8 trillion total parameters, where 16 experts are activated per token out of 896 total experts. The expert activation ratio is approximately 1.79%. This number describes the sparsity of the computation path, not the sparsity of the storage path. During inference, the router must select 16 experts from all 896, so before any inference request arrives, the system must already have all expert weights loaded into accessible storage.

In other words:

• Compute cost ≈ determined by activated parameters;
• VRAM/memory usage ≈ determined by the quantized volume of total parameters.

If you estimate VRAM based only on the 1.79% activation ratio, your conclusion will be off by more than an order of magnitude. This is the premise for all subsequent hardware reasoning.

2. Raw Weight Volume of 2.8 Trillion Parameters at 4-bit

For very large MoE models, mainstream feasible deployment routes typically rely on low-bit quantization. A commonly cited training-side quantization strategy is MXFP4 weights paired with MXFP8 activations, where quantization-aware training is introduced during the SFT stage so the model natively adapts to low-precision inference.

Calculating with pure 4-bit weights:

2.8 × 10^12 × 4 bit = 11.2 × 10^12 bit = 1400 GB

But this figure is only a theoretical lower bound. In practice, the MXFP4 format is not simply 4 bits per weight. Under common specifications, every 32 4-bit elements carry an additional 8-bit scale, which works out to approximately 4.25 bits per weight:

2.8 × 10^12 × 4.25 bit ÷ 8 ≈ 1487.5 GB

Note that this is still only the volume of the weight tensors themselves. When actually budgeting VRAM/memory, you must add at least the following:

1. Non-4-bit tensors, such as certain normalization layers, embedding layers, and output heads;
2. Metadata and routing tables;
3. Padding from tensor alignment;
4. Runtime workspace, such as activation recomputation buffers and temporary communication buffers;
5. KV cache;
6. Redundancy and mirrored regions on each card after multi-GPU partitioning.

Therefore, a stable, operational ultra-large MoE inference system typically requires 20% to 40% more storage than the raw weight volume, depending on how close the implementation gets to the theoretical minimum.

3. Using an Existing 671B Model as a Reference

To more intuitively grasp the scale of 2.8 trillion parameters, consider a MoE model with 671B total parameters, including 14B multi-token prediction parameters, as a reference.

Assume this 671B model is deployed in FP8:

• Main parameters: 671B × 1 byte ≈ 671 GB;
• Multi-token prediction parameters: 14B × 1 byte ≈ 14 GB;
• Total: approximately 685 GB.

This means that on accelerators with 80GB of VRAM per card, you need at least 8 cards with a total of 640GB of effective VRAM just to barely fit FP8 weights. With AWQ 4-bit quantization, the weight portion can be reduced to approximately 335GB, but adding activations and workspace brings the total to around 400GB, still requiring 8 or more cards.

And 2.8 trillion parameters is more than 4 times 671B. So even with a more aggressive 4-bit quantization scheme, raw weights already exceed 1.4TB. This is not a question of “can it be deployed on a single machine” but “what is the minimum cluster card count.”

4. Minimum Hardware Scale Corresponding to VRAM Requirements

Assuming each accelerator provides 80GB of VRAM:

• 4 cards: 320GB, not feasible;
• 8 cards: 640GB, not feasible;
• 16 cards: 1280GB, still below the 1487GB raw weight lower bound;
• 32 cards: 2560GB, theoretically sufficient but with limited headroom;
• 64 cards: 5120GB, relatively comfortable, and close to the officially recommended configuration of “64 or more accelerators forming a supernode.”

If accelerators offer 96GB, 128GB, or 141GB of VRAM, the minimum scale changes accordingly. Taking 141GB devices as an example:

1487GB ÷ 141GB ≈ 11 cards

But in practice, at least 16 or more cards are still recommended, because an inference system cannot only consider weight placement — it must also reserve space for KV cache, communication buffers, and runtime workspace.

Therefore, for a 2.8-trillion-parameter model, a minimum inference node is typically an investment at the level of dozens of accelerators. The more realistic approach is to treat it as a “supernode engineering” project rather than a single-machine deployment task.

5. Communication Topology: Why Card Count Is Not the Only Bottleneck

When the number of experts reaches 896 and they are distributed across multiple accelerators, computation is no longer the only bottleneck — communication also becomes a key factor determining end-to-end throughput.

A typical MoE inference flow is as follows:

1. Input tokens pass through the shared attention portion;
2. The router computes expert selection probabilities;
3. Activation values are sent to the devices hosting the selected 16 experts;
4. The target devices complete expert computation;
5. Output results are aggregated back to the original device.

Steps 3 through 5 involve all-to-all communication. The more widely experts are distributed, the more communication hops are required, and the more pronounced the latency becomes. Especially during the decode phase, every token must go through this path, and communication latency directly translates into reduced per-token generation speed.

Therefore, a simple “add more cards for more VRAM” approach is not advisable. The correct approach is to design the hardware topology so that the vast majority of communication occurs within a high-speed interconnect domain — for example, using high-bandwidth interconnects to form dozens of accelerators into a single communication domain. This also explains why ultra-large MoE inference often favors rack-level or supernode solutions with strong interconnect capabilities rather than simply purchasing multiple ordinary servers.

6. GPU-CPU Heterogeneous Inference: Trading Memory for Cost

When VRAM cost becomes the bottleneck, a common engineering approach is GPU-CPU heterogeneous inference.

The core idea is:

• The shared attention portion is compute-intensive but has relatively few parameters, making it suitable to keep on accelerators;
• The sparse expert portion, while having a huge total parameter count, only activates a few experts at a time, making it suitable for offloading to large-capacity CPU memory.

The specific flow can be summarized as:

Input
  ↓
GPU completes shared attention computation
  ↓
Router selects experts
  ↓
CPU reads corresponding expert weights from memory
  ↓
CPU completes expert computation
  ↓
Results sent back to GPU
  ↓
Continue to subsequent layers

The advantages of this approach are:

1. Single-node memory can reach hundreds of GB or even TB levels;
2. No need to provision equivalent VRAM for all total parameters;
3. Useful for low-cost prototype validation of ultra-large-scale models.

The trade-off is also obvious: data transfer between CPU and GPU becomes the new bottleneck. To avoid reloading expert weights each time, expert weights typically need to reside permanently in memory and be repeatedly read through high-speed PCIe or other interconnect channels.

Taking a dual-socket server with 512GB of memory as an example, if approximately 400GB of a 4-bit quantized MoE model is placed and paired with a single high-end consumer accelerator, the prefill phase might reach hundreds of tokens/s, while the decode phase would be in the range of low tens of tokens per second. These figures are not fixed values but fluctuate significantly with expert selection strategy, batch size, and input length.

For production-grade serving, this approach is typically only suitable for small-concurrency validation or offline batch processing, not for directly handling high-concurrency online inference.

7. Cost Estimation: From Hardware Lower Bound to Total Cost of Ownership

If “64 × 80GB accelerators” is taken as a basic inference node, the single-node hardware cost is already in the range of tens of millions of RMB. If higher VRAM specifications or stronger interconnect capabilities are considered, the cost will continue to rise.

Beyond hardware procurement costs, the following must also be accounted for:

1. Software adaptation and operator optimization;
2. Quantization verification and accuracy regression;
3. Distributed communication tuning;
4. Energy consumption under long-running operation;
5. Spare parts and maintenance;
6. Cluster scheduling and monitoring infrastructure.

For most enterprises, directly building a full-scale ultra-large MoE inference cluster is not necessarily the optimal return on investment. More prudent paths typically include:

• Using low-bit quantization to reduce weight volume;
• Adopting hybrid inference schemes such as expert offloading;
• Conducting small-scale validation first before deciding whether to scale up;
• Performing capacity planning separately for low-latency, high-throughput, and large-batch scenarios.

8. Conclusion: Do the Math Before Buying the Cards

Local deployment of ultra-large MoE models is fundamentally an engineering problem jointly constrained by storage, communication, and compute.

The key conclusions can be summarized as:

1. VRAM should be calculated based on the quantized volume of total parameters, not just activated parameters;
2. 4-bit weights represent only a theoretical lower bound; real storage requirements need to be scaled up;
3. A 2.8-trillion-parameter model requires at least dozens of accelerators, with an official recommendation of 64 or more in a supernode configuration;
4. Once card count is sufficient, communication topology determines inference performance;
5. GPU-CPU heterogeneous inference can reduce initial costs but sacrifices throughput and online service quality;
6. Before investing at the tens-of-millions level in hardware, priority should be given to completing quantization verification, communication simulation, and cost estimation.

For technical decision-makers, the biggest pitfall is deriving “deployment is cheap” from “activated parameters are small.” The truly rational approach is to put four tables together — total parameters, quantization format, cluster topology, and inference workload — before deciding whether to deploy locally, to what extent, and whether to choose a managed large-model inference service to lower the initial barrier.