A Guide to Estimating Local Deployment Costs for Trillion-Parameter MoE Models

When model parameter scales cross into the trillions, local deployment is no longer just a matter of “buying a few GPUs.” It becomes a complex system engineering challenge involving storage capacity, memory bandwidth, interconnect topology, and quantization strategies. Using a 2.8-trillion parameter MoE model with 896 experts as a sample, this article breaks down the calculation methods for VRAM usage, hardware configuration strategies across different deployment tiers, and the corresponding cost ranges, helping developers calculate the budget clearly before getting started.

1. Calculating VRAM: What Trillion Parameters Mean for Storage

1.1 Total Parameters Determine Storage, Activation Ratio Only Determines Compute

The core characteristic of the MoE architecture is “massive total parameters, sparse activation per token.” Taking a model with 2.8 trillion total parameters, 896 experts, and 16 experts activated per token as an example, the expert activation ratio is about 1.79%. However, two concepts must be distinguished: the expert activation ratio does not equal the parameter activation ratio. The routing mechanism only determines which experts participate in the computation. During deployment, the weights of all 896 experts must reside in addressable storage. During inference, the corresponding 16 experts are fetched based on the routing results.

This leads to the first golden rule of deployment: VRAM usage is calculated based on total parameters, while the activation ratio only affects compute consumption and bandwidth pressure. Any assumption of “only loading activated parameters” does not hold in standard MoE inference because routing is dynamic, and the combination of experts visited by each token may vary.

1.2 The Actual Impact of Quantization Formats on VRAM

This model introduced quantization-aware training from the supervised fine-tuning stage. Weights use a 4-bit floating-point format, and activations use an 8-bit floating-point format. Calculating the 2.8 trillion parameters directly at 4-bit, the raw data lower limit is approximately:

2.8e12 × 4 bit ÷ 8 = 1400 GB

However, the 4-bit floating-point format is not purely 4-bit packed. According to the OCP specification, every 32 4-bit elements are accompanied by an 8-bit scale factor, making the equivalent bit width:

4 + 8/32 = 4.25 bit
2.8e12 × 4.25 ÷ 8 ≈ 1487.5 GB

This 1487.5 GB is still only the theoretical value of the weight elements plus the scale factor. It does not account for non-4-bit tensors (such as embedding layers and normalization parameters), metadata, memory alignment padding, runtime workspace, and KV cache. In actual deployment, an additional 10% to 20% margin is typically reserved for overhead beyond the weights.

1.3 Establishing a Baseline Through Comparison

Take a common mid-scale MoE model in the industry as a reference: 671B main parameters plus 14B multi-token prediction parameters. In FP8 format, this is about 685 GB. Even with 4-bit weight quantization, the weights take up about 335 GB. Adding roughly 37 GB for activations, it totals about 400 GB to run on 8 80GB accelerators (640 GB VRAM).

By contrast, the volume of 2.8 trillion parameters is more than 4 times that of the former, and the lower limit of weights approaches 1.5 TB. This means a single-node 8-GPU solution is directly ruled out based on VRAM capacity. The discussion must move to multi-node or heterogeneous offloading solutions.

1.4 Interconnect Bandwidth: An Underestimated Cost Factor

Large-scale MoE inference is extremely sensitive to all-to-all communication. When 896 experts are distributed across multiple accelerators, the activations must be transferred to the cards hosting the target experts after routing, and then transferred back. Communication latency directly erodes compute utilization.

Therefore, official recommendations suggest supernode deployments consisting of 64 or more accelerators. By using high-bandwidth interconnects (such as NVLink or equivalent high-speed interconnects) to enlarge the communication domain, expert transfers do not cross a single communication domain. Behind this recommendation is a tangible cost: the cost of the supernode interconnect itself is often on the same order of magnitude as the accelerators. When planning the budget, one cannot only calculate the cost of the cards.

2. Tier 1: Just Make It Run, Regardless of Speed

2.1 Target Definition

The sole goal of this tier is to fit the nearly 1.5 TB of weights into addressable storage and let the model generate tokens normally, even if the decoding speed is only single digits per second. It is suitable for scenarios insensitive to throughput, such as algorithm validation, performance evaluation, and internal demonstrations.

2.2 GPU + CPU Heterogeneous Inference Approach

The most straightforward path is heterogeneous inference: keeping the compute-intensive, small-parameter-scale parts on the GPU while offloading sparse expert parameters to CPU memory.

Specific to this model’s structure, each layer contains a shared attention backbone (KDA plus Attention Residuals, with Gated MLA as a component). This part is compute-intensive and suitable for GPUs with high-bandwidth VRAM. Conversely, the 896 sparse experts account for the vast majority of weight storage, but only 16 are activated per token. This makes them naturally suitable for offloading to large-capacity DDR memory, supplied by multi-channel server platforms.

The resource ratio for this solution is roughly: a small number of GPUs handle attention and forward computation, while large-capacity memory accommodates the resident expert weights. The bottleneck usually occurs at the bandwidth for transferring weights from CPU to GPU, limiting the decoding speed to the single-digit tokens/s range.

2.3 Cost Estimation

For a single-node heterogeneous configuration aimed at “just making it run,” the costs primarily come from three areas:

• CPU and Motherboard: Requires a multi-channel server platform supporting large-capacity memory (1.5 TB+). Dual-socket, high-core-count processors are common choices.
• Memory: 1.5 TB+ of DDR5 ECC memory. At current market prices, this represents an expense in the tens of thousands of dollars.
• GPU: 1 to 2 high-end consumer-grade or professional-grade cards to host the attention backbone and activation computations.

The overall hardware cost falls roughly into the $100,000 range. Compared to cluster solutions that can cost millions, this is the most cost-effective entry point for the validation phase. The trade-off is slow decoding speed and weak concurrency, making it unsuitable for providing external services.

2.4 Engineering Considerations

The heterogeneous offloading solution demands high scheduling capabilities from the inference framework. Three points require attention: whether the on-demand loading strategy for expert weights introduces too many PCIe round trips; whether the placement of the KV cache competes with expert weights for memory bandwidth; and whether the resource allocation between the prefill and decode phases requires differentiated configurations. These details directly determine the gap between “can run” and “can run effectively.”

3. Tier 2: Small-Scale Serving, Targeting Usable Throughput

3.1 Target Definition

When the model needs to support internal business operations or low-traffic external services, a single-digit tokens/s decoding speed is no longer acceptable. The goal of this tier is to increase the decoding speed to between a dozen and several dozen tokens/s, along with a degree of concurrency.

3.2 Configuration Strategy

This tier still uses heterogeneous inference as its basic framework, but two areas must be significantly enhanced:

First, expand GPU-side compute. The computational load of the attention backbone grows linearly with concurrency. A single card struggles to support parallel multi-request processing. Typically, 4 to 8 accelerators are needed to form a small-scale compute pool dedicated to handling attention, KV cache, and routing computations.

Second, improve CPU-to-GPU transfer efficiency. Offloading expert weights generates continuous weight transfers during the decoding phase. Upgrading PCIe generations, optimizing the CPU-GPU direct connection topology, and caching hot experts on the GPU are common strategies. In practice, statistically analyzing the routing distribution and keeping high-frequency experts resident in VRAM can significantly improve decoding speed.

3.3 Cost Estimation

The hardware composition for this tier is roughly: one 8-GPU server equipped with 1.5 TB+ of memory and a high-frequency, multi-core CPU, plus high-speed local storage for weight loading. The cost of the complete machine is roughly in the hundreds of thousands of dollars. If multi-node expansion is required to accommodate all experts and increase concurrency, costs will rise further.

It is worth noting that the cost-effectiveness of this tier highly depends on specific business traffic. If requests are sparse and concurrency is low, the resource utilization of the heterogeneous solution will suffer. If traffic is stable, one should promptly evaluate the tipping point for migrating to a cluster solution.

4. Tier 3: Full Deployment, Supernode Clusters

4.1 Why Clusters Are Necessary

When the goal is to approach the officially recommended deployment form, single-node solutions hit the ceiling in both VRAM capacity and interconnect bandwidth. The nearly 1.5 TB of weights, plus KV cache and workspace, requires dozens of high-VRAM accelerators to carry in parallel. The all-to-all communication of 896 experts requires keeping the communication domain within the range of high-bandwidth interconnects.

The officially recommended supernode of 64 or more accelerators is designed precisely for these two constraints: the total VRAM is sufficient to hold all expert weights and a substantial KV cache, while the high-bandwidth interconnect ensures expert transfers do not cross domains.

4.2 Cost Magnitude

The cost of this tier consists of three parts:

• Accelerators: 64 or more high-VRAM accelerators form the main cost body.
• Supernode Interconnect: High-bandwidth switches and cables cost the same order of magnitude as the accelerators.
• Supporting Infrastructure: High-power racks, liquid or intense air cooling, and high-speed storage and networking.

The overall investment enters the multi-million dollar range and continues to rise with increases in concurrency targets and KV cache capacity. This tier is suitable for teams with clear expectations for service scale and is not recommended for premature investment during the validation phase.

4.3 Key Trade-offs

The core trade-off in a full deployment is not hardware, but utilization. The fixed costs of a trillion-parameter MoE cluster are extremely high. Only by maintaining a high GPU utilization rate can the cost per token be economically viable. Therefore, planning must simultaneously consider batching strategies, routing load balancing, KV cache tiering, and request scheduling. Otherwise, no matter how powerful the hardware is, the costs cannot be amortized.

5. Cost Comparison and Selection Recommendations

Tier	Target	Typical Configuration	Cost Magnitude	Applicable Scenarios
Tier 1	Just make it run	Single-node heterogeneous, 1-2 GPUs + large memory	~$100,000	Algorithm validation, performance evaluation
Tier 2	Usable throughput	8-GPU server + large memory	Hundreds of thousands of dollars	Internal business, low-traffic services
Tier 3	Full deployment	64+ GPU supernode cluster	Multi-million dollars	Scaled external services

When selecting a tier, it is recommended to follow three principles:

Validate before expanding. Use Tier 1 to validate model effectiveness and business value, then migrate upwards once it is confirmed to be worth the investment. The greatest value of the heterogeneous approach is not saving money, but minimizing the cost of trial and error.

Determine the tier based on bottlenecks. If the bottleneck is VRAM capacity, prioritize heterogeneous offloading; if the bottleneck is decoding speed, prioritize adding GPUs; if the bottleneck is concurrency and communication, only then consider a supernode.

Include interconnects in the budget. The communication overhead of trillion-parameter MoE scales non-linearly with the number of cards, and the cost of supernode interconnects is often underestimated. When budgeting, interconnects and supporting infrastructure should be reserved at the same order of magnitude as the accelerator costs.

6. Conclusion

The local deployment of trillion-parameter MoE models is essentially a three-way trade-off between storage, bandwidth, and cost. The lower limit of weights is determined jointly by total parameters and the quantization format—approximately 1.5 TB is an unavoidable starting point. The activation ratio only affects compute, not storage. Interconnect bandwidth determines the actual efficiency of multi-card solutions. By calculating these three accounts separately and allocating budgets across the three stages of validation, usability, and scaling, one can successfully navigate the complete path from “able to run” to “running well” within controllable costs.