Deploying Large MoE Models Locally: Breaking Down Costs from VRAM and Communication to Hardware Selection

1. Why “Fewer Activated Parameters” Does Not Mean “Less VRAM Usage”

Many large sparse models emphasize during release that “only a small number of experts are activated per token,” creating the impression that inference costs are very low. For example, a model may have trillions of total parameters, but only a small subset of experts is activated per token. Here, two fundamentally different concepts must be distinguished:

1. Compute consumption: primarily related to the number of parameters actually involved in computation per token.
2. VRAM usage: primarily related to total model parameters, quantization precision, model sharding strategy, and runtime caches.

In a typical MoE architecture, all experts are stored in VRAM or high-speed storage. During inference, the routing module selects a small number of experts based on token features for computation, but this does not mean unselected experts can be removed from VRAM. The reason is simple: different tokens, different batches, and different requests will hit different experts. If expert weights had to be reloaded from disk or remote storage on every request, latency would become unacceptable.

Therefore, for local deployment, baseline VRAM requirements must be estimated based on total parameter count, not activated parameter count. The activation ratio can significantly affect compute, throughput, and latency, but it cannot offset the cost of heavyweight weights residing permanently in VRAM.

2. Total Parameters and Activation Structure of a Trillion-Parameter MoE Model

Assume a MoE model has 2.8 trillion total parameters, 896 experts, and activates 16 experts per token. The expert activation ratio is approximately:

16 / 896 ≈ 1.79%

This ratio only represents the proportion of experts activated per token on the expert network; it does not directly equal the total parameter activation ratio. Beyond the expert network, the model typically also contains:

• Attention layer parameters;
• Routing network parameters;
• Embedding layer parameters;
• Normalization layer parameters;
• Output layer parameters;
• Possible auxiliary modules or multi-task heads.

These components may be activated for every token or participate in computation in different ways. Therefore, even if the expert activation ratio is very low, the total number of parameters activated during an actual forward pass may still be higher than intuition suggests.

Additionally, some MoE architectures include shared layers, shared attention layers, or shared expert layers. Shared experts may be accessed by more tokens, further increasing compute and bandwidth pressure. Evaluation must not only look at the “number of regular experts activated” but also consider whether always-on computation modules exist.

3. How Quantization Formats Affect the VRAM Floor

For very large models, deployment at original precision is usually infeasible. Taking FP16 or BF16 as an example, 2.8 trillion parameters require:

2.8T × 2 bytes = 5.6TB

This already exceeds the available VRAM of most single machines or even small clusters. Therefore, practical deployments almost always adopt low-precision quantization.

Common quantization approaches include:

• INT8 / FP8: weights and activations use 8-bit representation;
• INT4 / AWQ / GPTQ: weights are 4-bit, activations can use FP16 or INT8 as needed;
• MXFP4 / MXFP8: micro-scaling-based floating-point formats where weights use 4-bit elements with shared scales;
• Quantization-aware training: quantization error is introduced during training to reduce precision loss during inference.

Taking MXFP4 as an example, it does not simply store all parameters in 4-bit. According to common specifications, consecutive 4-bit elements may share an 8-bit scale. This introduces additional metadata overhead.

Assuming weight elements are 4-bit with one 8-bit scale per 32 elements, the average effective bits per element can be estimated as:

4 + 8 / 32 = 4.25 bit

For 2.8 trillion parameters, the raw weight storage alone is approximately:

2.8T × 4.25 bit ÷ 8 ≈ 1487.5GB

This figure does not yet include:

• Additional overhead from unquantized embedding or output layers;
• Non-4-bit tensors such as normalization parameters and routing parameters;
• Duplicated parameters or metadata from sharding;
• KV cache;
• Communication buffers;
• Runtime workspace for deep learning frameworks;
• VRAM alignment and fragmentation losses.

Therefore, actual VRAM required for deployment is typically significantly higher than raw weight volume. It is generally recommended to reserve at least 20%–50% extra space on top of raw weights for caching, communication, and runtime overhead.

4. Understanding the Scale of VRAM Through Comparison

To more intuitively understand the deployment threshold, a trillion-parameter model can be compared with a common several-hundred-billion-parameter dense or MoE model.

For example, a MoE model with approximately 671B total parameters, if using FP8 format, requires roughly:

671B × 1 byte = 671GB

If there is an additional multi-token prediction module, such as approximately 14B parameters, that part requires:

14B × 1 byte = 14GB

Total: approximately 685GB.

If 4-bit weight quantization is used, the main weight portion may drop to approximately 335GB, but activations and additional buffers still add tens of GB. Overall, approximately 400GB of VRAM may be needed to run on a node with 8 × 80GB accelerators, with total available VRAM of about 640GB.

A MoE model with 2.8 trillion total parameters is more than 4× the scale of the above model. Even with more aggressive 4-bit quantization, weight volume already falls in the 1.4TB to 1.5TB range. This means a single 8-GPU node, whether with 80GB or 96GB per GPU, basically cannot host the full model.

5. The Hidden Bottleneck Beyond VRAM: Communication Topology

Another critical bottleneck for large-scale MoE inference is communication. The reason is that experts are distributed across different devices, and each token’s forward pass may involve cross-GPU data transfers.

The typical process is as follows:

1. The routing module computes expert scores based on token representations;
2. Top-k experts are selected;
3. The token’s hidden state is sent to the device hosting the corresponding expert;
4. Expert computation is completed on the target device;
5. Results are sent back to the token’s original device;
6. Multiple expert outputs are merged and processing continues to subsequent layers.

When the number of experts reaches hundreds or nearly a thousand and they are distributed across dozens of GPUs, each step may generate all-to-all communication. At this point, communication latency may exceed expert computation itself, becoming the primary constraint on system throughput.

Key factors affecting communication efficiency include:

• Intra-node GPU interconnect bandwidth;
• Inter-node network bandwidth, such as RDMA bandwidth;
• Switch and topology structure;
• Whether all-to-all communication can be merged or sharded;
• Whether a combination of expert parallelism, tensor parallelism, and pipeline parallelism is used;
• Whether high-speed NVMe or dedicated caches are used for weight swapping.

Therefore, the official recommendation to use 64 or more accelerators in a supernode for deploying such models is not simply because VRAM is insufficient, but to reduce weight size per GPU, increase aggregate VRAM, and provide sufficiently high backplane bandwidth for communication.

6. Hardware Solution Boundaries for Local Deployment

If the deployment target is limited to a “local environment,” there are typically several options:

1. Single Machine, Single GPU or Single Machine, Multiple GPUs

Suitable for small to medium models or highly quantized models. For trillion-parameter MoE, even with 4-bit quantization, single-machine VRAM cannot meet the requirement for permanent weight residency. In this case, one may consider:

• Deploying only a subset of experts;
• Using layered offloading;
• Accepting very low concurrent throughput;
• Running demo-level inference only.

This approach is suitable for validating model formats, inference logic, or developing upper-layer applications, but not for production services.

2. Small-Scale Cluster of Multiple High-VRAM Nodes

If a single node has 8 GPUs with 80GB each, the theoretical VRAM per node is approximately 640GB. For approximately 1.5TB of 4-bit weights, at least 3 nodes are needed just to hold the raw weights. Considering KV cache, communication buffers, and system overhead, 4 or more nodes may be required in practice.

However, this is only a VRAM-level estimate. On the communication side, if inter-node network bandwidth is insufficient—for example, only low-bandwidth Ethernet without RDMA or other high-speed interconnect solutions—latency and throughput may degrade severely even if VRAM is sufficient.

3. Supernode Solution

Using a supernode composed of 64 or more accelerators, the weight burden per GPU is greatly reduced. For example, 1.5TB of weights distributed across 64 GPUs averages approximately:

1.5TB / 64 ≈ 23.4GB

After accounting for additional space, each GPU can still remain under 40GB. This not only provides more comfortable VRAM headroom but also significantly reduces all-to-all latency through high-speed backplane interconnect.

The key points of this solution are:

• Whether the network architecture supports large-scale lossless transmission;
• Whether there are sufficient switch ports;
• Whether expert parallelism is supported;
• Whether the inference framework can effectively schedule routing for 896 experts;
• Whether monitoring, recovery, and load balancing mechanisms are complete.

7. How to Estimate Actual Costs

Local deployment costs typically consist of the following parts:

1. Accelerator Card Costs

This is the main expenditure. If the per-card cost is high, 64 cards constitute a significant investment. Different procurement channels, VRAM specifications, and supply cycles all significantly affect the total price. Enterprises typically also need to consider:

• Spare cards;
• Test cards;
• Replacement solutions during repair cycles.

2. Node and Network Equipment Costs

Including:

• High-density servers;
• High-speed switches;
• Optical modules and cables;
• Racks and power supply;
• Cooling facilities.

For clusters of 64+ GPUs, network equipment costs may approach or even exceed a portion of accelerator card costs.

3. Software and Integration Costs

Including:

• Inference framework tuning;
• Expert parallelism strategy design;
• Quantization toolchain adaptation;
• Routing performance debugging;
• Monitoring and alerting system construction;
• Integration with business systems.

Even when using open-source inference frameworks, considerable manpower is needed for adaptation and stress testing.

4. Operations and Maintenance Costs

Including:

• Data center rack fees;
• Electricity costs;
• Network bandwidth;
• On-call and fault response;
• Regular stress testing and capacity assessment.

Therefore, simply asking “how much does it cost to deploy a trillion-parameter model” cannot yield a fixed number. Hardware procurement is only the most visible part; what truly affects total cost of ownership is the communication solution, inference framework optimization capability, and the team’s model engineering experience.

8. From Deployment Assessment to Engineering Implementation

If a team wishes to complete deployment of a similar-scale model locally, it is recommended to evaluate according to the following steps:

Step 1: Determine the Deployment Form

Clarify whether it is offline evaluation, internal trial, or online inference service for business. Different forms determine acceptable latency, concurrency, availability requirements, and acceptable hardware costs.

Step 2: Calculate the VRAM Floor Based on Total Parameters

Do not look only at activated parameters. It is recommended to prepare:

• FP8 estimates;
• INT4 estimates;
• MXFP4 estimates;
• Reserve space for KV cache and runtime overhead.

Step 3: Determine the Parallelism Strategy

Mainly including:

• Expert parallelism;
• Tensor parallelism;
• Pipeline parallelism;
• Data parallelism.

For very large MoE, expert parallelism is usually an unavoidable choice, but it must be combined with tensor parallelism and pipeline parallelism to control per-GPU VRAM.

Step 4: Measure Communication Bottlenecks

Before purchasing a complete cluster, tests can be conducted at a smaller scale:

• Single-GPU weight loading;
• Dual-GPU expert parallelism;
• Intra-node 8-GPU all-to-all;
• Small-scale cross-node communication.

Small-scale stress testing can more accurately determine whether the interconnect solution is suitable, avoiding the situation where the network cannot support the workload after a one-time purchase.

Step 5: Build a Cost Model

Incorporate hardware, software, manpower, and operations into a unified model, estimated annually. Do not look only at one-time purchase price; also consider:

• Utilization rate;
• Failure rate;
• Whether elastic scaling is possible;
• Whether multi-project sharing is possible.

9. Conclusion

The local deployment cost of very large MoE models is fundamentally determined by total parameter count, not by activated parameters. Even with 4-bit quantization, the raw weight storage of a trillion-parameter model may reach 1.4TB or more. After adding KV cache, communication buffers, and runtime workspace, actual VRAM requirements will be even higher.

Serious deployment of such models typically requires dozens or more accelerators forming a high-speed interconnect cluster, and the all-to-all communication problem under expert parallelism must be addressed as a priority. Otherwise, even if total VRAM barely meets the requirement, low-bandwidth networks will rapidly degrade inference throughput and latency.

For enterprises and developers, a more pragmatic approach is not to pursue “full local deployment” directly, but to first build an evaluation model based on total parameters, quantization scheme, communication topology, and business requirements, then gradually validate the inference framework, parallelism strategy, and network capability through small-scale testing, and finally decide whether to invest in a complete hardware cluster. This controls risk and gives every infrastructure investment a clearer basis.