Cost and Hardware Selection for Local Deployment of Large-Scale MoE Models

1. Deployment Goal Tiering: From Experimental Verification to Production-Grade Services

When planning the local deployment of large-scale Mixture-of-Experts (MoE) models, the primary task is not hardware selection but clarifying the business’s actual requirements for latency, throughput, and quantization precision. Deployment costs can vary by an order of magnitude across different scenarios, necessitating the division of requirements into three progressive tiers:

Tier 1: Functional Verification Deployment. This tier only requires the model to be successfully loaded and perform inference without specific output speed metrics. It is typically used in development, debugging, or small-scale experimental scenarios. In this context, lower quantization precision (such as 4-bit or lower) is acceptable, along with single-token generation latencies exceeding 20 seconds. Its core value lies in verifying the compatibility of model weights with the code framework, rather than providing a usable service.

Tier 2: Single-User Interactive Deployment. This tier requires a stable single-user output speed of 25 tokens per second (tps), which is the perceived threshold for a smooth reading experience. To achieve this target, the VRAM bandwidth during inference must be sufficient to support streaming weight loading, while requiring adequate reserved space for the KV cache.

Tier 3: Production-Grade Concurrent Deployment. Building upon the single-user 25 tps requirement, this tier mandates at least 10 concurrent inference requests, with a quantization level no lower than FP8 or an equivalent precision 4-bit quantization scheme. This tier imposes structural demands on total compute power, communication bandwidth, and multi-GPU collaboration efficiency. Simply stacking GPUs often fails to solve the problem and may instead lead to sub-linear acceleration ratios due to communication bottlenecks.

It is worth noting that the model’s officially announced Quantization-Aware Training (QAT) characteristics dictate that low-bit quantization strategies are employed during the Supervised Fine-Tuning (SFT) stage. This means quantized weights can be used directly upon deployment, eliminating the need for post-training secondary quantization calibration required by traditional methods. While this significantly reduces preprocessing costs on the deployment side, it imposes additional requirements on operator support within the inference framework.

2. VRAM Calculation: Physical Constraints of a 2.8-Trillion-Parameter Model

Taking a MoE model with a total parameter count of 2.8 trillion as an example, its architecture includes 896 expert modules, with 16 experts activated per token during forward propagation, resulting in a theoretical activation ratio of approximately 1.79%. A common misconception must be clarified: the expert activation ratio affects the computational volume (FLOPs), not the VRAM footprint. Whether activated or not, the weights of all 896 experts must reside in VRAM or high-bandwidth memory to ensure the routing module can dispatch them instantly during inference.

Regarding quantized storage, the model uses the MXFP4 format (4-bit floating-point) for weights, supplemented by MXFP8 format for activations. According to the OCP (Open Compute Project) specification, a data block in MXFP4 format requires an 8-bit scale factor for every 32 4-bit elements. Therefore, the theoretical storage lower limit for the 2.8-trillion parameters is calculated as follows:

• Pure weight data volume: 2.8T × 4 bit ÷ 8 = 1400GB
• Scale factor overhead: 2.8T × (4 + 8/32) bit ÷ 8 = 2.8T × 4.25 bit ÷ 8 ≈ 1487.5GB

The 1487.5GB mentioned above is merely the raw data size for weights and scale factors, excluding the following key overheads:

1. Non-4-bit tensors (e.g., embedding layers, routing networks, and LayerNorm parameters are typically stored in higher precision).
2. Padding space required for model metadata and weight alignment.
3. Temporary working buffers for the inference framework runtime.
4. KV cache, whose size is determined by sequence length, batch size, and the number of attention heads.

Comparing this with a common industry baseline: a 671B parameter open-source MoE model in FP8 format requires about 685GB of VRAM for main parameters and additional module weights. If 4-bit quantization is used, the weight size drops to roughly 335GB, but after adding approximately 37GB of activations, a total of 400GB is still required to run on 8 accelerators with 80GB of VRAM each. The 2.8-trillion-parameter model discussed in this article is over four times the size of this baseline. Its VRAM footprint lower limit (approximately 1.5TB) physically dictates that a single machine with eight GPUs (80GB each, totaling 640GB) is far from sufficient.

3. Hardware Architecture Planning: Super-Nodes vs. Communication Bottlenecks

The core bottleneck in large-scale MoE inference is not single-GPU compute power, but cross-GPU communication bandwidth. Given that 896 experts are distributed across multiple accelerators, after a token completes routing, its activations must be transmitted via a high-speed interconnect network to the target expert’s compute card. Once computed, the results must return along the same path. This process is known as All-to-All communication, and its overhead expands drastically as the dispersion of expert distribution increases.

To address this, the official recommendation is to adopt a super-node architecture consisting of 64 or more accelerators. The core value of a super-node lies in utilizing NVLink or equivalent high-speed interconnect technologies to build a non-blocking or near-non-blocking communication domain. This allows the point-to-point bandwidth between any two cards to approach local memory bandwidth. In contrast, cross-node communication based on traditional Ethernet or InfiniBand, even with RDMA technology, exhibits an order-of-magnitude gap in latency and bandwidth.

Within the super-node, the following engineering details must be considered during actual deployment:

• Total VRAM Configuration: If 64 cards each have 80GB, the total is 5120GB. After deducting system reservations and framework overhead, the actual available VRAM is about 4800GB. This capacity is sufficient to support roughly 1.5TB of weights and KV cache while retaining a safety margin of about 3.3x. This margin is not waste; it accommodates larger concurrent batches, longer context windows, and runtime memory fragmentation.
• Compute and Bandwidth Matching: Requiring a single-user output speed of 25 tps means generating at least 25 tokens per second. Each token triggers the forward computation of 16 experts, and expert computation typically involves matrix multiplications with small batch sizes. At this point, the computational intensity (floating-point operations per parameter read) is low, limiting compute utilization and making VRAM bandwidth the primary limiting factor. 4-bit quantization effectively reduces weight transmission volume; thus, the MXFP4 strategy is crucial for improving bandwidth efficiency without sacrificing too much precision.
• KV Cache Reservation: Estimating 10 concurrent requests with a 32K token context length per user, the KV cache footprint cannot be ignored. It can be significantly compressed using optimizations like GQA (Grouped-Query Attention), but even so, the total KV cache for 10 concurrent requests can still reach hundreds of gigabytes, requiring adequate space reservation in VRAM planning.

4. Cost Estimation Model and Configuration Recommendations

Based on the analysis above, a hardware cost estimation framework tailored for different requirement tiers can be constructed. The figures below use public pricing ranges of mainstream accelerators in the current market as a reference, with costs normalized to USD for an international audience:

Deployment Tier	Recommended Configuration	Total VRAM	Estimated Hardware Cost	Expected Capability
Tier 1: Functional Verification	4 GPUs (80GB/GPU)	320GB	$60K – $90K	Loadable, speed under 2 tps
Tier 2: Single-user 25 tps	16 GPUs (80GB/GPU)	1280GB	$240K – $360K	Single-user at 25 tps, no concurrency redundancy
Tier 3: 10 concurrency + 25 tps	64 GPUs (80GB/GPU)	5120GB	$960K – $1.44M	10 concurrent requests met, long context supported

The above costs cover only the purchase of accelerator cards. The following fixed overheads must also be factored in:

• Server and Rack Infrastructure: Super-node architectures typically require customized chassis and liquid cooling solutions, with single-node costs ranging from $7.5K to $15K.
• Storage and Networking: A high-throughput parallel file system can load 1.5TB of weights in a few minutes; using standard SSDs might extend startup time beyond 30 minutes. 10GbE or higher switches are mandatory.
• Power and Data Center: The full-load power consumption of 64 GPUs is approximately 40–60kW. Based on commercial data center electricity rates, annual power costs can range from $40K to $60K.
• Inference Framework Licensing and Maintenance: Developing in-house or adopting open-source solutions requires at least two full-time engineers for maintenance, equating to an annual labor cost of roughly $150K to $250K.

Key Reminder: Achieving the single-user 25 tps target under the Tier 3 configuration with 10 concurrent requests is not guaranteed. If a user’s context length exceeds 64K tokens, the KV cache will consume a massive amount of VRAM, squeezing the weight buffer space, forcing a reduction in batch size, and ultimately degrading single-user throughput. In such cases, scaling the GPU count to 80–96 may be necessary, increasing the total cost by approximately 20%–30%.

5. Deployment Strategy Recommendations and Risk Control

For the vast majority of organizations, building an on-premise super-node is not the most economically rational choice. The following strategies are worth considering:

1. Phased Verification: After weight release, first verify the actual benefits of QAT at a 16-GPU scale. During actual inference, MXFP4 weights exert only about 50% of the VRAM bandwidth pressure of FP8, but operator compatibility may cause performance degradation. It is recommended to use measured tokens/s data as the basis for scaling decisions.
2. Consider Hybrid Precision Dynamic Switching: For non-critical inference paths, weights can be temporarily switched to 4-bit to free up VRAM; for high-precision requirements, dynamically load the FP8 version. This approach requires the inference framework to support hot-switching and will increase engineering complexity.
3. Cloud Elastic Resources as an Alternative: If the budget is under $150K, or if concurrency demands fluctuate significantly, leveraging bare-metal instances or containerized clusters from cloud providers to rent 64 GPUs or more on demand can circumvent massive capital expenditures while utilizing cloud-optimized inter-node super-node connectivity. Some cloud providers already offer multi-GPU inference solutions based on high-speed RDMA, where the unit cost is significantly lower than on-premise builds for short-term rental scenarios.

Ultimately, the success of a local deployment is determined not by sheer parameter scale, but by the holistic design of communication topology, memory strategy, and quantization precision. It is recommended to conduct three sets of benchmark tests using a standard prompt set immediately after weight release: per-token latency, batch concurrency scalability, and KV cache stress testing under long contexts. Only when the measured data aligns well with the estimation model above is it worth proceeding with the formal procurement process.