Deploying a Trillion-Parameter MoE Model Locally: Hardware Costs and Architecture Options

Defining Deployment Goals by Tier

Before calculating hardware costs, you must define what “deployment” means for your use case. For the same model, performance requirements can change hardware costs by an order of magnitude. Requirements generally fall into three tiers:

Tier 1: Run the model at any speed. The goal is simply to load the full model weights into addressable memory. Generation at a few tokens per second is acceptable. This tier suits functionality checks, weight integrity verification, offline batch processing, or research experiments.

Tier 2: Single-user interaction at ~25 tokens/s. This approaches the threshold for comfortable daily conversation. Human reading speed typically ranges from 5 to 15 tokens per second, so 25 tokens/s means the model generates faster than you can read. This tier requires balanced GPU compute and memory bandwidth.

Tier 3: Support ~10 concurrent users, each at ≥25 tokens/s. This is the entry point for production-grade API services. Under concurrency, the system needs enough compute for batched inference and sufficient communication bandwidth to handle frequent expert routing across many simultaneous requests. Quantization should generally stay at FP8 or equivalent precision to balance latency and quality.

These three tiers demand completely different hardware architectures and budgets, each covered in turn below.

Resource Characteristics of Trillion-Parameter MoE Models

Current mainstream large-scale MoE (Mixture of Experts) models share several architectural traits that dictate deployment costs.

An MoE model consists of multiple expert subnetworks, activating only a fraction during each inference. For a model with roughly 2.8 trillion total parameters, 896 experts, and 16 experts activated per token, the activation ratio is about 1.79%. A common misconception is that this ratio reduces memory usage—it does not. All expert weights must still be loaded into addressable storage because the router dynamically selects different expert combinations per input; there is no way to pre-determine which experts will remain unused.

From a deployment perspective, this means storage capacity is driven by total parameter count, while compute demand is driven by activated parameters. These two figures differ by an order of magnitude—a defining characteristic of MoE deployment. You are effectively purchasing far more storage than compute.

Additionally, such models typically incorporate quantization-aware training in the later stages, storing weights in low-bit formats. At roughly 4 bits per parameter, raw weights for 2.8 trillion parameters amount to about 1,400 GB. In practice, accounting for quantization scale factors, non-quantized tensors, runtime workspace, KV cache, and memory alignment, actual storage needs typically increase by 10% to 30%.

Estimating Storage and VRAM Requirements

To make estimates concrete, consider a known reference model: a popular open-source MoE with ~671B total parameters. In FP8, its weights occupy ~685 GB. With 4-bit quantization, weights drop to ~335 GB; adding ~37 GB of activations brings the total to roughly 400 GB. This fits comfortably into eight 80 GB GPUs (640 GB total VRAM).

Scaling from this reference, a 2.8-trillion-parameter MoE model is over four times larger. Even with the same 4-bit quantization strategy, weights alone require approximately 1,400–1,500 GB of effective storage. When activation values and KV cache must also reside in high-speed memory, the total requirement rises further.

A key conclusion emerges: given that mainstream GPUs currently offer no more than 80–100 GB of VRAM per card, a pure-GPU solution for 1,500 GB-class weights requires at least 16–20 high-memory accelerators. With consumer GPUs (24 GB per card), you would need 60+ cards, and multi-card interconnect complexity escalates sharply. This is why official recommendations call for a supernode composed of dozens of accelerators.

Communication Bandwidth: The Hidden Bottleneck

For large-scale MoE inference, the performance bottleneck is often not compute but communication. With 896 experts distributed across devices, every token goes through a route–dispatch–compute–collect cycle: the router selects 16 active experts, sends the token’s activation from the current device to the devices hosting those experts, computes, then returns the results.

This creates an all-to-all communication pattern—any device may need to exchange data with any other. If devices are interconnected via PCIe or standard Ethernet, communication latency directly adds to each inference step. Even with strong per-GPU compute, overall throughput degrades rapidly.

The solution is to keep all expert scheduling within a single high-speed interconnect domain. Supernode architectures use high-bandwidth, low-latency interconnects (such as NVLink or equivalents) to place all accelerators in the same communication domain, so expert-to-expert data movement never crosses a slow link. This is why official guidance emphasizes supernodes over simply stacking multiple servers.

For budget-constrained local deployments, an alternative is optimized expert placement. By analyzing expert activation frequencies during inference, frequently accessed experts can be co-located on devices with the highest interconnect bandwidth, while infrequent experts reside on peripheral devices. This significantly reduces communication pressure—a common practical compromise in the community.

Heterogeneous Inference: A Cost-Effective Path

If your goal is simply to get the model running—not maximizing throughput—CPU+GPU heterogeneous inference is currently one of the most economical technical routes.

The core idea: keep compute-intensive components on the GPU, offload storage-intensive components to CPU memory. In MoE architectures, the attention part (or its variants) accounts for a tiny fraction of parameters but is compute-dense, making it GPU-appropriate. The large number of sparse experts occupies the bulk of storage but is accessed infrequently—only a few activate per token—so these can reside in large-capacity CPU memory and be loaded on demand.

Reference data from community practice: on a dual-socket server CPU paired with a single 24 GB consumer GPU, running a 671B-parameter 4-bit quantized MoE model achieves over 200 tokens/s during the prefill phase and approximately 14 tokens/s during decode. This falls short of the 25 tokens/s fluency threshold but suffices for functionality validation and offline tasks.

For a 2.8-trillion-parameter model, this route requires roughly 1.5 TB or more of system memory, corresponding to a dual-socket server with multi-channel, high-capacity RAM. Total system cost depends mainly on memory size and GPU model, typically far lower than a pure-GPU setup. Note, however, that decode speed may drop to single digits in tokens per second—suitable for batch processing rather than interactive dialogue.

Cost Framework for the Three Deployment Tiers

Based on the analysis above, here are approximate hardware frameworks for each tier. These figures serve as order-of-magnitude references, not precise quotes, pending the release of actual weights and official benchmark results.

Tier 1 (minimal functionality): Heterogeneous inference with a dual-socket server CPU, 1.5–2 TB of system memory, and one mid-to-high-end consumer GPU or entry-level datacenter GPU. Total system cost can be controlled within tens of thousands of RMB. The advantage is a low entry barrier; the downside is slow generation unsuitable for interactive use.

Tier 2 (single user at 25 tokens/s): More active weights must reside in high-bandwidth VRAM, while communication must not become the bottleneck. Expect at least eight high-memory datacenter accelerators interconnected in a single communication domain via high-speed links. Total hardware costs may land in the hundreds of thousands of RMB, depending on accelerator model, interconnect solution, and quantization precision.

Tier 3 (10 concurrent users × 25 tokens/s): Production-grade deployment requires multiplying compute and communication bandwidth. A conservative estimate calls for dozens of datacenter accelerators forming a multi-node supernode cluster. Hardware investment reaches millions of RMB, plus ongoing costs for facility power, cooling, and operations. At this tier, cloud-based elastic deployment is often considered to avoid a large one-time capital expenditure.

Conclusions and Recommendations

While weight files and official benchmarks are not yet fully public, the following principles should guide deployment planning for trillion-parameter MoE models:

Define your target tier first, then derive the hardware architecture. Do not request price quotes without a clear definition of “usable.”

Treat communication topology as a primary design factor. For large-scale MoE models, inter-GPU interconnect bandwidth and topology can impact performance more than per-GPU compute speed.

Do not mistake activation ratio for memory savings. MoE storage requirements are determined by total parameters—this is the most commonly underestimated cost driver.

If budget is tight, heterogeneous inference is a pragmatic choice for short-term validation. If the goal is a production service, wait for complete weights and community benchmark data before making procurement decisions to avoid locking in hardware based on estimates.