Local Deployment of Trillion-Parameter MoE Models: A Complete Guide from Hardware Selection to Cost Estimation

1. Tiered Thinking for Deployment Goals

Before deploying a large-scale Mixture-of-Experts (MoE) model locally, the first step is to define your performance target. Different targets lead to hardware investments that can differ by more than an order of magnitude. Using an industry reference model of approximately 2.8 trillion total parameters as an example, deployment goals can be divided into three typical tiers.

Tier 1: It runs. The only requirement is that the model can produce output tokens. Speed is not a concern, and a few tokens per second is acceptable. This tier is suitable for functional validation, educational demonstrations, or non-interactive batch processing.

Tier 2: Smooth single-user experience. The model should sustain interactive output of at least 25 tokens per second. This covers everyday Q&A, code generation, and similar scenarios, and is generally considered the practical baseline for individual developers.

Tier 3: Production-grade multi-user service. The system must support at least 10 concurrent requests while maintaining 25 tokens per second or more for each user. In addition, quantization precision must not fall below FP8 or an equivalent level, to avoid noticeable model quality degradation from aggressive compression. This is the minimum requirement for enterprise applications.

The differences among these three tiers essentially reflect trade-offs across three hardware dimensions: storage capacity, compute supply, and communication bandwidth. Each is discussed in detail below.

2. Quantifying Model Size: What 2.8 Trillion Parameters Actually Means

The reference model has 2.8 trillion parameters in an MoE architecture with 896 experts, activating 16 experts per token. This gives an expert activation ratio of about 1.79%. A key point for MoE deployment is that memory usage must be calculated based on total parameters—all expert weights must be loaded into storage, with only the routed subset activated during inference. The activation ratio therefore affects compute load, not storage requirements.

The model uses quantization-aware training from the SFT stage, with weights stored in MXFP4 format and activations in MXFP8. Converting 2.8 trillion parameters to 4-bit weights gives a theoretical lower bound of roughly 1,400 GB. However, MXFP4 is not simply 4-bit storage. Under the OCP (Open Compute Project) specification, every 32 4-bit elements carry an 8-bit scale factor. Accounting for both weight elements and scale factors, the actual storage requirement is approximately 2.8T × 4.25 bit ÷ 8 ≈ 1,487.5 GB. This does not include non-4-bit tensors, metadata, alignment padding, runtime workspace, or KV cache overhead.

For context, consider a well-known 671B-parameter model as a reference point. In FP8, that model requires about 685 GB when combining main parameters and auxiliary modules. Even with AWQ 4-bit quantization, it needs roughly 335 GB of weights plus 37 GB of activations—about 400 GB total—to barely fit on eight GPUs with 80 GB of memory each. The 2.8T model is more than four times that size. At full precision deployment, weights alone require at least 1.5 TB of storage, meaning a single-node 8-GPU setup with 80 GB per card (640 GB total) cannot physically hold the full weight set.

3. Tier 1 Deployment: CPU-GPU Heterogeneous Inference

When the goal is simply “it runs,” the most straightforward approach is a heterogeneous CPU-GPU inference architecture. The core idea is that in an MoE model, each layer contains a compute-intensive shared attention component, which remains on the GPU to take advantage of high-bandwidth memory. Meanwhile, the 896 sparse experts—which account for the vast majority of storage—are activated only 16 at a time and can be offloaded entirely to CPU memory, served from a large-memory server on demand.

Several key implementation points matter here:

1. Memory configuration: Weights alone require about 1.5 TB. Server memory should be at least 2 TB, preferably configured with multi-channel DDR5 or higher-bandwidth memory to reduce the CPU-to-GPU data transfer bottleneck.

2. CPU-GPU communication: PCIe 5.0 or faster is recommended, with Resizable BAR enabled so the CPU can efficiently page expert weights into GPU memory. Each inference pass must move 16 expert weights (about 1.79% of the total) from system memory to GPU memory, so transfer latency directly determines end-to-end response time.

3. Inference engine selection: Mature open-source frameworks in the KTransformers family already implement this kind of heterogeneous inference, supporting 4-bit quantized storage of sparse expert weights on the CPU side with dynamic loading at inference time. Be aware, however, that these frameworks have specific structural adaptation requirements. The reference model’s attention mechanism may need custom kernels to be fully utilized.

At this tier, a minimal hardware configuration is roughly: one dual-socket server with at least 64 cores per socket, 2 TB of RAM, and one GPU with no less than 80 GB of memory (or two 48 GB GPUs). The overall budget can fall in the range of approximately USD 30,000 to 60,000. A lower-cost pure CPU inference approach is possible in principle, but speed may drop below one token per second, making it impractical for most purposes.

4. Tier 2 Deployment: Planning Compute and Memory for 25 TPS Single-User Performance

To deliver 25 tokens per second for a single user, more of the computational load must stay on the GPU, minimizing frequent CPU-GPU data exchange. This means significantly expanding GPU memory so that most expert weights remain resident on the accelerators.

Starting from approximately 1,500 GB of MXFP4-format weights, plus KV cache and runtime workspace (typically an additional 20-30%), the total GPU memory requirement is around 2,000 GB. Since mainstream accelerators currently offer 80 GB or more per card, at least 25 cards are needed in theory. In practice, due to inter-card communication efficiency and fault tolerance, the industry generally recommends 32 or more accelerators in a single node or supernode.

For compute matching, 25 tokens per second means completing forward passes for roughly 16 experts per token. Estimating from the model’s activated parameters (about 2.8T × 1.79% ≈ 50 billion), each token generation involves roughly one trillion floating-point operations including the attention component. At 25 TPS, the pure compute requirement is about 25 TFLOPS or more. Current mainstream accelerators deliver 100–200 TFLOPS of FP16 dense compute per card, so compute is not the bottleneck—communication is.

Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth. With 896 experts distributed across multiple cards, each token after routing must have its activations sent to the cards hosting the selected experts, then returned after computation. This requires low-latency, high-bandwidth inter-card communication. NVLink or equivalent supernode interconnect technology is the recommended approach, allowing all data movement to complete within a single communication domain and avoiding the long-tail latency of cross-node transfers.

When 32 cards are connected in a fully interconnected topology, per-card communication bandwidth needs to reach 900 GB/s or higher to sustain 20–30 TPS. If a multi-node approach using 8-card nodes is chosen instead, cross-node communication becomes the bottleneck, and measured speed may drop below 10 TPS. For Tier 2, the recommended path is to purchase a full rack-level solution supporting supernode interconnect, with budgets typically in the range of USD 200,000 to 400,000.

5. Tier 3 Deployment: Scaling for 10 Concurrent Users with Precision Constraints

Supporting 10 concurrent requests while maintaining 25 TPS per user is far more complex than the single-user case because it simultaneously tests throughput and latency control. Simply multiplying the single-user compute requirement by 10 yields about 250 TFLOPS—beyond the capacity of a single accelerator—so multi-card parallelism is mandatory. But the bigger challenge lies in shared KV cache management.

5.1 The Large Capacity Requirement of KV Cache

The reference model uses a Gated MLA (Multi-head Latent Attention) mechanism, which significantly compresses KV cache compared to traditional MHA (Multi-head Attention). Even so, with 10 concurrent users and 32K tokens of context per user, KV cache is estimated at 80–150 GB, requiring dedicated GPU memory or high-speed storage. If context length extends to 128K tokens, KV cache may exceed 500 GB, substantially affecting memory planning.

5.2 Balancing Quantization Precision and Model Quality

Tier 3 requires quantization no lower than FP8 (or an equivalent 8-bit format), meaning MXFP4-style aggressive compression is not acceptable. Under FP8, weight storage for this model doubles to about 3,000 GB, and total GPU memory rises above 4,000 GB. At current maximum per-card memory, at least 50 accelerators are needed. Although FP8 is relatively widely supported, its precision allows only about 2–3 significant decimal digits, which may amplify quantization error for sensitive parameters in MoE models. Thorough calibration-set validation is therefore essential before deployment.

5.3 Supernode Architecture and Parallelism Strategy

For 10 concurrent users, the recommended approach is a three-dimensional hybrid parallelism strategy combining tensor parallelism + expert parallelism + data parallelism. Tensor parallelism splits the weight matrices in attention layers. Expert parallelism distributes the 896 experts evenly across cards, ensuring expert routing communication for each token completes within a single communication domain. Data parallelism handles batch scheduling for the 10 concurrent requests.

Inside the supernode, a fully interconnected topology is required, with no less than 3.2 TB/s of unidirectional communication bandwidth. This exceeds typical NVLink capabilities and may require NVLink C2C or more advanced optical interconnect solutions. In practice, the supernode configuration recommended for industry-leading models serves as a useful reference: 64 accelerators per logical node, high-bandwidth low-latency interconnect between cards, and intra-node communication latency of no more than 2 microseconds.

5.4 Cost Estimation

The total cost of Tier 3 deployment includes:

• GPU hardware: 64 accelerators at 80 GB each, estimated at roughly USD 650,000–900,000 based on typical per-card pricing.
• Supernode networking: High-speed switching equipment and cabling, estimated at USD 110,000–210,000.
• Storage systems: High-speed NVMe storage pool for weight loading and checkpoints, at 20 TB usable capacity, approximately USD 28,000–55,000.
• Servers and racks: Including CPU, memory (1 TB per node recommended), cooling, and power, approximately USD 110,000–170,000.
• Software licensing and optimization: If a commercial inference framework is used, about 10–20% of total hardware cost.

Taken together, a full production-grade deployment for 10 concurrent users typically requires USD 1,000,000 to 1,400,000. If cloud GPU rental is used instead of purchased hardware, running a 64-card supernode at full load would cost roughly USD 160,000 per month based on typical per-card hourly rates, before data transfer and storage fees. Over an extended period, cloud costs may exceed the cost of building in-house.

6. Cost Reduction Strategies and Practical Recommendations

Before committing to a deployment, evaluate the cost-effectiveness of your plan from four angles:

1. Assess real needs: Do you actually need local deployment? For experimental validation, API access with per-token billing or short-term cloud GPU rental may be the better choice financially.
2. Optimize quantization strategy: If precision requirements are less strict—for example, an internal knowledge base Q&A system—a mixed quantization scheme may work. Keeping attention layers in FP8 while using MXFP4 for expert layers can achieve over 90% of full-precision quality at roughly half the cost.
3. Track progress in sparse inference frameworks: CPU-GPU heterogeneous approaches are maturing quickly. In the future, they may reach 25 TPS single-user speeds at much lower hardware thresholds.
4. Expand in stages: Start with an 8–16 card setup to validate the full pipeline, confirm accuracy and stability, then scale to 32 or 64 cards as needed.

7. Conclusion

Locally deploying a trillion-parameter MoE model is ultimately a dual exercise in mathematics and engineering. You must work through the physical formulas for memory and bandwidth while also navigating the “impossible triangle” of precision, speed, and cost. For a 2.8-trillion-parameter reference model, the minimum viable deployment starts at around USD 30,000, while production-grade service for 10 concurrent users approaches USD 1.4 million. Before making a final decision, validate your hardware and software stack against your actual business scenario through benchmarking. Avoid blindly pursuing a “fully uncompromised” setup that could result in significant sunk costs.