Local Deployment of a Trillion-Parameter MoE Model: Architecture, Hardware Requirements, and Cost Estimates
This article presents a practical framework for estimating the hardware and deployment costs of a trillion-parameter Mixture-of-Experts model before its complete weights are publicly available. It explains why MoE inference must be planned around total storage, expert-routing communication, KV cache capacity, and system-level bandwidth—not just active parameters. Three deployment tiers are compared, ranging from CPU–GPU heterogeneous inference to 64+ accelerator supernodes, with estimated budgets and performance expectations.
1. Why This Is an Estimation Problem
Before calculating anything, one reality must be made clear: when a target model has only publicly disclosed architecture details and API pricing, while its complete weights are not yet available, all local deployment costs are necessarily estimates based on public information.
Once the weights are released, actual quantization error, KV cache behavior, and operator implementation quality may cause real-world results to differ from these estimates. The value of this analysis is therefore not to provide a definitive answer, but to offer a reusable estimation framework. After the weights become available, each assumption can be calibrated through benchmarking.
Assume the target model has the following publicly disclosed characteristics, described here using general architecture terminology:
- Approximately 2.8 trillion total parameters, using a Mixture-of-Experts (MoE) architecture;
- 896 experts, with 16 experts activated per token, producing an expert activation ratio of approximately 1.79%;
- An attention backbone based on a linear-attention variant with gated KV compression and an attention-residual design;
- Quantization-aware training (QAT) applied from the supervised fine-tuning stage onward, with MXFP4 weights and MXFP8 activations.
Two points must be clarified in advance. Otherwise, the entire calculation can go wrong.
First, the expert activation ratio is not the same as the parameter activation ratio. The 1.79% figure describes routing sparsity. Each layer still contains shared attention components and other non-expert parameters that participate fully in computation. Actual FLOPs cannot be calculated simply by multiplying the total parameter count by 1.79%.
Second, MoE memory requirements are determined by total parameters. All 896 experts must be placed in addressable storage. During inference, the system selectively invokes 16 experts based on routing decisions. Sparse activation reduces compute requirements, but it does not reduce storage requirements. This is the fundamental reason why local deployment of large MoE models remains expensive.
2. Minimum Memory Estimate: 1,500 GB Is a Hard Lower Bound
2.1 The Raw Weight Storage Floor
For 2.8 trillion parameters stored at 4 bits per parameter:
2.8T × 4 bit ÷ 8 ≈ 1,400 GB
However, MXFP4 is not simply pure 4-bit storage. Under the OCP microscaling format, every 32 4-bit elements share an additional 8-bit scale value. The practical storage requirement is therefore approximately:
2.8T × 4.25 bit ÷ 8 ≈ 1,487.5 GB
This is only the theoretical lower bound for the weights. A complete deployment must also account for:
- Non-4-bit tensors: Sensitive layers, such as attention modules, embeddings, and the output head, are commonly retained at higher precision;
- Metadata and alignment overhead: Block storage, routing tables, memory alignment, and fragmentation;
- Runtime workspaces: Activation buffers and all-to-all communication buffers;
- KV cache: Dependent on context length and concurrency, potentially reaching several hundred gigabytes in long-context scenarios.
Taking these factors together, a full-precision-capable deployment is likely to require approximately 1,500–1,700 GB for weight storage alone, before adding the KV cache.
2.2 A Reference Point: Comparison with Mainstream Hundred-Billion-Parameter Models
The industry already has practical deployment experience with hybrid dense-and-sparse models in the 671B-parameter range. At FP8, their weights require roughly 685 GB and typically need eight 80 GB-class accelerators, totaling 640 GB, combined with CPU offload. Alternatively, they can be compressed to 4-bit, requiring approximately 335 GB for weights and around 400 GB after accounting for activations.
The model discussed here is more than four times larger. This means that an eight-accelerator single-node configuration is no longer viable for full deployment. Planning must move from a single server to a cluster-level architecture.
3. Why a “Supernode” Is Recommended: Communication Is the Hidden Bottleneck
The recommended deployment approach is a supernode composed of at least 64 accelerators. The reason is the central challenge of large-scale MoE inference: all-to-all communication bandwidth.
When 896 experts are distributed across multiple accelerators, the processing flow for each token is roughly as follows:
- The token passes through the shared attention computation;
- The router selects 16 target experts, which may be located on different accelerators;
- The activation is transferred to the accelerators hosting those experts;
- After expert computation, the results are transferred back for weighted aggregation.
A forward pass may repeat this transfer–compute–return cycle across dozens of layers. If interconnect bandwidth is insufficient, accelerators spend most of their time waiting for data rather than performing computation. Communication latency can consume a large portion of the available compute capacity. This is why teams can purchase expensive accelerators yet achieve only 20–30% effective utilization.
A supernode addresses this problem by using high-speed, NVLink-class interconnects to combine dozens or hundreds of accelerators into a single communication domain. Expert-to-expert data exchange can then remain within that domain, reducing cross-server communication bottlenecks at the physical layer.
If a supernode is beyond the budget, the alternatives are lower throughput, higher latency, or the more modest heterogeneous architecture described in the first deployment option below.
4. Three Deployment Options and Cost Estimates
The following three tiers cover common requirements, from basic feasibility testing to production workloads with concurrent users. All prices are approximate estimates based on public market pricing and are intended for budget planning only. The amounts below are expressed in RMB.
Option 1: Just Make It Run—CPU–GPU Heterogeneous Inference
Estimated budget: RMB 150,000–300,000
Approach: Each MoE layer contains a compute-intensive shared component, primarily the attention backbone, which remains on the GPU to benefit from high compute throughput and memory bandwidth. The 896 sparse experts, which account for most of the storage requirement, are placed in CPU memory. Only the 16 experts selected by the router are loaded for each token.
This “GPU for attention, CPU memory for experts” heterogeneous inference pattern already has open-source implementations and can run trillion-parameter MoE models on conventional multi-socket servers.
Hardware configuration:
- A dual-socket server with 2 TB of DDR5 memory, costing approximately RMB 100,000–150,000;
- One or two consumer or professional GPUs with 24–48 GB of VRAM, costing approximately RMB 30,000–80,000;
- Expert weights kept resident in system memory and transferred over PCIe as needed.
Expected performance: Single-digit tokens per second. This configuration is suitable only for functional validation, offline batch processing, or experimentation.
The primary bottleneck is PCIe bandwidth. Each token may require random access to several dozen gigabytes of expert weights. Quantization and caching can reduce the actual data transferred, but the result remains far slower than GPU memory access.
Option 2: 25 Tokens per Second for a Single User—Multi-GPU Inference
Estimated budget: RMB 2–4 million
A generation speed of 25 tokens per second is roughly comparable to human reading speed and can be considered the lower bound for a comfortable interactive experience. To reach this level, expert weights must be accessed from accelerator memory or through high-speed accelerator interconnects. CPU offload is generally insufficient.
Estimation logic:
- Approximately 1,500 GB of weights plus reserved KV cache capacity requires a total memory pool of around 1,800–2,000 GB;
- With 80 GB-class accelerators, at least 24 accelerators are required, equivalent to three eight-accelerator servers. For operational headroom, four servers with 32 accelerators would be preferable;
- A single eight-accelerator server provides only 640 GB of accelerator memory and cannot hold the full model. Experts must be distributed across servers, with all-to-all communication carried over RDMA. The resulting latency is a key factor in determining whether 25 tokens per second can be sustained;
- For higher quality and more predictable performance, organizations can consider renting or purchasing systems with a supernode-style, high-density interconnect topology.
An eight-accelerator H-class server may cost approximately RMB 800,000–1.2 million. After adding three or four servers and a 200G or 400G InfiniBand network, the total investment is approximately RMB 2.5–4 million.
Cloud-based rental is another option. The monthly cost of equivalent accelerators can be compared with the purchase price to calculate the break-even period.
Option 3: Ten Concurrent Users at 25+ Tokens per Second—Supernode-Scale Deployment
Estimated budget: RMB 8–15 million
Supporting ten concurrent users at 25 tokens per second requires total system throughput of at least 250 tokens per second, without sacrificing per-request latency. This workload enters the range of the recommended supernode architecture.
- A supernode with at least 64 accelerators, using a fully interconnected communication domain for expert routing;
- Assuming approximately RMB 1 million per eight-accelerator server, the accelerator servers alone cost more than RMB 8 million;
- KV cache capacity increases with concurrency. Ten concurrent sessions with long contexts may require several hundred additional gigabytes of accelerator memory. High-end switching, redundant power, and suitable data-center facilities must also be included;
- The realistic deployment budget is approximately RMB 10–15 million;
- Continuous batching and other scheduling optimizations can improve accelerator utilization and bring the cost closer to the lower bound, but this requires a highly capable inference engine and substantial systems engineering.
5. Decision Guide
| Requirement | Recommended Architecture | Estimated Cost |
|---|---|---|
| Functional validation or offline workloads | CPU–GPU heterogeneous inference with 2 TB of system memory | RMB 150,000–300,000 |
| Smooth single-user experience | 24–32 accelerator cluster with high-speed interconnects | RMB 2–4 million |
| Production service for 10 concurrent users | Supernode with at least 64 accelerators | RMB 8–15 million |
Three practical recommendations follow.
First, if expected usage does not reach the concurrency level of Option 3, elastic cloud accelerator rental is likely to be the better choice. Building an on-premises system becomes economically attractive mainly under sustained, high utilization.
Second, once the weights are released, run consistency evaluations before and after quantization as soon as possible. QAT models generally experience much less quality degradation than models quantized after training, but this must be verified through testing.
Third, measure KV cache usage at the target context length before finalizing the hardware design. For Options 2 and 3, KV cache capacity may have a greater impact on memory planning than the weights themselves.
Local deployment of a large MoE model is never simply a matter of buying enough memory. Storage, interconnects, and scheduling can each become the next system bottleneck.
