Local Deployment of a Trillion-Parameter MoE Large Model: Practice and Cost Estimates

As of now, a highly anticipated mixture-of-experts large model with a reported total of 2.8 trillion parameters has not yet released its full weights or a complete technical report. According to the official announcement, the full capability is expected to be open-sourced in the near future. The model has 2.8 trillion total parameters, uses a mixture-of-experts architecture with 896 expert networks, and activates only 16 experts per token during inference. This results in an expert activation ratio of approximately 1.79%.

Before estimating the cost of deploying such a model locally, one core technical premise must be clear: for MoE architectures, memory usage is determined by the total parameter count. The activation ratio only affects compute consumption during inference. In other words, no matter how few experts are activated, the full weight set still needs to be loaded into storage first. In addition, the model introduces quantization-aware training during the supervised fine-tuning stage and natively supports MXFP4 weights with MXFP8 activations.

Based on this background, local deployment requirements can be divided into three progressive levels, each with its own hardware configuration and cost estimate.

1. The Underlying Logic of Memory Footprint and a Baseline Estimate

To estimate deployment cost, the first step is to calculate the minimum memory requirement. For 2.8 trillion parameters at 4-bit precision, the theoretical lower bound for raw data is about 1,400 GB. However, under the OCP specification, MXFP4 requires one 8-bit scaling factor for every 32 4-bit elements. Combining weight elements and scaling factors gives:

(2.8T × 4.25 bit) ÷ 8 ≈ 1,487.5 GB

This figure of nearly 1,500 GB is purely for weight data. It does not yet include non-4-bit tensors, metadata, memory alignment overhead, temporary workspace during inference, or the dynamically growing KV cache. For reference, a widely used 671B-parameter model requires about 685 GB of memory in FP8. Even with AWQ 4-bit quantization, it still needs 335 GB for weights plus 37 GB for activations, totaling close to 400 GB to barely run on eight H100 GPUs with 640 GB of total memory. Since the target model is more than four times larger, the memory barrier for a full deployment is extremely high.

2. Level 1: Just Enough to Run

Goal: Fit the roughly 1,500 GB of weights into addressable storage so the model can generate tokens, even if the output is only a few tokens per second.

Approach: CPU-GPU heterogeneous inference

At this level, the budget is limited but extremely high latency is acceptable. The most direct approach is CPU-GPU heterogeneous inference. In an MoE model, each layer typically contains a shared attention component — in this case, a KDA architecture with a Gated MLA component — and many sparse experts. The attention component is compute-intensive and bandwidth-sensitive, so it should remain on GPUs with high-bandwidth memory. The 896 sparse experts account for most of the storage but only 16 are activated per inference step, so the full set of expert networks can be offloaded to CPU memory on a high-capacity memory server.

Cost estimate:

This requires a server with large memory capacity, such as 2 TB to 4 TB of DDR5, paired with a small number of high-end consumer GPUs or entry-level professional GPUs for handling attention and routing. The hardware cost can be kept relatively low, but inference speed is severely constrained by CPU memory bandwidth, typically in the range of hundreds of GB/s. This setup is only suitable for technical validation and very low-frequency offline data processing.

3. Level 2: Single-User Deployment at 25 TPS

Goal: With quantization no lower than FP8 or an equivalent 4-bit format, deliver a smooth single-user experience of around 25 tokens per second.

Approach: GPU-resident weights and KV cache optimization

To reach 25 TPS, a pure CPU offloading approach will hit a memory bandwidth wall. Most frequently accessed weights must remain resident in GPU memory. For a single user and a single concurrent request, compute is not the main bottleneck. The bottlenecks are total memory capacity and per-GPU memory bandwidth.

At this level, the 1,487.5 GB of weights must be distributed across multiple accelerators. With 80 GB per card, at least 19 cards are needed just for the raw weights. Accounting for system overhead and KV cache, a cluster of 20 to 24 accelerators is more realistic. On the software side, an efficient scheduling framework is required, using techniques such as Paged Attention to reduce KV cache fragmentation, along with high-speed PCIe switches to prevent CPU-GPU data transfer from becoming a drag on performance.

Cost estimate:

This approach requires a small compute cluster made up of more than 20 high-memory accelerators, pushing hardware cost into the millions of yuan. To further reduce cost, some organizations may opt for high-memory consumer GPUs, but this introduces risks in driver stability and long-term reliability. For production environments, professional accelerators remain the preferred choice.

4. Level 3: Enterprise-Grade Supernode Architecture for 10 Concurrent Users at 25 TPS or Higher

Goal: Support 10 concurrent requests, each with an output speed of at least 25 TPS, with quantization no lower than 4-bit. This is a typical high-concurrency production scenario for enterprises.

Approach: Supernode cluster and high-speed interconnect

The official recommendation for deploying such an extremely large MoE model is to use a supernode cluster of 64 or more accelerators. Why such a large scale? The key reason is that large-scale MoE inference is highly sensitive to all-to-all communication bandwidth.

With 10 concurrent requests and 896 experts distributed across multiple cards, every token generation step requires the router to send activations to the physical card where the target expert resides, perform the computation, and then move the result back. This cross-card communication is extremely frequent. Without sufficient inter-accelerator bandwidth, network latency will directly consume GPU compute capacity and cause concurrent performance to collapse.

A supernode enlarges the communication domain through NVLink or an equivalent high-speed interconnect, so expert activation traffic stays within a single communication domain and avoids the latency penalty of cross-node networks such as InfiniBand or RoCE. At the same time, 64 accelerators with 80 GB each provide a 5,120 GB memory pool, which comfortably holds the 1,500 GB of weights while leaving ample buffer for multi-concurrent KV cache.

Cost estimate:

Building a supernode cluster with 64 high-end accelerators, high-speed switches, large-capacity storage nodes, and redundant power supplies requires an infrastructure investment in the tens of millions of yuan. For AI-native application developers that need to support high concurrency, this is a heavy upfront capital expenditure. Organizations that want to reduce initial investment can consider cloud platforms offering large-model compute support, converting capital expenditure into operational expenditure through on-demand instances or dedicated capacity, allowing more flexibility in handling traffic peaks and valleys.

Summary

Locally deploying a trillion-parameter MoE large model is not simply a matter of stacking GPUs. It is a systems engineering challenge involving storage addressing, communication topology, and heterogeneous computing. From a minimal heterogeneous offloading setup, to a GPU-resident configuration optimized for single-user experience, to a supernode interconnect architecture for high concurrency, the cost rises from hundreds of thousands to tens of millions of yuan. Before the official weights are released, all planning should be based on rigorous memory and bandwidth calculations. Organizations preparing for large-model deployment should objectively evaluate their concurrency requirements and latency targets, then choose the most cost-effective deployment level.