The Real Cost of Deploying a Trillion-Parameter MoE Model Locally

As of July 2026, the complete weights and technical report for a certain ultra-large mixture-of-experts (MoE) model with 2.8 trillion total parameters have not yet been released, with an official announcement expected in the near future. This article uses publicly available information about the architecture to answer one central question: how much does local deployment actually cost? It analyzes three performance tiers. One important caveat: aside from the officially published architecture and API pricing, all speed, memory, and hardware figures below are estimates based on publicly available information. They will need to be verified against real benchmarks once the weights are formally released.

1. Understanding Where the Model’s “Weight” Comes From

1.1 Sparse Activation Does Not Mean Small Storage

The model uses an MoE architecture with 2.8 trillion total parameters. Among its 896 experts, 16 are activated per token, giving an expert activation ratio of roughly 1.79%. There is a critical distinction to clarify here: expert activation ratio is not the same as parameter activation ratio. The activation ratio affects compute consumption, not memory footprint.

The memory logic of MoE deployment is straightforward: you must first load all 896 experts’ weights into addressable storage. At inference time, activations are routed to the corresponding 16 experts for computation. In other words, storage scales with total parameters, while compute scales with activated parameters.

1.2 The Lower Bound After Quantization

The model has undergone quantization-aware training since the SFT stage, with weights in a 4-bit floating-point format and activations in an 8-bit floating-point format. Converting 2.8 trillion parameters to 4-bit yields a raw data lower bound of approximately 1,400 GB.

However, a 4-bit floating-point format is not purely 4 bits per parameter. According to the OCP specification, every 32 4-bit elements carry an additional 8-bit scale value, making the effective size approximately 4.25 bits per weight element:

2.8T × 4.25 bit ÷ 8 ≈ 1487.5 GB

This is still just the raw data for weight elements plus scale values. It does not include non-4-bit tensors, metadata, alignment padding, runtime workspace, or KV cache.

1.3 Size Comparison with the Previous Generation

Using a widely known 671B-parameter MoE model as a reference: in FP8 format, the 671B main parameters plus 14B multi-token prediction parameters total about 685 GB. Even with a 4-bit quantization scheme, approximately 335 GB of weights plus 37 GB of activations—roughly 400 GB total—are needed to run on eight 80 GB accelerator cards (640 GB).

The target model is more than four times larger, which sets the storage floor for a full-scale deployment. To run it locally, the first challenge is figuring out how to store and address nearly 1,500 GB of weights.

2. Cost Breakdown Across Three Performance Tiers

Tier 1: Just Getting It to Run

The only goal here is to fit nearly 1,500 GB of weights into addressable storage and have the model produce tokens, even at single-digit speeds.

The most direct approach is GPU+CPU heterogeneous inference. Each MoE layer contains a shared attention component (including linear attention and gated multi-head latent attention in this model), which is compute-intensive and should stay on the GPU for high-bandwidth processing. The 896 sparse experts account for the bulk of storage but only 16 are activated at a time, so they can be offloaded entirely to CPU memory on a large-memory server.

This heterogeneous inference pattern has proven practical in industry. Using an open-source heterogeneous inference framework to run the 671B MoE model in 4-bit quantized form (approximately 377 GB), a dual-socket server CPU paired with a single flagship consumer GPU can deliver over 200 tokens/s during prefill and about 14 tokens/s during decode (with expert selection constraints enabled).

For the 2.8-trillion-parameter model, weights alone are approximately 1,487 GB. Adding KV cache and runtime overhead, plan for at least 2 TB of memory. For pure CPU inference, a dual-socket server with 512 GB of RAM is far from sufficient. You would need a 4-socket or 8-socket server with memory expanded to 2 TB or more. In terms of cost, large-memory server systems typically run in the hundreds of thousands of yuan. Adding one or two GPUs for attention computation, Tier 1 total hardware investment falls roughly in the RMB 200,000–500,000 range, depending on memory capacity and GPU selection.

Expected performance at this tier: single-digit tokens/s during decode and perhaps tens of tokens/s during prefill. It is suitable for validating model capabilities and offline batch inference, but not for interactive use.

Tier 2: Single-User Deployment at 25 Tokens/s

Twenty-five tokens per second is a key threshold—it is near the upper limit of human reading speed. At this rate, single-user interaction feels responsive.

To reach 25 tokens/s, the bottleneck shifts from “can we fit it” to “do we have enough compute.” MoE inference compute scales with activated parameters, but routing across 896 experts and all-to-all communication consume significant effective compute.

Key optimization directions:

• Expert caching strategies: Not all experts are activated equally often. Keep high-frequency experts resident in GPU memory and leave low-frequency experts in CPU memory to reduce data movement.
• GPU count and interconnect: A single GPU cannot hold all the weights, so multi-GPU is mandatory. Inter-GPU communication bandwidth directly determines decode speed. Over PCIe, all-to-all communication becomes a bottleneck; with high-speed interconnect (such as NVLink or equivalent technology), the communication domain is larger and latency is lower.
• Quantization level: Staying at or above 4-bit is the baseline. If weights can be stored at 4-bit with activations computed at 8-bit, compute cost is manageable. Compressing activations to 4-bit as well would require careful accuracy evaluation.

Hardware estimate: eight 80 GB accelerator cards give 640 GB total—not enough for 1,487 GB of weights. Sixteen 80 GB cards (1,280 GB) are still insufficient and would need CPU memory for expert offloading. Twenty-four 80 GB cards (1,920 GB) would be required to hold all weights in GPU memory.

With a 16-GPU plus large-memory CPU heterogeneous setup, where GPUs handle attention and high-frequency experts while the CPU covers low-frequency experts, hardware cost is roughly in the RMB 1–2 million range. A 24-GPU all-GPU configuration could push cost to RMB 3–5 million.

Expected performance at this tier: around 25 tokens/s decode for a single user and roughly 100 tokens/s prefill. Suitable for individual developers or small teams doing interactive inference.

Tier 3: 10 Concurrent Users at 25 Tokens/s Each

Ten concurrent users means total throughput must exceed 250 tokens/s. This is no longer a single-request optimization problem but a system throughput problem.

Key challenges:

• KV cache explosion: Ten concurrent requests each consume GPU memory for KV cache. Even with optimization, KV cache per token for a 2.8-trillion-parameter model is substantial. Under 10 concurrent requests, KV cache could occupy tens or even hundreds of gigabytes.
• Batching efficiency for expert routing: Within a batch, different tokens may route to different experts. Larger batches lead to more dispersed routing and more complex all-to-all communication. An efficient expert-parallelism strategy is essential.
• Supernode communication: The official deployment recommendation calls for a supernode of at least 64 accelerators. Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth. With 896 experts distributed across many cards, each token requires activation data to be moved to the target expert’s card, computed, and moved back. A supernode uses high-speed interconnect to enlarge the communication domain so that this movement stays within a single communication domain—a necessary condition for 10 concurrent users.

Hardware estimate: start with 64 accelerators plus supernode high-speed interconnect. Hardware cost is roughly in the RMB 10–20 million range. More aggressive configurations with additional cards and higher-bandwidth interconnect could cost more.

Expected performance at this tier: 25+ tokens/s per user under 10 concurrent requests, with prefill speed meeting interactive needs. Suitable for enterprise deployment serving a small team.

3. Directions for Cost Optimization

3.1 Further Quantization

4-bit is the baseline, but more aggressive schemes are worth exploring. For example, applying 3-bit or even 2-bit quantization to select layers while keeping accuracy-sensitive layers at 4-bit. This requires quantization-aware training or post-training quantization calibration, with accuracy loss evaluated layer by layer.

3.2 Expert Pruning and Distillation

Among the 896 experts, some may rarely be activated for domain-specific tasks. Expert pruning tailored to specific use cases can significantly reduce storage and compute requirements. Alternatively, distillation can transfer the large model’s capabilities to a smaller model.

3.3 Fine-Tuning Heterogeneous Inference

The GPU/CPU division of labor is not fixed. It can be dynamically adjusted based on actual load: high-frequency experts on GPU, low-frequency experts on CPU; attention computation on GPU, some feed-forward computation on CPU. The key is finding the balance point between communication overhead and compute overhead.

3.4 Leveraging Elastic Cloud Resources

If local deployment is too expensive, cloud-based elastic solutions are worth considering. Rent GPU instances on demand and release them when finished, avoiding large upfront hardware investment. Cloud providers typically offer GPU clusters with high-speed interconnect, suitable for large-scale MoE inference. Baidu AI Cloud and other cloud service providers offer various GPU instance types and high-speed interconnect options that can be configured flexibly based on actual requirements.

4. Summary and Recommendations

Performance Tier	Hardware Configuration	Estimated Cost	Use Case
Just running	Large-memory server + 1–2 GPUs	RMB 200,000–500,000	Offline batch inference, capability validation
Single user at 25 tps	16–24 GPUs + large-memory CPU	RMB 1–5 million	Individual interactive inference
10 concurrent users at 25 tps	64+ accelerators + supernode	RMB 10–20 million	Enterprise deployment

Recommendations:

1. Wait for the weight release first. All figures in this article are estimates. Once the weights and technical report are available, verify against real benchmarks.
2. Start with Tier 1 validation. Do not chase high performance from the start. Get the minimal configuration working first, then optimize incrementally.
3. Pay attention to the communication bottleneck. In large-scale MoE inference, the bottleneck is often not compute but all-to-all communication. Supernode high-speed interconnect is a necessity.
4. Consider cloud options. If local deployment is too costly or demand fluctuates significantly, cloud-based elastic solutions may be more economical.

Deploying an ultra-large MoE model locally is a demanding exercise in both hardware and engineering. Hopefully this breakdown helps clarify the path and identify the deployment approach that fits your needs.