Deploying Trillion-Parameter MoE Models Locally: Hardware Thresholds and Cost Estimation Guide

1. Understanding the Scale: What Makes Trillion-Parameter MoE Models Heavy?

Before discussing local deployment, we must break down the actual size of the model. Taking a Mixture of Experts (MoE) architecture with approximately 2.8 trillion total parameters as an example, its structural characteristics dictate the deployment method and cost structure:

• Total Parameter Scale: ~2.8 trillion, with 896 experts;
• Activation Mechanism: Each token activates only 16 experts after routing, meaning the expert activation ratio is ~1.79%;
• Backbone Architecture: A linear attention backbone (KDA, a linear attention variant like Delta Attention) combined with Attention Residuals, using Gated MLA as the attention component;
• Quantization Scheme: Quantization-aware training is introduced from the SFT stage. Weights use a 4-bit floating-point format (MXFP4), and activations use an 8-bit floating-point format (MXFP8).

A common misconception needs clarification: the expert activation ratio does not equal the parameter activation ratio. The 1.79% only indicates the proportion of experts involved in the forward pass per token. However, during deployment, all 896 experts must reside in addressable storage, and the router dispatches tokens to the corresponding experts during inference. Therefore:

• VRAM/Memory footprint is calculated based on total parameters;
• Compute consumption is calculated based on activated parameters.

This creates the core contradiction in deployment: compute demand is relatively manageable, but storage demand is exceptionally massive.

2. Weight Footprint: From Raw Data to Actual Usage

2.1 Theoretical Lower Bound

2.8 trillion parameters stored in 4-bit format yield a raw data lower bound of approximately:

2.8e12 × 4 bit ÷ 8 = 1400 GB

2.2 Actual Weight Size with Scale Factors

MXFP4 is not a pure 4-bit packed format. According to the OCP (Open Compute Project) Microscaling standard, every 32 4-bit elements are accompanied by an 8-bit scale factor. Averaged out, the equivalent bit width per weight element is:

4 + 8/32 = 4.25 bit

Thus, the weight volume becomes:

2.8e12 × 4.25 bit ÷ 8 ≈ 1487.5 GB

2.3 Additional Overhead

The above figure still only represents “weight elements + scale factors”. Real-world deployment must also account for:

• Non-4-bit tensors (e.g., certain norm layers, embeddings, and routing networks usually retain higher precision);
• Tensor metadata and alignment padding;
• Runtime workspace (activation buffers, temporary tensors);
• KV cache (grows linearly with context length, crucial for long-context scenarios).

An empirical rule in practice is to reserve a 15% to 30% margin over the theoretical value. This means the actual storage requirement for a full deployment roughly falls in the 1.7TB to 1.9TB range.

2.4 Horizontal Comparison

For comparison, a MoE model with roughly 671B main parameters + 14B MTP parameters requires about 685GB of weights in FP8 format. Even with a 4-bit weight quantization scheme, the weights are around 335GB, plus ~37GB of activations, totaling ~400GB to fit into 8 80GB accelerators (640GB VRAM).

The 2.8-trillion-parameter model is more than four times that size. This means an 8-card machine cannot even “fit” the model. The hardware lower limit for a full deployment is consequently raised by an order of magnitude.

3. Why Super-Nodes Are Recommended: Communication is the Hidden Ceiling

The performance bottleneck for large-scale MoE inference is often not compute, but all-to-all communication.

With 896 experts distributed across multiple accelerators, the activations of each token must be transferred to the device hosting the target expert for calculation, and then transferred back. In this process:

1. Communication volume is determined by the activation ratio, expert distribution, and batch size;
2. If expert distribution crosses multiple communication domains (across nodes), latency directly eats up compute utilization;
3. The higher the expert parallelism, the more all-to-all hops occur, increasing sensitivity to interconnect bandwidth.

Therefore, a reasonable approach is to enlarge the communication domain using high-bandwidth interconnects (NVLink or equivalent) to ensure that activation transfers between experts stay within a single domain. This explains why the recommended deployment topology for such models is a super-node composed of 64 or more accelerators—it is not about lacking compute, but about confining communication to one domain.

4. Path 1: Minimum Viable Inference via Heterogeneous Setup

If the goal is simply to get the model to output tokens without pursuing high throughput, a heterogeneous GPU+CPU inference setup is the most realistic starting point.

4.1 Core Concept

Each layer of an MoE model consists of two parts:

• Shared Attention Component (KDA + Gated MLA in this case): Compute-intensive with a relatively small parameter footprint, placed on GPUs for high-bandwidth processing.
• Sparse Expert Component: Occupies the vast majority of storage but only activates a few experts per token. This is entirely offloaded to CPU memory, hosted by large-capacity memory servers.

During inference, after the GPU completes the attention calculation, it routes the activations to the corresponding experts on the CPU side, retrieves the results, and continues. This division of labor—keeping compute on the GPU and storage in system memory—is the typical design of heterogeneous inference frameworks.

4.2 Reference Metrics

In public high-performance heterogeneous inference cases, a dual-socket, high-core-count server CPU paired with a single consumer-grade flagship GPU running ~377GB of 4-bit quantized weights achieved prefill speeds of over 200 tokens per second and decode speeds of ~14 tokens per second (note: these figures include expert selection strategy constraints and represent ideal conditions).

Extrapolating this scale to 1.5TB of weights: memory capacity needs to be at least 1.5TB (2TB recommended for margin), the CPU must have sufficient memory channels and cores to support expert computation, and the GPU handles attention and scheduling. The decode speed will be significantly lower than the reference above. Single-digit token/s is a highly possible outcome, but this already satisfies the definition of “minimum viable inference”.

4.3 Cost Breakdown (Estimation Framework)

Component	Specification Direction	Estimated Cost Range
CPU	Dual-socket, high-core-count server CPU	Tens of thousands (RMB)
Memory	2TB DDR5 ECC	Hundreds of thousands (RMB)
GPU	Single large-VRAM consumer/professional card	10k~30k (RMB)
Motherboard/PSU/Chassis	Server or workstation platform	~10k (RMB)
Storage	2TB-class NVMe (for weights)	~1k (RMB)

The overall budget falls roughly in the 150,000 to 250,000 RMB range, fluctuating based on memory capacity and CPU model. This is the minimum threshold to run the model, at the cost of slow speeds and weak concurrency.

5. Path 2: Full Performance via GPU Super-Nodes

If the goal is a usable inference service (multi-concurrency, acceptable latency), a full-GPU route is necessary.

5.1 Hardware Scale Estimation

Assuming 80GB of VRAM per card, the 1.7TB~1.9TB actual weight + runtime footprint requires at least:

1900 GB ÷ 80 GB ≈ 24 cards

However, this is just the lower bound to “fit” the model. Factoring in KV cache, activation workspace, communication buffers, redundancy, and the officially recommended 64+ card super-node topology, actual configurations typically range between 32 to 64 high-bandwidth interconnected accelerators, depending on context length and concurrency targets.

5.2 Cost Scale

Based on current market conditions (order-of-magnitude reference only, subject to actual procurement quotes):

• A single 80GB-class accelerator card: Tens of thousands (RMB);
• A 64-card node including high-speed interconnect switches and server chassis: Tens of millions (RMB);
• Supporting facility power, cooling, and networking: Additional millions (RMB).

Even when scaled down to a 32-card cluster, the total investment generally sits in the single-digit millions (RMB) range. This is clearly beyond the reach of individual developers or small teams.

5.3 A Compromise Option

If you want better speed than the heterogeneous approach but cannot afford a super-node, consider a hybrid setup: medium-scale GPU cluster + weight sharding + CPU offloading. High-frequency experts can reside permanently on GPUs, while low-frequency experts stay in system memory, scheduled dynamically based on routing popularity. This approach has high engineering complexity but can suppress hardware costs to the hundreds of thousands (RMB) range, serving as a realistic middle ground.

6. Cost Quick Reference

Deployment Target	Typical Configuration	Budget Scale	Expected Performance
Minimum Viable	Dual-socket CPU + 2TB Memory + Single GPU	150k~250k RMB	Single-digit token/s (decode)
Hybrid Compromise	8~16 GPUs + Large Memory	Hundreds of thousands RMB	Tens of token/s (decode), limited concurrency
Full Service	32~64 card super-node	Millions to tens of millions RMB	Production-grade concurrency

7. Pre-Deployment Checklist

1. Verify quantization format and weight size: 4-bit does not mean 4 bits/parameter. Always factor in scale overhead according to the Microscaling standard;
2. Verify storage addressability: In heterogeneous setups, system memory must be able to hold all experts. Relying on SSD swapping is unacceptable due to latency;
3. Evaluate communication domains: For multi-card setups, calculate all-to-all volume before deciding on the interconnect topology;
4. Reserve runtime margin: Leave at least 15%~30% capacity beyond the raw weights;
5. Run benchmark tests: Any publicly available speed figures are estimates. You must stress test with actual workloads before going live.

Conclusion

Deploying a trillion-parameter MoE model locally is fundamentally a storage problem, not a compute problem. The weight volume dictates the hardware lower bound, the communication topology dictates the performance ceiling, and the quantization format details determine if your calculations are accurate. For most developers, heterogeneous inference is the only realistic starting point; for enterprises capable of investing, super-nodes are the correct topology to unlock the model’s full capabilities. Before signing off on the budget, run through the checklist above—what you save will be more than just money.