Trillion-Parameter MoE Local Deployment: Hardware Tiers and Cost Analysis

The primary challenge of deploying a trillion-parameter MoE model in a local data center is not just securing the budget, but understanding exactly which constraint the money must solve. Memory capacity determines if the model fits, compute determines token generation speed, and interconnect bandwidth dictates whether multi-GPU scaling yields linear performance improvements. Failing to meet any of these requirements simply shifts the bottleneck elsewhere.

This article discusses capacity planning and hardware configurations across three deployment tiers, using a reference MoE model with approximately 2.8 trillion total parameters (896 experts, 16 active per token). All speed, memory, and hardware figures are estimates based on publicly available architectural information and should be verified with benchmarks once the official weights are released.

1. Establishing an Accurate Capacity Model

1.1 Expert Activation Ratio ≠ Parameter Activation Ratio

A common misconception with MoE models is assuming that because only 16 out of 896 experts are activated per token (~1.79%), the memory requirement can also be calculated at 1.79%. This is incorrect.

All experts must reside in addressable memory at all times, fetched during inference based on routing results. The activation ratio only affects compute consumption (FLOPs), not the memory footprint for weights. In other words, you can generate a token with very little compute, but only if all 2.8 trillion parameters are already loaded into memory.

1.2 Minimum Weight Footprint Post-Quantization

This model class introduces quantization-aware training from the SFT phase, utilizing a 4-bit floating-point format for weights and an 8-bit floating-point format for activations. Calculating based on 2.8 trillion parameters:

• Pure 4-bit raw data: 2.8T × 4 bit ÷ 8 ≈ 1400 GB
• Including block-wise quantization scale overhead (one 8-bit scale per 32 4-bit elements, equivalent to 4.25 bits): 2.8T × 4.25 ÷ 8 ≈ 1487.5 GB

This 1487.5 GB accounts only for the weight elements themselves. Actual deployment requires additional overhead:

Overhead Item	Description
Non-4-bit Tensors	Certain layers, normalization parameters, and embeddings usually retain higher precision
Metadata & Alignment	Tensor headers, alignment padding, framework structural overhead
Runtime Workspace	Operator temporary buffers, communication buffers
KV Cache	Scales linearly with concurrency and context length

Conclusion: The minimum storage for a full deployment is around the 1.5 TB mark. In engineering practice, 1.6 to 1.8 TB should be provisioned.

1.3 A Reference Baseline

Taking a 671B-scale MoE model as a reference: in FP8 format, the main parameters plus multi-token prediction parameters occupy about 685 GB. With 4-bit weight quantization, this drops to roughly 335 GB for weights plus 37 GB for activations, totaling about 400 GB. This fits onto 8x 80GB GPUs (640 GB total) with room for workspace. The 2.8 trillion parameter model is over four times this size, establishing the lower bound for a full deployment—the single-node 8-GPU route is completely unviable from a capacity standpoint.

1.4 Interconnect Bandwidth: The Underestimated Constraint

Large-scale MoE inference is extremely sensitive to all-to-all communication. With 896 experts distributed across multiple GPUs, the activations must be transferred to the target expert’s device after routing, and transferred back after computation. Communication latency will directly erode compute utilization.

Therefore, official deployment recommendations for such models suggest a supernode architecture comprising 64 or more accelerators, utilizing high-bandwidth interconnects (like NVLink or equivalent high-speed buses) to create a large communication domain, ensuring expert transfers do not cross individual domains. This has a structural impact on cost: it’s not about “buying more GPUs,” but rather “stepping up an interconnect tier.”

2. Three Target Tiers and Corresponding Solutions

Tier 1: Just Run It, Ignore Speed

The sole objective is to fit the ~1.5 TB weights into addressable storage and generate tokens, even if the speed is only single-digit tokens per second.

The most straightforward approach is GPU + CPU heterogeneous inference: The shared attention component in each MoE layer (compute-intensive) remains on the GPU for high-bandwidth computation. The 896 sparse experts, which consume the bulk of storage but only activate 16 at a time, are offloaded entirely to CPU memory using a large-memory server.

This approach has mature engineering implementations. A typical configuration looks like:

• Compute side: 1 to 2 high-end consumer or data center GPUs for attention and routing computation.
• Storage side: Dual-socket server platform + large-capacity DDR5 memory, provisioned at 1.5 to 2 TB.
• Interconnect: PCIe between GPU and CPU, swapping expert weights in and out on demand.

Public data for similar approaches on a 671B-scale model with 4-bit quantization (~377 GB) shows prefill speeds of 200+ tokens/s and decode speeds around 14 tokens/s. However, these figures usually come with conditions like specific expert selection strategies and cannot be directly extrapolated to a 2.8 trillion parameter scale. At 4x the parameter size, Tier 1 expectations should be lowered to single-digit tokens/s.

Cost Estimate (complete system):

Component	Estimated Cost Range (CNY)
Dual-socket server platform (incl. 2TB memory)	80,000 ~ 150,000
GPU(s) (1 to 2 units)	20,000 ~ 60,000
Storage, power, and cooling	20,000 ~ 40,000
Total	~120,000 ~ 250,000

This is the only route affordable for individuals or small teams, at the cost of speed and concurrency.

Tier 2: Single-User Output at 25 tps

25 tps is the threshold for a smooth reading experience—anything below this, and the user can visibly see tokens generating one by one. To achieve this, the memory bandwidth of the CPU offload route is typically insufficient. Most experts must be placed in GPU VRAM or high-bandwidth unified memory.

Key estimation logic:

1. Capacity: For ~1.5 TB weights, at least 24 to 32 80GB-class accelerators are needed for pure VRAM capacity, or a hybrid architecture combining GPU VRAM and a high-bandwidth unified memory pool.
2. Compute: Although the activated parameters per token are small, the attention portion and routing overhead are fixed. Combined with expert transfers, the effective compute utilization is typically only 20% to 40% of peak performance.
3. Interconnect: At minimum, full interconnectivity across 8 GPUs within a single node is required. Cross-node deployments need high-bandwidth, low-latency networks, otherwise expert transfers become the primary source of latency.

A pragmatic intermediate form: An 8-GPU high-bandwidth interconnected node + large host memory acting as a secondary cache for experts. Hot experts reside permanently in VRAM, while cold experts are loaded on demand. Under this configuration, 25 tps for a single user is achievable, but it requires tuning cache hit rates based on the routing distribution. This is a “trade engineering effort for money” solution.

Cost Estimate:

Component	Estimated Cost Range (CNY)
8-GPU high-bandwidth interconnected node	1.5M ~ 2.5M
Host memory expansion (1 to 2 TB)	50,000 ~ 100,000
High-speed network and storage	100,000 ~ 200,000
Total	~1.7M ~ 2.8M

Tier 3: 10 Concurrent Users, Each at 25+ tps

This is the production-grade threshold. 10 concurrent users × 25 tps = 250 tps aggregate throughput, with quantization no lower than 4-bit floating-point.

Three key changes:

1. KV cache becomes a major memory consumer. As concurrency rises, KV cache usage can reach tens to hundreds of GBs. It must be planned as a separate item in capacity planning, rather than squeezing into the “leftover space” from the weights.
2. Batch scheduling is mandatory. Continuous batching fills the compute bubbles across multiple requests, but it also means higher peak memory usage and more complex scheduling.
3. The communication domain must be large enough. At 10 concurrent users, the frequency of all-to-all communication multiplies. If cross-node communication is not within the same high-speed domain, throughput will be locked by a communication wall. This is why official recommendations suggest supernodes with 64 or more GPUs.

Cost Estimate:

Component	Estimated Cost Range (CNY)
64-GPU supernode (incl. high-speed interconnect)	15M ~ 25M
Storage and cluster management nodes	500,000 ~ 1M
Datacenter upgrades (power, cooling)	1M ~ 2M
Total	~16.5M ~ 28M

3. Pre-Deployment Verification Checklist

Once the weights are released, do not rush to place orders. Run through these checks first:

1. Actual on-disk weight size (including scales, non-4-bit tensors, alignment padding)
2. Idle resident VRAM / memory footprint
3. Single-request prefill throughput and decode throughput (differentiated by context length)
4. Aggregate throughput and P99 latency at concurrency levels 1 / 4 / 10
5. Expert routing distribution histogram -> determines caching strategy and hot expert replication count
6. all-to-all communication proportion -> determines if interconnect is the bottleneck
7. Measured KV cache usage at the target context length

Item 5 is the most frequently skipped, yet it directly determines the viability of the Tier 2 solution: if routing is highly concentrated, replicating a few hot experts will significantly boost speed; if the distribution is uniform, caching strategies yield limited returns, and you must brute-force with more VRAM.

4. Conclusion

The fundamental difference between the three tiers is not “how many GPUs you have,” but which constraint you are willing to pay for:

• Tier 1 pays for capacity, trading speed for cheap, large memory.
• Tier 2 pays for bandwidth, using high-bandwidth interconnects to push expert transfer overhead to an acceptable range.
• Tier 3 pays for the communication domain, using a supernode to lock all-to-all communication within a single enclosure.

For most teams, the recommended path is: use Tier 1 to validate business feasibility, move to Tier 2 for small-scale piloting, and only consider Tier 3 after confirming the ROI. After all, 1.5 TB of weights won’t shrink just because you buy more GPUs, but your bill certainly will grow.