Deploying Trillion-Parameter MoE LLMs Locally: Hardware Options and Cost Breakdown

1. Why Trillion-Parameter MoE Deployment Is a New Challenge

Large language models continue to scale up, and the Mixture of Experts (MoE) architecture has become a mainstream approach for balancing capability with compute cost by using sparse activation and dense storage. One frontier trillion-parameter MoE model, for example, has approximately 2.8 trillion total parameters and uses a routing mechanism with 896 experts, activating 16 experts per token. That puts the expert activation ratio at only about 1.79%, but the deployment complexity is far higher than that of traditional dense models.

At the time of writing, the complete weights and technical report for this model have not been officially released, but the official release is expected in the near future. All speed, memory, and hardware configuration figures in this article are estimates based on publicly available technical information. They will need to be validated through benchmarks once the weights are officially available.

This article covers three progressive deployment levels:

1. Minimum viable deployment: The model can be loaded and generate output, with no strict requirement on generation speed.
2. Single-user interactive deployment: Output speed reaches 25 tokens/s in a single-user interactive scenario.
3. High-concurrency production deployment: Supports 10 concurrent requests with per-user speed no lower than 25 tokens/s, and quantization no lower than 4-bit.

By breaking down these levels, readers can better understand the hardware investment and technical trade-offs required for different goals.

---

2. Model Architecture and Storage Requirements

2.1 Core Architecture Characteristics

This model uses an MoE architecture. Its backbone attention mechanism is KDA, a Delta Attention variant, combined with Attention Residuals and Gated MLA components. A key consideration for MoE deployment is that memory usage is determined by total parameters, while compute consumption is determined by activated parameters. Even though only 16 experts are activated per inference step, the weights of all 896 experts must still be loaded into addressable storage.

2.2 Theoretical Lower Bound for Weight Storage

The model incorporates quantization-aware training from the SFT stage, using MXFP4 weights with MXFP8 activations. Based on common industry conversion practices:

• 2.8 trillion parameters x 4 bit = 1,400 GB (raw lower bound)
• However, MXFP4 attaches an 8-bit scale factor for every 32 4-bit elements. According to the relevant specification: 2.8T x 4.25 bit / 8 = approximately 1,487.5 GB

This figure covers only the weights themselves. It does not include non-4-bit tensors such as embedding layers and normalization layers, model metadata, memory alignment overhead, inference runtime workspace, and dynamic KV Cache memory. In practice, the memory required for weights is typically 15% to 25% higher than the theoretical value.

2.3 Comparison with Mainstream Dense Models

For reference, a mainstream dense model with 671B parameters requires roughly 685 GB in FP8 format when including main parameters and multi-token prediction parameters. After AWQ 4-bit quantization, the weights take about 335 GB, with approximately 37 GB for activations, totaling around 400 GB. This can run on eight high-end data center GPUs with 80 GB each, providing 640 GB of total memory. A trillion-parameter MoE model is more than four times larger, which determines the lower bound for full deployment: weights alone approach 1.5 TB, posing a serious challenge for both single-node and multi-node deployments.

---

3. Three Deployment Levels and Hardware Cost Analysis

3.1 Level 1: Minimum Viable Deployment with Heterogeneous Inference

Goal: Load the model and complete inference without any commitment to speed.

Core approach: GPU + CPU heterogeneous inference. Each MoE layer consists of a shared attention computation module, which in this model includes KDA and Gated MLA. This part is compute-intensive and bandwidth-sensitive, making it suitable for GPU execution. The 896 sparse experts, although accounting for most of the storage, only activate 16 per inference step. They can be offloaded to large-capacity CPU memory on a high-memory server.

Typical hardware configuration:

Component	Recommended Specification	Notes
CPU	Dual-socket high-performance server CPU	Supports addressing beyond 1.5 TB of memory
System memory	1.5 TB - 2 TB DDR5 ECC	Holds all expert weights
GPU	1-2 mid-to-high-end data center GPUs	Runs only the attention portion
Storage	NVMe SSD array, >= 2 TB	Weight persistence and loading cache

Cost estimate: A dual-socket high-memory server with 1.5 TB of RAM and one or two GPUs would cost on the order of hundreds of thousands of RMB in hardware. The bottleneck is the PCIe bandwidth between CPU and GPU, which may limit inference speed to single-digit tokens per second. This configuration is suitable for offline batch tasks or functional validation.

3.2 Level 2: 25 Tokens/s for a Single User with a GPU Cluster

Goal: Achieve 25 tokens/s output in a single-user interactive scenario.

Core challenge: At 25 tokens/s, the system must complete roughly 2.8T x 1.79% = approximately 50 billion floating-point operations per second, plus KV Cache management and routing overhead. A single GPU or a small number of GPUs cannot handle this.

Hardware architecture:

• GPU count: At least 8-16 high-end data center GPUs with 80 GB each, providing 640-1,280 GB of total memory. Combined with CPU offload or model parallelism, this can barely cover the 1.5 TB weight requirement.
• Interconnect requirements: In MoE inference, activation values for each token must be routed to the GPU hosting the target expert. This all-to-all communication is extremely bandwidth-sensitive. If communication crosses node boundaries, latency will directly erode compute gains.
• Recommended topology: Single-node multi-GPU with NVLink or equivalent high-speed interconnect to form a unified communication domain, avoiding cross-node all-to-all.

Cost estimate: 8-16 high-end GPUs represent a hardware investment in the millions of RMB. With the server platform, high-speed networking, and storage, the total TCO can reach several million RMB.

3.3 Level 3: 10 Concurrent Users at 25 Tokens/s with a Supernode

Goal: Support 10 concurrent requests, each at no less than 25 tokens/s, with quantization no lower than 4-bit.

Official recommendation: A supernode consisting of 64 or more accelerators.

Technical rationale: The core bottleneck in large-scale MoE inference is all-to-all communication when experts are distributed across multiple GPUs. With 896 experts spread across many devices, every token goes through a cycle of “route -> transfer to target expert GPU -> compute -> transfer back.” If these transfers cross node boundaries, network latency becomes a hard ceiling. A supernode extends the high-speed interconnect domain to cover the entire node or rack, keeping expert data movement within the high-speed domain.

Hardware configuration overview:

Dimension	Specification
Accelerators	>= 64 high-end data center GPUs
GPU memory	80 GB per GPU, 5,120 GB total
Interconnect	NVLink / high-speed switching fabric with non-blocking all-to-all
CPU	Multi-socket high-performance CPU with large memory
System memory	>= 2 TB to support offload and KV Cache
Storage	High-throughput NVMe array or distributed storage

Cost estimate: GPU costs alone reach into the millions to tens of millions of RMB. A complete supernode, including servers, networking, storage, and data center facilities, can reach tens of millions of RMB. Ongoing costs for power, cooling, and operations must also be considered.

---

4. Key Technical Challenges and Mitigation Strategies

4.1 Balancing Quantization Precision and Performance

MXFP4 quantization compresses weights to 4-bit, but the 8-bit scale attached to every 32 elements adds storage overhead. During deployment, teams need to evaluate whether more aggressive quantization, such as 3-bit, is acceptable for speed, or whether 4-bit should be maintained for accuracy. This requires ablation experiments against specific tasks during benchmarking.

4.2 Communication Optimization

All-to-all communication is the primary performance bottleneck in MoE inference. Common optimization approaches include:

• Expert grouping and topology-aware routing: Place related experts within the same communication domain to reduce cross-domain data movement.
• Hybrid pipeline and tensor parallelism: Split the model across layers and within layers simultaneously to balance communication and computation.
• Gradient checkpointing and KV Cache compression: Reduce peak runtime memory to free up space for larger batches.

4.3 Elastic Scaling and Cost Control

For non-supernode scenarios, cloud platforms can provide elastic compute capacity: run a smaller cluster for off-peak traffic and temporarily scale up GPU instances during peak periods. However, MoE model loading times are long because of the 1.5 TB scale of data movement, so cold-start latency must be factored into SLA planning.

---

5. Summary and Recommendations

Deploying a trillion-parameter MoE model locally is a systems engineering challenge. The core tension lies in balancing approximately 1.5 TB of weight storage against efficient sparse-activation computation. The three levels correspond to three orders of investment:

• Validation-level: Hundreds of thousands of RMB, using a CPU plus a small number of GPUs in a heterogeneous setup.
• Production-level for a single user: In the millions of RMB, using an 8-16 GPU cluster.
• High-concurrency production-level: In the tens of millions of RMB, using a supernode with 64 or more GPUs.

Technical teams should perform detailed capacity planning and PoC validation based on publicly available architecture parameters before the official weights are released. Once the weights become available, the first step should be to validate memory usage, throughput, and latency with standardized benchmarks, then adjust the final hardware procurement plan accordingly. Cloud platforms offering GPU instances and high-performance storage services can serve as supplementary options for elastic deployment, helping teams reduce upfront investment risk during the early validation phase.