Hardware Cost to Deploy a Trillion-Parameter Model Locally

1. Starting Point: Parameter Scale Sets the Memory Floor

Any discussion of locally deploying a very large sparse model has to begin with memory. Take a MoE model with roughly 2.8 trillion total parameters as an example. Its defining characteristic is that all expert parameters must remain resident in addressable storage, while each token activates only a very small fraction of them for computation.

Assume the model contains 896 experts and each token routes to 16 of them, giving an expert activation ratio of about 1.79%. Note that this is the expert activation ratio, not strictly the parameter activation ratio. Dense components such as the attention backbone and shared feed-forward layers always participate in computation. The architecture may use a linear attention variant with residual attention connections, with gated MLA as one component.

The key engineering implication is that MoE deployment memory usage is determined by total parameter count. The activation ratio affects compute consumption, not storage requirements. You must first place all 896 experts into storage, then dynamically load 16 of them at inference time based on routing. This is the premise for understanding all hardware options below.

2. Memory Accounting: From Weights to Runtime Overhead

2.1 Lower Bound for Raw Weight Storage

Assume the model has undergone quantization-aware training from the supervised fine-tuning stage, with 4-bit floating-point weights and 8-bit floating-point activations. For 2.8 trillion parameters at 4 bits, the raw data lower bound is:

2.8T x 4 bit / 8 = 1400 GB

Actual usage is higher. Under the common convention for 4-bit floating-point formats, every 32 4-bit elements carry one 8-bit scaling factor, so the effective bit width is:

4 + 8/32 = 4.25 bit
2.8T x 4.25 bit / 8 = ~1487.5 GB

This still excludes non-4-bit tensors such as embedding and normalization parameters, metadata, memory alignment, runtime workspace, and KV cache.

2.2 Comparison with a Mid-Scale MoE Model

For reference, a MoE model with about 671B total parameters, including around 14B multi-token prediction parameters, requires roughly 685 GB of weights in 8-bit floating-point format. Even with 4-bit weight quantization, it needs about 335 GB for weights plus 37 GB for activations, totaling around 400 GB to run on eight high-bandwidth accelerators with 640 GB of combined memory.

A 2.8-trillion-parameter model is more than four times the size of that mid-scale model. The memory floor for full deployment is therefore set: at least around 1.5 TB of addressable storage for weights, plus KV cache and runtime overhead.

2.3 KV Cache and Activation Overhead

KV cache size depends on context length, batch size, and attention head configuration. For long-context scenarios such as 128K tokens, KV cache can reach tens or even hundreds of gigabytes. Activation overhead scales linearly with batch size and sequence length. Both must be included when estimating total memory requirements.

3. Communication Topology: Why a Supernode Is the Recommended Approach

Large-scale MoE inference is highly sensitive to all-to-all communication bandwidth. With 896 experts spread across multiple accelerator cards, each token must have its activations moved to the cards holding its routed experts, computed there, and the results moved back. Communication latency in this process directly eats into compute efficiency.

The official recommendation is to use a supernode consisting of 64 or more accelerators. The core reason is that high-bandwidth interconnect such as NVLink or equivalent high-speed fabrics keeps the communication domain large enough that expert traffic never leaves a single domain. If communication crosses nodes, network bandwidth and latency become severe bottlenecks, and decode throughput drops sharply.

From an engineering perspective, the size of the communication domain directly determines the efficiency ceiling for MoE inference. When budget allows, ensuring that all experts fit within a single communication domain should be the central design principle.

4. Feasible Paths: GPU and CPU Heterogeneous Inference

4.1 Why a Pure GPU Deployment Is Extremely Expensive

If GPU memory alone must hold roughly 1.5 TB of weights, even 80 GB accelerators would require at least 19 cards just for weight storage. Adding KV cache and runtime overhead pushes the requirement above 24 cards. With supernode interconnect, a 64-card configuration would put hardware cost at an extremely high level.

4.2 Core Idea of Heterogeneous Inference

A more practical approach is hybrid CPU+GPU inference. The key insight is:

• The shared attention components, including linear attention and gated MLA, are compute-intensive and should remain on GPUs for high-bandwidth computation.
• The sparse expert components account for the vast majority of storage, roughly 95% or more of all weights, but only 16 experts are activated per token. These can be offloaded entirely to CPU memory and served from high-capacity servers.

A typical configuration uses a dual-socket server CPU with large DDR5 memory, starting at 512 GB and preferably 1 TB or more, plus one or two high-memory GPUs. The CPU handles expert weight storage and retrieval, while the GPU handles attention computation and expert forward computation after routing.

4.3 Performance Expectations

Based on publicly reported heterogeneous inference practices, a dual-socket server CPU with a single consumer GPU running a 671B MoE model at 4-bit quantization, roughly 377 GB of weights, can reach:

• 200+ tokens/s in the prefill stage
• About 14 tokens/s in the decode stage with an expert selection strategy enabled

For a 2.8-trillion-parameter model, the weight footprint is more than four times that of the 671B model, and the larger number of experts adds routing overhead. Decode speed can be expected to drop further. But the core goal remains the same: fit nearly 1500 GB of weights into addressable storage so the model can emit tokens, even at only single-digit tokens per second.

5. Cost Estimation Framework

5.1 Pure GPU Supernode Deployment

Item	Specification	Estimated Cost Range
Accelerators	64 x 80GB	Extremely high
Supernode interconnect	High-bandwidth fabric	High
Server platform	8+ sockets	High
Total	-	Tens of millions RMB

5.2 GPU + CPU Heterogeneous Deployment

Item	Specification	Estimated Cost Range
CPU	Dual-socket server grade	50,000-150,000 RMB
Memory	1TB DDR5	30,000-80,000 RMB
GPU	1-2 high-memory cards	50,000-200,000 RMB
Storage	NVMe SSD 4TB+	5,000-10,000 RMB
Total	-	150,000-450,000 RMB

The cost advantage of the heterogeneous approach is an order of magnitude, at the cost of decode speed dropping from interactive levels to single-digit tokens per second. For research, validation, and low-frequency inference scenarios, this is currently the most practical local deployment path.

6. Practical Recommendations

1. First confirm whether model weights are actually available. Without weight files, all hardware discussion is moot.
2. Quantization is mandatory. A 4-bit quantization-aware trained model is the prerequisite for local deployment. 8-bit or higher precision doubles storage requirements.
3. Memory capacity matters more than bandwidth. In a hybrid setup, CPU memory capacity determines whether all experts can be loaded, while bandwidth determines retrieval speed. Prioritize capacity first.
4. Keep all experts within one communication domain. For multi-card deployments, make sure a single communication domain can hold all 896 experts to avoid cross-domain traffic.
5. Verify with benchmarks. All speed, memory, and hardware figures should be treated as estimates based on publicly available information. Once weights are released, validate each item against actual benchmarks.

Local deployment of a trillion-parameter model is fundamentally an engineering trade-off among storage capacity, communication bandwidth, and compute density. There is no universal solution, only an optimal configuration for a specific budget and use case.