Deploying a 2.8-Trillion-Parameter MoE Model Locally: Hardware and Cost Guide
This guide estimates the storage, hardware, and budget needed to run a 2.8-trillion-parameter Mixture-of-Experts (MoE) model locally. It covers three targets: getting the model to run, serving one user at 25 tokens per second (TPS), and serving 10 concurrent users. It also explains how to size memory, GPU count, interconnect, and replica count, and how to reduce memory use and cost.
Overview
Before deploying a trillion-parameter-class MoE model locally, you need to calculate two things: how much storage the weights require, and what hardware each speed target needs. This article uses an MoE model with about 2.8 trillion total parameters as an example and walks through the calculation from storage to budget.
1. Define Goals and Assumptions
- Goal 1: Run the model. The model loads and completes inference. There is no output speed requirement.
- Goal 2: Smooth single-user use. Output speed for one user is at least 25 TPS (output tokens per second). This is a common minimum for chat applications.
- Goal 3: 10 concurrent users. Ten users run at the same time, and each still gets at least 25 TPS, for a total throughput of about 250 TPS.
All prices below are estimates in Chinese yuan (RMB) for new equipment. They exclude electricity, data center renovation, and operations staff. Hardware prices depend heavily on market conditions and supply, so confirm with formal quotes before purchasing.
2. Storage: Sparse Activation Does Not Reduce Memory Proportionally
The model activates only a small subset of experts for each token, for example 16 out of 896 experts. However, routing is decided dynamically by the input tokens, so any GPU may receive a request for any expert. All expert weights must therefore stay resident in GPU memory or system memory. The minimum storage is set by total parameters, not active parameters.
At a 4-bit floating-point format (similar to MXFP4):
- Weights: 2.8T × 0.5 bytes ≈ 1400 GB
- Scaling factors: one 8-bit scaling factor per 32 elements, adding about 87.5 GB
- Total: about 1487 GB
This covers only the weights. Tensors that are not 4-bit, metadata, alignment padding, runtime workspace, and the KV cache (key-value cache) require additional memory. Treat 1487 GB as the lower bound for planning.
3. Three Deployment Paths
Path 1: CPU Memory Plus GPU (Runs, but Slow)
This path places the compute-heavy shared layers, such as attention, on the GPU. It places the sparse experts, which make up most of the model’s size, in CPU memory.
- Memory requirement: At least 1487 GB for the weights, plus runtime overhead. A single machine with at least 2 TB of memory is recommended. For example, 24 DDR5 sticks of 96 GB each give about 2304 GB in total.
- GPU: One 24 GB GPU can hold the shared layers and the KV cache for shorter contexts. As context length grows, the KV cache competes for GPU memory, so plan for it separately.
Speed limit: Each generated token requires reading the weights of the activated experts from memory. The theoretical upper bound can be estimated with:
TPS upper bound ≈ effective memory bandwidth ÷ weight bytes actually read per token
A dual-socket server has a theoretical peak memory bandwidth of about 600 GB/s. If each token reads 20 GB of weights, the theoretical limit is about 30 TPS. In practice, NUMA access, CPU computation, and PCIe transfers all reduce speed, so real results are usually well below the theoretical limit. Measure performance on your own hardware.
- Cost: An estimated 150,000 to 300,000 RMB, depending mainly on memory capacity and quotes.
Several machines can form a distributed memory pool over RDMA to hold the model. However, cross-machine memory access latency is high, so this is not a good entry-level option.
Path 2: Single Server with 8 GPUs (Balance of Speed and Cost)
To reach 25 TPS for one user, the model should fit entirely in GPU memory. The experts also need to exchange data over a high-speed interconnect.
- Memory capacity: Eight accelerators with 288 GB each provide about 2304 GB in total. That leaves an average of about 186 GB of weights per card and about 100 GB of headroom for the KV cache and workspace.
- Configurations that will not work: Eight 192 GB cards provide 1536 GB in total. That leaves only about 6 GB of headroom per card after the weights, which cannot run stably. Sixteen 96 GB cards have the same 1536 GB total and the same problem. If only 192 GB cards are available, you need 12 or more, or you need to span two servers. Cross-node communication lowers speed.
- Interconnect: Each MoE layer typically requires two All-to-All communications, one for dispatch and one for combine. If the cards connect only through PCIe, communication latency cancels out the GPU compute advantage. These eight cards should form a single communication domain through NVLink or a similar technology.
- Expected performance: In this configuration, single-user output reaches 25 to 40 TPS. Report the test conditions with the result: 4-bit weights, 1024 input tokens, 512 output tokens, and batch size 1. Also record the inference framework version and CUDA version, and report P50 speed.
- Cost: Prices for an 8-GPU server vary widely by hardware generation, roughly in the million-RMB range. Confirm with quotes.
Path 3: Multi-Node Cluster (Supports 10 Concurrent Users)
Concurrency is fundamentally a throughput problem.
- Total throughput: Concurrent users × per-user speed = 10 × 25 = 250 TPS.
- Per-step latency limit: In batched decoding, each step generates one token for every request. To keep each user at 25 TPS, each step must take no more than 1000 ÷ 25 = 40 ms. Measure this at your target context length.
- Replica count: Replicas = ⌈10 ÷ the number of concurrent users one replica can serve stably⌉. For example, if a single 8-GPU replica keeps each step under 40 ms at batch size 4, you need ⌈10 ÷ 4⌉ = 3 replicas. Add one redundant replica for a total of 4 replicas, or 32 GPUs.
- Parallelism: Use expert parallelism for the experts and tensor parallelism for the shared layers. Cross-node RDMA bandwidth and latency directly limit the usable concurrency, so run load tests before launch.
- Cost: Multiple 8-GPU servers, plus switches, storage, racks, and power upgrades, add up to several million RMB.
4. Reducing Memory Use and Cost
- Quantization. 4-bit is the current common baseline. 2-bit quantization noticeably increases perplexity. Avoid it unless the model was trained with low-bit quantization-aware training.
KV cache optimization. Use paged memory management to reduce memory fragmentation. The KV cache size can be estimated with:
KV bytes per token = 2 × layers × KV heads × head dimension × bytes per element
This formula applies to standard multi-head or grouped-query attention. For compressed structures such as MLA, use the formula for that architecture.
Example: Assume the model has 61 layers, 8 KV heads, and a head dimension of 128, stored in FP8. Each token uses about 122 KB. At a 32K context with 10 concurrent users, the KV cache needs about 41 GB.
Separate prefill and decode. Prefill, which processes the long input prompt, is compute-heavy. Decoding is memory-bandwidth-heavy. Running them on separate deployments reduces interference between the two. The gain depends on the workload, so measure it with your own traffic rather than applying a fixed ratio.
5. Budget Reference
- Runs-only setup: A server with 2 TB or more of memory and one 24 GB GPU, estimated at 150,000 to 300,000 RMB. Speed must be measured. This setup suits functional validation, not interactive use.
- Single user at 25 TPS: One 8-GPU server with at least 288 GB per GPU, roughly in the million-RMB range.
- 10 concurrent users at 25 TPS: Determine the node count with the replica formula. A common configuration is four 8-GPU replicas, costing in the millions of RMB.
All figures are estimates based on public architecture details and stated assumptions. They are not measured results. Before formal deployment, rehearse the full process with a public model of similar scale. After the official weights are released, calibrate the configuration with benchmarks.
