Local Deployment of Trillion-Parameter MoE Models: Cost Estimates and a Three-Tier Plan
This article examines the storage, memory bandwidth, interconnect, scheduling, and TCO constraints of deploying trillion-parameter MoE models locally. It provides a capacity-planning method for 4-bit weights, KV cache, and expert-parallel communication, then outlines three deployment tiers—from a validation environment that merely runs to single-user 25 tokens/s and 10-concurrent-user production service—with order-of-magnitude cost ranges and pre-launch benchmark checks.
Local Deployment of Trillion-Parameter MoE Models: Cost Estimates and a Three-Tier Plan
Note: This article discusses a general technical problem—local deployment and cost estimation for very large MoE models. The parameters here are capacity-planning examples and do not correspond to any specific product. Before weights and implementation details are publicly available, all speed, VRAM, and cost figures should be treated as engineering estimates. Each item must be verified with benchmarks before launch.
1. Clarify the Problem: Local Deployment Is Not Just “Can One GPU Hold It?”
The core challenge of locally deploying a trillion-parameter MoE model is not a single compute metric. It is the combined constraint of storage capacity, memory bandwidth, inter-GPU interconnect, concurrency scheduling, and total cost of ownership. MoE architectures have a large total parameter count, but each token is routed to only a few experts. A common mistake is to assume that because the activation ratio is low, VRAM can also be sized by the activation ratio. That is incorrect. During deployment, all experts must reside in an accessible storage hierarchy, and at inference time tokens are routed to the devices hosting the corresponding experts. VRAM must be planned based on total parameters; the activation ratio mainly affects compute consumption and communication patterns.
For estimation, assume a set of example parameters: about 3 trillion total parameters, about 1,000 experts, 16 experts activated per token, 4-bit weights, and an 8-bit activation or scaling-factor quantization scheme. These are used only to illustrate the method and do not represent any specific model.
1.1 How to Calculate Weight Memory
With 4-bit quantization, the lower bound for bare weights is:
3e12 × 4 bit ÷ 8 ≈ 1500 GB
However, 4-bit weight quantization is usually not a clean 4 bits per parameter. If every 32 4-bit elements carry one 8-bit scaling factor, the average per weight element is about:
4 + 8/32 = 4.25 bit
The weight data is then approximately:
3e12 × 4.25 ÷ 8 ≈ 1594 GB
This does not include non-4-bit tensors, metadata, alignment padding, runtime workspace, communication buffers, or KV cache. For engineering purposes, the first version should be planned at more than 2 TB, rather than forced into a 1.5 TB card boundary.
1.2 KV Cache and Concurrency
KV cache depends on concurrency, context length, number of layers, KV heads, head dimension, and precision. It can be abstracted as:
kv_bytes = batch * seq_len * kv_heads * head_dim * layers * dtype_bytes * 2
For a single user with a short context, KV cache may not be large. But 10 concurrent users, long contexts, and continuous conversations can raise usage quickly. A safer approach is to reserve 15%–30% of total VRAM for KV cache and record peak usage during stress testing.
1.3 Interconnect Is More Critical Than Per-GPU Compute
MoE inference involves expert parallelism: experts are distributed across different accelerators. After routing, each token’s activations must be moved to the device hosting the target expert, computed, and moved back. This all-to-all communication is extremely sensitive to bandwidth and latency. If the interconnect domain is too small, communication latency directly consumes compute, manifesting as “GPU utilization is not high, but throughput does not increase.” Therefore, large-scale MoE deployment usually requires a high-bandwidth interconnect domain, keeping communication for frequently used experts within a single high-speed domain. For scenarios with 10+ concurrent users, interconnect design often affects actual tokens/s more than per-GPU peak compute.
2. Three Deployment Tiers and Configuration Logic
The following sections follow the three tiers from the original question. Costs are given only as engineering budget orders of magnitude, not specific quotes. Actual figures depend on vendor quotes and benchmarks.
Tier 1: Just Make It Run, Speed Not Important
Goal: the model can load, produce correct output, and complete a single request. Interactive experience is not required.
Configuration approach:
- Total VRAM + host memory ≥ 2.5 TB; local high-speed SSD ≥ 3 TB for weights, metadata, and swap space.
- Use tiered loading across “VRAM + host memory + local disk,” allowing expert weights to be swapped in and out as needed.
- Set batch size to 1, disable long context and high concurrency, and prioritize avoiding OOM.
- Quantization must be no lower than 4-bit to avoid further loss of control over precision and size.
- Sufficient host memory bandwidth and CPU cores are required; otherwise paging becomes the bottleneck.
If all weights are held on 80GB-class accelerators, weights alone require about 20–24 cards. In practice, workspace and KV cache must also be reserved, so 24–32 cards are usually needed for stability. At tens of thousands of CNY per accelerator, accelerators alone may reach hundreds of thousands to one million CNY. Adding hosts, memory, SSDs, power, and racks, the overall budget is usually in the hundreds of thousands to one million CNY range. If large amounts of host memory offload are used, the accelerator count can be reduced and the budget pushed down to the hundreds of thousands of CNY range, but speed may drop to seconds per token, making it suitable only for validation, offline tasks, or very low-frequency use.
Tier 2: Single-User Use at 25 tokens/s
Goal: single-user interaction is usable, with output speed stably above 25 tokens/s.
This tier cannot rely on frequent paging. 25 tokens/s means the decode phase must continuously fetch weights, perform routing, complete expert computation, and communicate. Any disk swap will cause noticeable stalls. It is recommended that all experts reside in VRAM, or at least that hot experts and high-frequency routing paths are pinned within the high-bandwidth domain.
Rough compute demand can be estimated as “2 × activated parameters × tokens/s,” but the actual bottleneck is usually memory bandwidth and communication. In the example, 16 of 1,000 experts are activated per token, giving about 48 billion activated parameters. The theoretical floating-point demand for 25 tokens/s is not extreme; the real difficulty is memory access efficiency at low batch and cross-card routing latency. Therefore, the focus of Tier 2 is:
- Enough accelerators to hold all weights with KV cache headroom, usually 20–32 80GB-class cards.
- High-bandwidth inter-card interconnect, avoiding low-speed cross-host networks for frequent expert communication.
- Continuous batching or lightweight batching to improve expert utilization without sacrificing single-user latency.
- Monitoring of routing hotspots to prevent a small number of experts from becoming a source of tail latency.
In cost terms, this tier is usually at the million-CNY level: accelerators, high-speed switching, hosts, and storage together raise the budget. If low-bandwidth multi-host networking is used blindly, even a sufficient number of cards may fail to reach 25 tokens/s.
Tier 3: 10 Concurrent Users, 25 tokens/s or More per User, Quantization No Lower Than 4-bit
Goal: under 10 concurrent requests, each user still receives more than 25 tokens/s of output, for total output throughput of about 250 tokens/s, with quantization no lower than 4-bit.
This is the production-grade threshold. As total throughput rises, KV cache, scheduling queues, expert routing conflicts, and all-to-all communication are all amplified. Industry experience suggests that large-scale inference for trillion-parameter MoE often requires 64 or more accelerators in a high-bandwidth interconnect domain so that expert-parallel communication stays within the domain or mostly within it. The exact number depends on expert distribution, interconnect topology, context length, and SLA. However, 10 concurrent users at 25 tokens/s should not be planned as a “single machine, multiple GPUs” problem; it should be planned as a “supernode/high-bandwidth domain” problem.
Recommended configuration logic:
- 64 or more accelerators, partitioned by expert parallelism and pipeline parallelism, prioritizing bandwidth within the communication domain.
- High-bandwidth, low-latency networking with a switching hierarchy that is as flat as possible.
- A scheduling layer that supports continuous batching, priority queues, timeout control, and degradation strategies.
- Monitoring of TTFT, TPOT, P99 latency, tokens/s, VRAM peak, interconnect bandwidth utilization, and power.
- Stress testing at 10 concurrent, 20 concurrent, and burst concurrency to confirm that per-user speed is not achieved by sacrificing other users.
The cost order of magnitude is usually several million to tens of millions CNY, including accelerators, high-speed switching, hosts, storage, data center power, cooling, and O&M. If redundancy and high availability are required, backup nodes and disaster recovery budget must be added.
3. Cost Components and Selection Advice
Total local deployment cost is not just accelerators. Typical TCO includes:
| Cost item | Description | Share tendency |
|---|---|---|
| Accelerators | Determine VRAM capacity and compute; the largest item | High |
| Hosts and memory | Support weight loading, KV cache, and CPU offload | Medium-high |
| High-speed interconnect and switching | Determine MoE all-to-all efficiency | High |
| Local storage | Weights, logs, datasets, and checkpoints | Medium |
| Data center power and cooling | Must be calculated in advance for high-power scenarios | Medium-high |
| Software adaptation and O&M | Inference framework adaptation, stress testing, monitoring, and upgrades | Medium |
For selection, build a capacity model first, then run a small-scale POC:
total_params = 3e12avg_bits = 4.25weight_tb = total_params * avg_bits / 8 / 1e12planning_tb = weight_tb * 1.3 # Reserve for metadata, workspace, and KV cacheprint(weight_tb, planning_tb)
If the planning result exceeds single-machine VRAM, do not buy cards first. Instead, determine the interconnect domain size and expert parallelism strategy. For budget-limited teams, general cloud GPU instances or container platforms can be used for a POC to validate 25 tokens/s and 10-concurrency metrics before deciding on local procurement scale. For long-term high-utilization scenarios, local deployment has advantages in controllability and marginal cost. For short-term validation and elastic demand, renting general compute is more prudent.
4. Metrics That Must Be Verified Before Launch
Regardless of tier, each item must be verified with benchmarks:
- Peak VRAM and host memory after weight loading;
- Single-user TTFT, TPOT, and stable tokens/s;
- Per-user tokens/s, total throughput, and P99 latency at 10 concurrency;
- Expert routing distribution and all-to-all bandwidth utilization;
- KV cache growth curve under long context;
- Post-quantization accuracy regression and abnormal output rate;
- Whole-system power, cooling, and data center load.
The conclusion is clear: the budget range for local deployment of trillion-parameter MoE models is extremely wide. To merely make it run, a validation environment can be built for hundreds of thousands to one million CNY. Single-user 25 tokens/s usually requires million-CNY-level investment. For 10 concurrent users at 25 tokens/s with quantization no lower than 4-bit, production planning should be at the several-million to tens-of-millions CNY level. What really determines success is not simply “whether there are enough cards,” but whether VRAM capacity, interconnect bandwidth, and benchmark data match the business SLA.
