Cost Estimation for Local Deployment of Trillion-Parameter MoE Models
This article is intended for teams planning to deploy very large Mixture-of-Experts (MoE) models in their own data centers. It breaks down three targets—"can run," "25 tokens/s for a single user," and "ten concurrent users"—and explains the calculation methods for weight storage, expert offloading, super-node interconnect, and electricity costs. It presents VRAM lower bounds for 4-bit and 8-bit precision, GPU counts and cost examples for the three options, and how to verify the numbers with benchmarks. Readers can apply the formulas directly to estimate their own budgets.
Decision Recommendations
The analysis covers 4-bit and 8-bit precision lower bounds; the concurrency row in the table below uses 8-bit precision.
| Target | Recommended approach | GPU count or hardware | Main limitation |
|---|---|---|---|
| Can run; speed unrestricted | Heterogeneous inference, with experts placed in host memory | A 2 TB memory server plus 1 GPU | Decoding is limited by memory bandwidth; suitable only for offline tasks |
| 25 tokens/s for a single user | Experts resident in VRAM, multi-GPU parallelism | 32 × 80 GB accelerators with high-speed interconnect | Communication latency determines speed; requires super-node-level interconnect |
| Ten concurrent users, 25 tokens/s each, 8-bit precision | Full weights resident, batch scheduling | 64 × 80 GB accelerators | VRAM headroom and KV cache growth are the main risks |
Selection can proceed in the following order:
- First load the model using option one to confirm output quality and file integrity. This step has the lowest cost and also verifies whether the model format matches the inference framework.
- Measure single-user decoding speed on the target hardware. If it does not reach 25 tokens/s, first investigate communication latency before considering adding more GPUs.
- After clarifying the concurrency level and context length, determine the KV cache budget and decide whether to choose option three.
Before purchasing, verify the following:
- The officially released weight file size and the actual precision distribution of each type of tensor.
- Whether the target inference framework supports the model’s expert parallelism and quantization formats.
- Whether the data center’s power supply, cooling, and network can meet the super-node design requirements.
- Whether the benchmark workload is close to real business traffic, such as input/output length, concurrency pattern, and request distribution.
The figures for VRAM, GPU count, and electricity cost in this article are derived from formulas; unit prices and energy-efficiency coefficients are assumed values. Before formal project approval, replace these figures with official release data and measured results, and retain some budget headroom.
