Trillion-Parameter MoE Model Local Deployment Costs: Hardware, Bandwidth, and TCO
This article breaks down the practical cost of deploying a roughly 2.8-trillion-parameter MoE model locally across three tiers: basic bring-up, single-user generation at 25 tokens per second, and 10 concurrent users at 25 tokens per second each. It explains why expert activation is not the same as parameter activation, why VRAM must hold all experts, and why decode performance is mainly limited by memory bandwidth and expert placement. It also covers MXFP4 weight sizing, heterogeneous CPU/GPU designs, accelerator and interconnect requirements, electricity costs, break-even analysis, and the trade-offs between self-hosting and cloud inference.
Notes on sources and scope
As of this writing, the full weights and technical report for a 2.8-trillion-parameter MoE model, referred to below as “the model,” have not yet been officially released. The architecture and precision information in this article come from publicly available technical materials. Speed, VRAM, hardware, and price figures are estimates based on public information; they are neither official guarantees nor procurement quotes. Once the weights are released, each item should be re-verified with benchmarks before making purchasing decisions.
1. Cost map: hardware thresholds for three practical tiers
| Tier | Main bottleneck | Typical configuration | Weight precision | Estimated hardware cost, excluding data center and operations |
|---|---|---|---|---|
| 1: Runs, but speed is not a priority | CPU memory capacity | Dual-socket server + ≥2 TB memory + 1–2 24 GB-class GPUs | 4-bit | About RMB 160,000–350,000 |
| 2: Single user ≥25 tps | Memory bandwidth and expert residency strategy | Dual-socket high-bandwidth server + 2–8 24 GB-class GPUs, with hot experts kept resident in VRAM | 4-bit | About RMB 250,000–600,000 |
| 3: 10 concurrent users, each ≥25 tps | Accelerator VRAM capacity and inter-GPU interconnect | 24 × 80 GB-class or 32 × 96 GB-class accelerators, 8-GPU servers + high-speed network | At least 4-bit; FP8 roughly doubles requirements | About RMB 4,000,000–8,000,000 |
| Reference: official supernode recommendation | Large-scale MoE communication domain | 64 or more accelerators in a single communication domain | 4-bit | Tens of millions of RMB and up |
Key points:
- Tiers 1 and 2 use a heterogeneous inference approach. They allow the model to run at a much smaller scale than full deployment, but require accepting trade-offs in speed, concurrency, and quantization quality.
- Tier 3 is the closest to “near-full” deployment. The bottleneck shifts from memory capacity to accelerator VRAM capacity and inter-GPU interconnect, causing costs to jump by one to two orders of magnitude.
- Prices are rough market-range estimates. Actual costs depend on the purchase timing and supplier.
2. Three concepts that determine the cost structure
2.1 Expert activation ratio is not the same as parameter activation ratio
The model uses a sparse mixture-of-experts architecture with 896 experts. Each token is routed to only 16 experts. That gives an expert activation ratio of:
16 / 896 ≈ 1.79%
If all 896 experts were the same size, the parameter activation ratio would also be about 1.79%, or roughly 50B parameters per token. In practice, two factors make the actual activated parameter count slightly different:
- The model also contains shared attention and other per-layer structures. These parameters are computed for every token and are not sparse.
- If a small number of shared experts are always active, they further increase the actual activated amount.
Therefore, 1.79% should be treated only as a lower-bound reference for compute estimation, not as a strict parameter activation ratio.
2.2 VRAM is based on total parameters; compute is based on activated parameters
This is one of the most overlooked points in MoE deployment: the activation ratio affects how much compute each token consumes, but it does not reduce the amount of weights that must be resident.
During inference, the router first decides which experts to use. Activations are then sent to the devices holding those experts for computation, and results are returned. As long as a token might be routed to an expert, that expert must be stored somewhere addressable. Therefore, even if only 1.79% of experts are used per token, all 896 experts must still be resident.
2.3 A useful approximation: decode is like a small matrix multiplication
In the decode phase, the compute required per token is roughly:
2 × activated parameters
With about 50B activated parameters, one token requires about 0.1 TFLOP, and 25 tps requires about 2.5 TFLOPS. For any modern accelerator, this compute demand is trivial.
The real decode bottlenecks are therefore:
- Every generated token must read the full weights corresponding to the 16 selected experts.
- Those experts may be spread across multiple devices, requiring cross-device transfers.
Once this is clear, the differences among the three tiers become straightforward: they are fundamentally trade-offs between how many weight bytes must be moved per token and how quickly those bytes can be moved.
3. How large are the weights: from raw minimum to real deployment requirements
3.1 Exact math for the 4-bit micro-scaling format
The model introduces quantization-aware training from the SFT stage: weights use 4-bit micro-scaling floating point in MXFP4 format, and activations use 8-bit micro-scaling floating point in MXFP8 format. This means the native 4-bit weights can be loaded directly for local inference, without self-quantization, and the precision loss is more controllable.
Using only 4-bit weights, the raw data lower bound for 2.8T parameters is:
2.8T × 4 bit ÷ 8 = 1.4 TB
However, MXFP4 is not pure 4-bit. Under the format specification, every 32 weight elements require an additional 8-bit shared exponent, E8M0, as a scaling factor. This adds 0.25 bit per weight element. Including that:
2.8T × 4.25 bit ÷ 8 ≈ 1487.5 GB, or about 1390 GiB
The following are not included in that figure:
- Tensors that are not quantized to 4-bit, such as some normalization parameters and routing/gating layers, which usually remain at higher precision;
- Metadata and alignment overhead;
- Runtime workspace;
- KV cache.
For engineering purposes, a 15%–25% buffer should be added on top of the 1,490 GB figure. In other words, the practical deployment requirement for weights plus runtime overhead is around 1.7–1.9 TB.
3.2 Comparison with a 671B-class MoE model
A common reference point in the industry is a 671B-class MoE model, with about 671B main parameters and about 14B additional multi-token prediction parameters:
- At FP8 precision, weights are about 685 GB.
- With 4-bit quantization, weights are about 335 GB. Adding about 37 GB for activations and other overhead gives about 400 GB.
- Eight 80 GB accelerators provide 640 GB of VRAM, which is just enough after runtime overhead.
The model discussed here is more than four times larger. That is the fundamental reason why full-scale deployment has such a high threshold.
4. Tier 1: just getting it to run
4.1 Approach: GPU/CPU heterogeneous offloading
The most direct approach is heterogeneous inference, for which mature industry solutions already exist. The division of labor is:
- The shared attention portion in each layer, such as KDA and Gated MLA in this model, is compute-intensive and stays on the GPU, using high-bandwidth VRAM.
- The 896 experts dominate storage but only 16 are activated per token. They can all be offloaded to CPU memory and hosted by a large-memory server.
In this setup, expert weights remain in memory. When an expert is selected, its weights are transferred through the standard bus to the GPU for computation. With 4-bit weights, about 1.5 GB per expert, and 16 experts hit per token, each token requires about 23–25 GB of cross-bus transfer. Decode speed is therefore mainly determined by memory and bus bandwidth.
4.2 Configuration and cost
- Server: dual-socket processors, with as many memory channels as possible, plus at least 2 TB of memory for 1.5 TB of weights, KV cache, and page cache.
- Accelerators: 1–2 24 GB-class GPUs for the attention portion and a small number of resident hot experts.
- Local storage: 2 TB or more of NVMe SSD for fast weight loading.
- Cost: about RMB 160,000–350,000. Used or previous-generation GPUs may bring this down to around RMB 100,000.
4.3 Expected performance
Public industry practice provides a useful reference: a dual-socket server-grade CPU system with a single high-end consumer GPU running a 4-bit version of a 671B-class MoE model, with weights around 377 GB, achieved more than 200 tokens/s in prefill and about 14 tokens/s in decode, with an expert-selection policy in place.
Applying that to this model:
- Total weight size is about four times larger.
- Per-token expert data transferred from memory during decode, about 24 GB, is roughly 2.6 times the earlier case of about 9 GB.
- Decode speed is expected to fall into the single-digit to low-teens tokens/s range.
Therefore, Tier 1 is best positioned as functional validation: verifying model usability, API compatibility, and business workflow integration, rather than serving end users in production.
5. Tier 2: single-user output of at least 25 tps
5.1 Quantitative threshold
The effective weight-read bandwidth required per user during decode can be approximated as:
Required bandwidth ≈ activated parameters × bytes per parameter × output speed
Using 50B activated parameters, 4-bit weights, and 25 tps:
50B × 0.5 B × 25 ≈ 625 GB/s, or about 583 GiB/s
The following example code can be used to estimate thresholds under different quantization precisions:
# Example code: estimate weight-read bandwidth and compute requirements at different quantization precisionsGiB = 1024 ** 3TOTAL_PARAMS = 2.8e12 # total parametersEXPERTS, TOP_K = 896, 16def bandwidth_gibps(tps, concurrency=1, bits=4):"""Effective weight-read bandwidth required during decode (GiB/s)"""active = TOTAL_PARAMS * TOP_K / EXPERTS # activated parameters per tokenreturn active * bits / 8 * tps * concurrency / GiBdef tflops(tps, concurrency=1):"""Decode compute lower bound (TFLOP/s); about 2 FLOPs per parameter"""active = TOTAL_PARAMS * TOP_K / EXPERTSreturn active * 2 * tps * concurrency / 1e12for bits in (4, 8):print(f"{bits}-bit, single user at 25 tps: {bandwidth_gibps(25, bits=bits):.0f} GiB/s")print(f"{bits}-bit, 10 concurrent × 25 tps: {bandwidth_gibps(25, 10, bits):.0f} GiB/s")print(f"10 concurrent × 25 tps compute lower bound: {tflops(25, 10):.0f} TFLOP/s")# 4-bit, single user at 25 tps: 583 GiB/s# 4-bit, 10 concurrent × 25 tps: 5832 GiB/s# 8-bit, single user at 25 tps: 1166 GiB/s# 8-bit, 10 concurrent × 25 tps: 11664 GiB/s# 10 concurrent × 25 tps compute lower bound: 25 TFLOP/s
5.2 Feasible path
High-end servers typically provide 8–12 DDR5 memory channels. High-end configurations offer about 460–600 GB/s of bandwidth, and dual-socket systems can reach about 1 TB/s in total. Although this theoretical value exceeds the threshold, general inference frameworks interleave weight reads, matrix multiplications, and result return during decode. Effective bandwidth is often only 40%–70% of the theoretical peak.
A feasible Tier 2 path is:
- Use 2–8 24 GB-class GPUs, providing 48–192 GB of VRAM, to keep the attention portion and some frequently selected experts resident in VRAM.
- Keep the remaining experts in memory so that most decode-step expert reads occur within VRAM.
- Reduce per-token overhead with mechanisms such as paged KV cache management and continuous batching.
One misconception should be avoided: increasing memory capacity alone does not improve decode speed, because speed depends on memory bandwidth. Once all memory channels are occupied, adding more DIMMs may only lower memory frequency.
5.3 Cost and limitations
- Hardware cost is about RMB 250,000–600,000.
- If FP8 activation computation is used, single-user 25 tps requires more than about 1.2 TB/s of effective bandwidth. A heterogeneous approach would need more GPUs to reach that level.
- At 10 concurrency, decode can be handled by a single machine because it requires only about 0.1 TFLOP per token. However, prefill touches almost all experts, and compute demand grows linearly with input length. GPU-to-memory transfer becomes a clear bottleneck, and first-token latency increases significantly with input length.
6. Tier 3: 10 concurrent users, each ≥25 tps, with quantization no lower than 4-bit
6.1 VRAM threshold
To serve 10 requests simultaneously while maintaining 25 tps per request, the most reliable approach is to keep all weights resident in accelerator VRAM and eliminate the “where are the experts” problem. VRAM demand can be estimated as weights plus runtime headroom:
- 4-bit: weights are about 1.49 TB, or about 1.7–1.9 TB including headroom.
- 80 GB-class accelerators: about 24 cards.
- 96 GB-class accelerators: about 20 cards.
- 141 GB-class accelerators: about 14 cards.
- FP8, if quantization must be no lower than FP8: weights are about 2.99 TB, or about 3.4–3.7 TB including headroom. Accelerator count roughly doubles: about 48 80 GB-class cards or 30 141 GB-class cards.
Mainstream servers generally use an 8-GPU configuration. Twenty-four 80 GB cards correspond to three 8-GPU servers. Weights and inter-GPU communication must cross machines, requiring a high-speed network such as 400G-class RDMA NICs. This is also why costs jump from hundreds of thousands of RMB to millions.
6.2 Compute threshold
At 10 concurrency and 25 tps per user, total throughput is 250 tokens/s, corresponding to a decode compute lower bound of:
2 × 50B × 250 ≈ 25 TFLOPS
Twenty-four flagship accelerators can provide thousands of TFLOPS of FP8 dense compute. However, MoE routing introduces all-to-all communication, load imbalance, and kernel overhead, making actual utilization far below theoretical peak. Compute is therefore not the bottleneck; bandwidth, interconnect, and communication efficiency are.
6.3 How to define the quantization lower bound
If “quantization no lower than 4-bit” only means avoiding more aggressive compression than 4-bit, native MXFP4 weights can be used directly. If the requirement is no lower than FP8, VRAM and total system count must at least double, and cost roughly doubles as well. Whether that is worthwhile depends on the output-quality requirements of the business.
6.4 Why the official recommendation is a supernode of 64 or more accelerators
Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth and latency. The 896 experts are distributed across different devices. For every token at every MoE layer:
- The router decides the target experts.
- Activations are sent to the devices holding those experts.
- Results are collected after computation.
The number of layers multiplied by concurrency makes this transfer frequency extremely high, and communication latency can directly erase compute gains. A supernode builds a large communication domain through high-speed intra-machine interconnect or equivalent technology, keeping expert placement and token movement mostly within that domain.
6.5 Cost and output
- Hardware cost:
- 24 80 GB-class accelerators: about RMB 3,600,000–6,000,000; 96 GB/141 GB-class: about RMB 4,500,000–10,000,000.
- 3–6 8-GPU servers, including CPUs, memory, local disks, and power supplies: about RMB 900,000–2,400,000.
- High-speed network, including switches, NICs, and cables: about RMB 800,000–3,000,000.
- Total: about RMB 5,300,000–11,400,000, with a typical configuration around RMB 7,000,000–9,000,000.
- A 24 GB-class GPU design with about 40 cards is also in the tens-of-millions RMB range and adds networking complexity, making it unattractive.
- If the officially recommended 64-card supernode is used, hardware cost starts in the tens-of-millions RMB range.
The approximate annual output from this investment is:
- Assuming 365 days per year and 10 effective service hours per day, annual output is about 3.29 billion tokens.
- If batching doubles utilization, annual output is about 6.58 billion tokens.
6.6 Do not forget prefill
Decode only touches the 16 activated experts. Prefill processes the entire input, and tokens at different positions are routed to many different experts. In long-input scenarios, such as analyzing documents with tens of thousands of words, prefill can touch most of the weights, putting far more pressure on VRAM bandwidth and communication than decode. Capacity planning should be based on total input-plus-output token volume and the actual input/output ratio, not output speed alone.
7. Self-hosting or cloud? Use TCO and break-even analysis
7.1 Electricity and data-center costs
An approximate electricity cost formula is:
Annual electricity cost ≈ system power (kW) × 24 × 365 × PUE × electricity price (RMB/kWh)
Assuming PUE = 1.3 and electricity price = RMB 0.8/kWh:
- A 10 kW-class single machine for a 4-bit Tier 3 build with 24 cards costs about RMB 91,000 per year.
- An 8-GPU server costs about RMB 73,000 per server per year.
- A 40 kW-class supernode costs about RMB 365,000 per year.
Self-hosted data centers must also account for racks, cooling, UPS, bandwidth, and operations staff, assuming about 0.5–1 engineer.
7.2 Break-even analysis: using Tier 3 as an example
| Item | Amount |
|---|---|
| One-time hardware investment | About RMB 8,000,000 |
| Straight-line depreciation over 3 years | About RMB 2,670,000 per year |
| Electricity and data center | About RMB 550,000 per year |
| Operations staff | About RMB 500,000 per year |
| Total | About RMB 3,720,000 per year |
- With annual output of about 3.29 billion tokens, the implied cost is roughly RMB 1,130 per million output tokens, or about RMB 1.1 per 1,000 tokens.
- Self-hosting has a cost advantage only if cloud inference prices for a comparable model remain consistently above this level and annual usage reaches tens of billions of tokens.
- If annual usage is only a few hundred million tokens, spreading the same hardware depreciation across each token can make the cost hundreds or thousands of times higher, often far above direct API usage.
This calculation also explains why the official recommendation points to a 64-card or larger supernode: it assumes large-scale, high-concurrency production service. Individuals and small teams trying to “run a trillion-parameter model locally” are essentially paying for a communication domain and VRAM pool they cannot fully utilize.
7.3 When to use local deployment and when to use the cloud
- If data cannot leave the internal network, compliance requirements are strict, and workloads are stable and high-load, self-hosting is suitable. However, Tier 1 and Tier 2 setups are better suited to internal tools and R&D validation.
- If workloads have peaks and valleys, or the team lacks GPU operations experience, on-demand cloud GPU instances, such as general GPU cloud hosts or containerized inference services, are more economical and avoid data-center and operations investment.
- Cloud GPU providers can be used for small-scale validation on demand before committing to self-hosting.
- For the most cost-sensitive scenarios with higher tolerance for output quality, direct API usage is often the best option.
Hidden self-hosting costs include procurement and delivery lead times of several days to several weeks, re-adaptation after model upgrades, and redundancy for GPU failures. If one GPU in an 8-GPU server fails, the entire server may become unusable.
8. Key levers for reducing cost
- Use native quantized formats. After weights are released, prioritize the official QAT-produced 4-bit version to avoid extra precision loss and conversion costs from self-quantization.
- Use tiered storage with hot experts resident in VRAM. Experts should be layered by routing frequency: high-frequency experts in VRAM, low-frequency experts in memory or SSD. This is the key to Tier 2.
- Control context length. KV cache grows linearly with context length. Limiting maximum context can save substantial VRAM headroom and reduce the number of cards required.
- Use batching reasonably. Continuous batching can improve GPU utilization, but higher concurrency increases VRAM and interconnect requirements. Balance speed and concurrency.
- Use dynamic scaling. Keep base capacity locally and use cloud instances for peak traffic to avoid purchasing for peak demand.
- Monitor model updates. If larger open-source models appear later, the formulas in this article can be reused for recalculation.
9. Conclusion
Returning to the original question: how much does it cost to deploy a trillion-parameter MoE model locally? The answer depends on which tier you choose:
- Just getting it to run: a dual-socket server with about 2 TB of memory and one or two 24 GB-class GPUs, costing roughly RMB 160,000–350,000.
- Single user at 25 tps: a heterogeneous setup costing about RMB 250,000–600,000, combined with hot-expert residency and bandwidth tuning.
- 10 concurrent users at 25 tps each, with quantization no lower than 4-bit: a cluster with about 24 80 GB-class or 32 96 GB-class accelerators, commonly costing RMB 7,000,000–9,000,000, with electricity and operations costs additional.
The real cost driver is not the absolute parameter count, but the hard constraint that all experts must reside in addressable storage. Whether self-hosting or using the cloud, the right approach is to first estimate VRAM, bandwidth, and throughput thresholds with the formulas above, then verify against official benchmark data, and finally run a break-even analysis based on real business workload. This avoids paying for communication domains and VRAM pools that are not actually used.
