Local Deployment of Trillion-Parameter MoE Models: Hardware Options and Cost Analysis
As open-weight models scale into the trillions of parameters, locally deploying a Mixture-of-Experts (MoE) model has become a serious capacity-planning and budgeting exercise. This article analyzes three deployment tiers—"just make it run," "smooth single-user inference," and "multi-user production serving"—explaining the hardware configurations and cost ranges for each. It covers how MoE memory requirements are determined by total parameters, how quantization formats affect storage, why interconnect bandwidth becomes the bottleneck at scale, and how to weigh self-hosting against cloud API costs using a simple TCO model.
Local Deployment of Trillion-Parameter MoE Models: Hardware Options and Cost Analysis
As open-weight models keep growing into the hundreds of billions and even trillions of parameters, “local deployment” of large MoE (Mixture of Experts) models has shifted from something you can try on a consumer GPU into a full engineering project that demands careful capacity planning and budget evaluation. Rather than targeting any specific vendor or product, this article takes a vendor-neutral technical perspective and examines three deployment tiers for a trillion-parameter-class MoE model: just make it run, smooth single-user inference, and multi-user production serving—along with the hardware and cost implications of each.
A caveat up front: all memory and throughput figures below are estimates derived from publicly available architectural information and are intended to demonstrate the estimation method. Actual numbers must be validated against benchmarks once model weights are officially released.
1. Do the Math First: How Much Storage Do the Weights Need?
1.1 MoE Memory Characteristics: Total Parameters Set the Floor
The defining feature of MoE architecture is “enormous total parameters, very few activated per token.” Take a typical configuration: roughly 2.8 trillion total parameters, hundreds of experts, with each token routed to only a small fraction of them. A common beginner mistake is assuming that “low activation ratio means low memory usage.”
Deployment memory is calculated from total parameters. At inference time, the router decides which experts handle a given token, but all expert weights must remain addressable at all times—you cannot predict where the next token will be routed. So even if the expert activation ratio is only around 1–2%, every expert’s weights must be loaded into storage. The activation ratio determines compute consumption and throughput ceilings, not the storage floor.
1.2 Adjusting for Quantization Formats
If the model was quantization-aware from training—for example, weights stored in a 4-bit floating-point format (block floating-point schemes similar to MXFP4)—a rough estimate looks like this:
- Raw data floor: 2.8T parameters × 4 bits ÷ 8 ≈ 1,400 GB
- Scale-factor overhead for block floating point: one 8-bit scale per 32 4-bit elements, which works out to roughly 4.25 bits per element under OCP-style specs: 2.8T × 4.25 ÷ 8 ≈ 1,487.5 GB
- Additional headroom needed: non-quantized tensors (embeddings, the router, and some normalization layers may stay in higher precision), metadata and alignment overhead, runtime workspace, and the KV cache, which grows linearly with context length
For reference: a 671B-parameter MoE model needs about 685 GB for weights in FP8; in AWQ 4-bit, roughly 335 GB for weights and about 400 GB including activations and overhead—fitting on eight 80 GB GPUs (640 GB total). A trillion-scale model is several times that size, which directly raises the hardware floor for full deployment.
Conclusion: the weights alone require roughly 1.5 TB of fast storage. This is the starting point for every plan.
2. Tier One: Just Make It Run, Speed Doesn’t Matter
2.1 Technical Approach: CPU-GPU Heterogeneous Inference
The goal here is singular: fit nearly 1.5 TB of weights into addressable storage and get the model to emit tokens, even if only a few per second.
The most direct approach is heterogeneous inference, a common industry practice:
- Keep the shared attention layers on the GPU: attention computation in each MoE layer (including linear attention or gated attention components) is compute-intensive and benefits from high GPU memory bandwidth.
- Offload all sparse experts to CPU memory: the hundreds of experts account for the bulk of storage, but only a handful are activated per token. A high-memory server (e.g., 512 GB–1 TB of DDR RAM) hosts them, fetched on demand based on routing decisions.
2.2 Hardware Configuration and Cost
| Component | Configuration | Estimated Cost |
|---|---|---|
| GPU | One 24 GB card (attention + scheduling) | A few thousand RMB |
| CPU memory | 1–1.5 TB (8–12 channel DDR server platform) | Tens of thousands of RMB |
| Other | Server chassis, storage | Tens of thousands of RMB |
The overall budget lands around 100,000 RMB. The trade-off: expert weights live in CPU memory, so throughput is entirely memory-bandwidth-bound, typically delivering single-digit tokens per second. This is only suitable for validating that the environment works and running functional tests—it has no practical serving value.
3. Tier Two: Single User at 25 tokens/s
3.1 Why 25 tokens/s?
From a reading-experience standpoint, comfortable human reading speed corresponds to roughly 20–30 tokens/s of generation. This is the empirical threshold for “usable by a single person.”
3.2 Compute Estimation
The effective compute per token in an MoE model ≈ total parameters × activation ratio, roughly 2.8T × 1.79% ≈ 50B equivalent parameters. With 4-bit weights, each token requires reading about 25 GB of weights (attention layers plus the 16 activated experts). Hitting 25 tokens/s demands weight-read bandwidth of roughly 25 GB × 25 ≈ 600+ GB/s, with additional headroom for attention compute and KV cache access.
This implies:
- A single consumer GPU (~1 TB/s bandwidth class) is theoretically close, but real-world scheduling, communication, and non-weight memory access eat into the budget—you’d be running at the absolute edge of the hardware.
- A safer approach is multi-GPU aggregation: 3–4 high-bandwidth professional GPUs providing 2 TB/s+ of aggregate bandwidth, with total VRAM covering the weights (~1.5 TB must be addressable, achievable through multi-GPU setups plus tiered caching).
3.3 Cost Range
Based on multiple 80 GB professional GPUs (each costing tens of thousands of RMB) plus the host system and interconnects, the budget falls in the 300,000–800,000 RMB range. If aggregated VRAM fully holds the weights, you avoid CPU paging bottlenecks entirely, with noticeably better speed stability and latency.
4. Tier Three: 10 Concurrent Users, Each Still at 25 tokens/s
4.1 The Compute Multiplier Effect of Concurrency
Concurrency is the cost inflection point. Ten concurrent requests mean the total tokens processed per second is roughly 10× that of a single user (continuous batching improves utilization, but total demand still scales up). Rough estimate:
- Per-token compute: ~50B equivalent parameters × 2 FLOPs ≈ 100 GFLOPs
- 10 concurrent × 25 tps = 250 tokens/s, requiring 25+ TFLOPS of effective compute; accounting for utilization losses (real-world MFU for large-scale MoE is typically not high), raw compute demand multiplies severalfold
- The more致命 bottleneck is communication: hundreds of experts are distributed across many GPUs, and each routed token’s activations must be moved to the GPU hosting the target expert, computed, then moved back. All-to-all communication directly consumes the compute gains.
4.2 Supernodes and High-Speed Interconnects
This is why official deployment recommendations for models of this class typically call for “supernodes” of dozens of accelerators: using NVLink or equivalent high-speed interconnects to enlarge the communication domain, so expert-to-expert data movement stays within a single domain and avoids cross-node network latency. An 8-GPU node has high internal interconnect bandwidth, but 1.5 TB of weights requires at least ~20 80 GB GPUs—so the quality of inter-GPU and inter-node connectivity directly determines concurrent performance.
4.3 Cost Range
Based on 24–32 80 GB professional accelerators across four 8-GPU nodes, plus high-speed switching and complete systems, the budget reaches several million RMB. At this point, most organizations should rationally compare against model API pricing: unless data privacy requirements are extremely strict or daily token consumption is enormous, the TCO of self-hosting (including power, facilities, and operations staff) rarely beats pay-as-you-go cloud inference. A simple quantitative model—“daily token volume × API unit price vs. hardware depreciation + electricity + operations”—works well for this decision.
5. Summary and Recommendations
| Tier | Requirement | Core Approach | Cost Range |
|---|---|---|---|
| Just run it | Speed irrelevant | Single GPU + high-memory server, heterogeneous inference | ~100,000 RMB |
| Smooth single user | ≥25 tokens/s | Multi-GPU aggregate bandwidth + VRAM expansion | 300,000–800,000 RMB |
| Production concurrency | 10 concurrent × 25 tokens/s | Dozens of GPUs in supernodes + high-speed interconnect | Several million RMB |
Practical recommendations:
- Validate before purchasing: all figures before weights are released are estimates. Validate business value via the API first, and only invest in hardware after confirming that privatization is worthwhile.
- Size memory by total parameters, throughput by activated parameters—these are the two fundamental axioms of MoE capacity planning.
- Never skip the KV cache budget: under concurrency, KV cache grows linearly with context length and concurrent requests. Reserve an extra 20–30% of VRAM for the maximum context scenario.
- Weigh self-hosting against the cloud: absent hard data-compliance constraints, mature cloud inference services (major cloud platforms offer LLM inference and private deployment options) typically have the edge in elasticity and operational cost for small-to-mid-scale usage.
Locally deploying a trillion-parameter MoE model is a systems engineering effort. Only by calculating all four dimensions—storage, bandwidth, interconnect, and concurrency—can you make sure every dollar of hardware budget is spent where it counts.
