Local Deployment Cost Estimation for Ultra-Large MoE Models
This article provides an engineering cost estimate for locally deploying ultra-large sparse Mixture-of-Experts (MoE) models with trillions of total parameters. It explains why storage must be sized by total parameters while compute and bandwidth follow activated parameters, estimates weight and KV-cache footprints, compares three deployment tiers from “can run” to 10-concurrency 25-tps service, and presents a parameterized CapEx/OpEx/TCO model plus a deployment checklist. All figures are estimates based on public architecture information and must be verified against released weights and benchmarks.
Local Deployment Cost Estimation for Ultra-Large MoE Models
This article discusses local deployment of ultra-large models with trillions of total parameters that use a sparse Mixture-of-Experts (MoE) architecture. One boundary must be stated first: as of public information, the complete weights and technical reports for such models may not yet be fully released. The speed, storage, hardware configuration, and cost figures below are engineering estimates based on public architecture information; after weights are released, they must be verified item by item with benchmarks. This article does not recommend any specific vendor product. It discusses general technical paths and cost models only.
1. Start With the Storage Budget: Total Parameters Set the Lower Bound; Activation Ratio Only Affects Compute
The most common pitfall in MoE deployment is mistaking the “activated expert ratio” for a storage ratio. Sparse activation reduces the computation per token, but it does not reduce weight storage. During inference, all experts must be addressable; after routing hits, the corresponding experts are fetched. Therefore, GPU memory and host memory planning must be based on total parameters, while compute and bandwidth planning are based on activated parameters.
Take an example scale: 2.8 trillion total parameters, 896 experts, 16 experts activated per token, and an expert activation ratio of about 1.79%. If weights use a 4-bit microscaling floating-point format, with one 8-bit scaling factor for every 32 4-bit elements, the effective bits per element are about 4 + 8/32 = 4.25 bit. The bare weight lower bound is:
total_params = 2.8e12bits_per_elem = 4.25weight_gb = total_params * bits_per_elem / 8 / 1e9print(weight_gb) # about 1487.5 GB
This does not yet include non-4-bit tensors, metadata, alignment padding, runtime workspace, communication buffers, and KV cache. In engineering practice, reserve 1.2–1.5× the bare weight footprint as addressable storage, i.e., roughly 1.8–2.3 TB as a starting point. As concurrency and context length increase, KV cache will keep raising the upper bound. For reference, a sparse model around 700B parameters is on the order of 700 GB with 8-bit weights; with 4-bit weights plus activations, roughly 400 GB is needed for relatively comfortable operation. A 2.8-trillion-parameter model is several times larger, which determines the lower bound for a full deployment.
KV cache can be estimated with a general formula:
kv_gb = 2 * layers * kv_heads * head_dim * seq_len * batch * bytes_per_elem / 1e9
Actual estimates must also include paged fragmentation, prefix caching, and scheduling headroom. The conclusion is: first ensure nearly 1.5 TB of weights fit into addressable storage; then discuss throughput.
2. Hardware Paths and Cost Ranges for Three Target Tiers
Tier 1: It Only Needs to Run; Speed Is Not a Concern
The goal is to make the model emit tokens, even if only single-digit tokens per second. The most direct path is heterogeneous GPU-CPU inference: attention, routing, and a small amount of dense computation stay on accelerators, while sparse expert layers—which account for most storage but activate only a few experts at a time—are offloaded to large CPU memory, with high-memory-bandwidth servers feeding the GPU. The key to this approach is not the number of GPUs, but addressable memory capacity, CPU memory bandwidth, and inter-GPU and inter-host data movement efficiency.
For configuration, addressable memory should be no less than 2 TB, accelerators should carry attention and hot experts, and local high-speed disks or general object storage should store weight shards. The cost range is typically tens of thousands to low hundreds of thousands of RMB, depending on memory capacity, accelerator count, and used versus new hardware choices. The main risks are very slow single-stream performance due to insufficient CPU memory bandwidth, expert swap-in/swap-out jitter, and immature quantization kernels.
Tier 2: Single-User Output Speed Reaches 25 tps
A single stream at 25 tps means a complete forward pass must be completed every 40 ms during decoding. The MoE decoding bottleneck is often not peak compute, but weight read bandwidth, expert routing all-to-all communication, and kernel launch overhead. If experts are distributed across multiple GPUs, each token must move activations to the GPU hosting the target expert and then move them back; communication latency directly consumes compute.
For this tier, it is not advisable to rely only on a single machine with large memory. Experts should be sharded across multiple accelerators or a high-bandwidth memory pool, and routing communication should stay within the same high-speed interconnect domain. The specific number of cards depends on per-card memory, interconnect bandwidth, quantization precision, and kernel efficiency; it cannot be decided solely from parameter count. The cost range typically enters hundreds of thousands of RMB, mainly consisting of accelerators, high-speed switching, memory, and power and cooling. Before launch, a single-stream stress test must be run: record time to first token, decode tps, expert hit distribution, all-to-all time, and KV cache usage.
Tier 3: 10 Concurrent Users, 25+ tps per User, and Quantization Level No Lower Than an Agreed Threshold
10 concurrent users × 25 tps equals at least 250 tps total throughput, while single-stream experience must not collapse. This requires continuous batching, paged KV cache, expert parallelism, and load balancing. The communication domain must be large enough to avoid cross-domain transfers. For models of a similar scale, industry practice often suggests dozens of accelerators or more forming a supernode; an example magnitude is 64 or more. This is not a hard formula, but it shows that the communication domain and total GPU memory capacity must both be satisfied.
The cost range is typically millions of RMB, and accelerators are only part of it. High-speed switching, optical modules, power supply, cooling, datacenter facilities, and operations all significantly raise total cost of ownership. The main risks are uneven expert load, all-to-all congestion, increased tail latency caused by batching, and quality regression caused by insufficient quantization precision.
3. Cost Model: Break “How Much Does It Cost” Into a Formula You Can Fill In
Do not ask directly “how much does one setup cost.” Instead, build a parameterized model:
CapEx = accelerator count × unit price + memory capacity × unit price + CPU/motherboard/power supply/chassis+ high-speed switching and cables + local storage + datacenter modificationOpEx = system power × electricity price × operating hours + bandwidth/storage/operations laborTCO = CapEx + OpEx × depreciation years
The three tiers compare as follows:
| Tier | Target | Key configuration | Cost range | Main risks |
|---|---|---|---|---|
| Tier 1 | Can run, single-digit tps | 2 TB-class memory + a few accelerators + heterogeneous inference | Tens of thousands to low hundreds of thousands of RMB | Memory bandwidth, swap-in/swap-out jitter |
| Tier 2 | Single-stream 25 tps | Multi-GPU expert sharding + high-speed interconnect + quantization kernels | Hundreds of thousands of RMB | Routing communication, decode bandwidth |
| Tier 3 | 10 concurrency × 25+ tps | Dozens of GPUs in a supernode + continuous batching + paged KV | Millions of RMB | All-to-all congestion, tail latency |
For initial validation, teams can request large-memory and accelerator resources on demand in a general-purpose cloud environment, store weight shards in object storage, orchestrate inference services with a container platform, and collect latency and memory metrics through monitoring and alerting. General-purpose cloud platforms such as Baidu AI Cloud can also serve as short-term validation environments, but the final decision should compare on-demand cloud costs with self-built TCO.
4. Implementation Checklist
- Weight sharding and verification: split by expert and layer, record hashes and versions, and avoid loading wrong shards.
- Quantization calibration: confirm weight and activation precision thresholds, and keep a fallback plan for high-precision hot layers.
- Expert parallelism: measure routing hit distribution, replicate hot experts, and offload cold experts.
- Communication optimization: keep all-to-all within a single high-speed interconnect domain where possible, and stress test tail latency under different batch sizes.
- Scheduling strategy: continuous batching, paged KV cache, prefix caching, and timeout circuit breaking.
- Stress-test metrics: time to first token, single-stream tps, total throughput, P99 latency, GPU memory and host memory watermarks, and communication time.
- Cost review: rerun benchmarks after weights are released, and replace estimated parameters with measured data.
In summary, local deployment cost for such models starts from the storage lower bound set by total parameters, is shaped in experience by the communication domain and decode bandwidth, and is capped by concurrency and precision thresholds. Tier 1 can start with heterogeneous inference at tens of thousands to low hundreds of thousands of RMB; Tier 2 usually enters hundreds of thousands of RMB; Tier 3, at 10 concurrency and 25+ tps, is more engineering-realistic with dozens of accelerators in a supernode and millions-of-RMB TCO. Before purchasing, verify every item against real weights and benchmarks.
