What It Actually Costs to Deploy a Trillion-Parameter MoE Model Locally

1. The Short Answer: This Is a Storage and Interconnect Problem, Not a Compute Problem

Many teams see “trillion parameters” and immediately assume they don’t have enough compute. But for very large MoE (mixture-of-experts) models, the first hurdle in local deployment is actually two things: fitting all experts into addressable storage, and keeping the cross-GPU communication triggered by token routing from stalling computation. Compute determines how fast you run; storage determines whether you can run at all; interconnect determines whether your cluster budget is wasted.

The following analysis uses an open-source MoE model with roughly 2.8 trillion total parameters (referred to below as “the target model”) as the example. One caveat up front: apart from architecture and public deployment guidance, all speed, VRAM, and hardware configuration figures in this article are order-of-magnitude estimates based on public information. Once the final weights are released, each of these should be verified item by item with benchmarks.

2. Breaking Down Model Size: Total Parameters, Experts, and Activation Ratio

Start with the key numbers:

• Total parameters: roughly 2.8 trillion;
• Architecture: MoE with 896 experts, 16 activated per token, giving an expert activation ratio of about 1.79% (16/896);
• Backbone: a hybrid attention design—gated linear attention combined with gated KV-compressed attention, plus attention residual connections.

Two easily confused points here directly determine whether your resource planning is correct:

First, the expert activation ratio is not the same as a strict parameter activation ratio. The embedding layer, attention components, and shared modules participate in computation at every layer, expert parameter counts may be uneven, and routing distributions are naturally imbalanced. So the share of parameters actually involved in computation is noticeably higher than 1.79%; the exact figure should come from the official technical report.

Second, MoE VRAM usage is calculated on total parameters; the activation ratio only affects compute consumption. During inference you cannot predict which experts the next token will route to, so all 896 experts must reside in storage at all times, ready to serve. A low activation ratio only means fewer floating-point operations per unit time—i.e., “cheaper to compute”—and does not change the “can it fit” threshold. This is the first principle of MoE deployment cost estimation.

One favorable factor for deployment: the linear attention backbone can significantly compress KV cache overhead in long-context scenarios, easing VRAM pressure for long-text inference. But sparse experts remain the bulk of storage usage, and that conclusion holds.

3. Quantization and the VRAM Floor: How 1500GB Is Calculated

The target model underwent quantization-aware training starting from the SFT stage, using 4-bit microscaling floating-point weights with 8-bit activations. In other words, 4-bit deployment is the “within-official-design” path—precision loss was optimized during training rather than patched afterward with post-training quantization.

The VRAM floor is calculated as follows:

TOTAL = 2.8e12  # total parameters
# 1) Theoretical floor for pure 4-bit (decimal GB)
raw_4bit = TOTAL * 4 / 8 / 1e9            # ≈ 1400 GB
# 2) Add microscaling format overhead: one 8-bit scale shared per 32 4-bit elements
effective_bits = 4 + 8 / 32               # = 4.25 bits/parameter
weight_mx = TOTAL * effective_bits / 8 / 1e9   # ≈ 1487.5 GB
# 3) Engineering headroom: non-4-bit tensors, metadata, alignment, runtime workspace, KV cache
budget = weight_mx * 1.15                 # plan to start from 1.7 TB

Note the second step: the microscaling format is not “pure 4-bit.” Every 32 4-bit elements carry an 8-bit shared scaling factor, giving an effective bit width of 4.25 bits, so the weights alone come to nearly 1487.5GB. On top of that you must add: non-4-bit tensors such as the embedding layer that typically stay at higher precision, metadata and alignment overhead, inference runtime workspace, and KV cache that grows with context length and concurrency. A 10%–20% margin is safer for actual planning.

For a cross-reference: a flagship open-source MoE model with 671B total parameters has about 685GB of main parameters plus roughly 14B of multi-token prediction parameters in FP8; in 4-bit weights that is about 335GB, plus about 37GB of activation overhead, for roughly 400GB total—just enough to run on a single machine with 8 80GB-class accelerators (640GB of VRAM combined). The target model is more than 4× that size—a full 4-bit deployment floor of close to 1.5TB does not fit in an 8-GPU single server and must span machines. By capacity alone, 1.5TB of weights needs at least 19 80GB-class accelerators, and that’s before KV cache and workspace.

4. Why the Official Guidance Recommends a 64-GPU Supernode: All-to-All Is the Hidden Bottleneck

The model’s official deployment guidance recommends a supernode of 64 or more accelerators. The reason is not capacity—it’s communication.

Large-scale MoE inference generally uses expert parallelism: 896 experts are distributed across many GPUs, and after each token is routed, its activations must be dispatched to the GPUs hosting the target experts and then combined back after computation. This is classic all-to-all communication, with volume proportional to token count, activated expert count, and hidden dimension. Once concurrency rises, insufficient inter-GPU bandwidth leaves GPUs idling on communication and compute utilization collapses—you pay for expensive cards but the money goes to waiting.

The supernode approach uses proprietary high-speed inter-GPU interconnects or equivalent solutions to bring a larger communication domain into the same high-bandwidth fabric, so dispatch/combine does not have to cross slow networks. This also directly determines that the starting point for “full deployment” is not one server but a cluster.

5. The Low-Cost Path: CPU+GPU Heterogeneous Inference

If the goal is simply “make the model emit tokens, even if only single-digit tokens per second,” heterogeneous inference is the most cost-effective approach. The core idea is to split by compute density:

• Keep attention on the GPU: it is compute-intensive, has few parameters, and needs high bandwidth;
• Offload the 896 sparse experts to CPU memory: they account for the bulk of parameter storage, but only 16 are activated per layer, so a large-memory server can host them and fetch computation on demand.

Public benchmark data from open-source heterogeneous solutions offers a reference point: a dual-socket server-grade CPU paired with a single 24GB consumer flagship GPU, running a 4-bit quantized version of a 671B-class model (about 377GB of weights), reached over 200 tokens/s prefill and about 14 tokens/s decode (that result came with constraints on the expert selection strategy).

Extrapolating to the target model requires discounts: weight volume is 4× the reference, CPU memory needs to start at 1.6TB, decode speed will most likely fall to single-digit tokens/s, and long-context prefill will slow noticeably. Several boundaries should also be recognized: it is basically only suitable for single or very low concurrency; expert caching and selection strategy heavily affect speed; and the inference framework must support MoE expert offloading kernels. It suits personal research validation and offline batch processing, not online serving.

6. Cost Breakdown for the Three Paths (Order-of-Magnitude Estimates)

The following are order-of-magnitude estimates. Hardware prices fluctuate noticeably with model, channel, and supply-demand, so real-time quotes should govern.

Path A: Full supernode deployment

• 64+ 80GB-class data center accelerators: accelerators alone typically reach the multi-million RMB range;
• Supporting hosts, 64-port 400G-class high-speed lossless networking and optical modules, inter-GPU high-speed interconnect;
• Storage: nearly 1.5TB of weights, all-flash NVMe with redundancy recommended;
• Power and cooling: rack power can reach tens of kilowatts;
• Total investment typically lands in the tens of millions of RMB range, plus electricity, operations staffing, and depreciation from 3–5 year hardware refresh cycles.

Path B: CPU+GPU heterogeneous single server

• Dual-socket server platform with 1.5–2TB of memory (memory often accounts for about half the machine cost);
• 1–2 24GB-class flagship GPUs, plus 2TB+ NVMe;
• The whole machine is roughly in the hundreds of thousands of RMB range;
• The trade-off is single-digit tokens/s decode and very weak concurrency.

Path C: Pay-as-you-go cloud usage

• No fixed hardware investment, billed by usage, no need to manage drivers, quantization kernels, or cluster operations;
• Suitable for moderate usage, bursty traffic, and teams that don’t want to staff ops. For example, you can use a cloud provider’s hosted large-model service (such as the model services offered by Baidu AI Cloud) and pay per token;
• If usage stays high long term, compare cumulative API costs against self-built depreciation plus operating costs before deciding.

7. Pre-Deployment Checklist

1. Storage and loading: 1.5TB of weights takes about 5 minutes even at 5GB/s sequential read; all-flash storage with paged loading support is recommended;
2. Quantization kernels: microscaling 4-bit requires hardware-software co-support; confirm your inference stack’s kernel coverage before deployment;
3. Memory bandwidth and channel count: during decode, expert computation happens on the CPU, so prioritize multi-channel, high-memory-bandwidth platforms;
4. KV cache capacity: estimate as “concurrency × context length × per-token KV overhead”; hybrid linear attention can compress this substantially, but actual measurements should govern;
5. Expert load balancing: uneven routing can overload some GPUs; watch balancing strategies and hotspot monitoring;
6. Observability: token-level latency distribution, VRAM and memory watermarks, and all-to-all bandwidth utilization should all be in monitoring and alerting.

8. Self-Host or Cloud: A Simple Decision Framework

• Low usage, occasional calls: use a cloud API directly; don’t touch hardware;
• Sensitive data, single user or small-team research, limited budget: CPU+GPU heterogeneous single server, accepting single-digit tokens/s;
• High concurrency, low latency, data cannot leave your domain, dedicated ops staff: self-built supernode cluster, budget in the tens of millions of RMB range;
• In between: consider dedicated cloud compute instances, balancing compliance isolation against managed operations.

9. Final Thoughts

The ledger for local deployment of very large MoE models can be condensed into three sentences: capacity is calculated on total parameters (even at 4-bit, start from 1.5TB), compute is calculated on activated parameters (sparsity saves power, not VRAM), and communication determines cluster efficiency (without high-speed interconnect, more GPUs just idle). The speed, VRAM, and price figures in this article are estimates based on public information; once the weights are officially released, verify each item with benchmarks before committing to a purchase.