Deploying Massive MoE Models Locally: A Three-Tier Cost Breakdown from “It Runs” to Multi-User Concurrency

1. First, a Necessary Premise: Parameter Count and Memory Footprint Are Not Linearly Related

When discussing the cost of deploying ultra-large Mixture-of-Experts (MoE) models locally, the first step is to steer clear of a common misconception: the activated-parameter ratio is not the memory-footprint ratio.

Take, as an example, a typical trillion-scale MoE architecture described in current public information: total parameters at the trillion level, 896 experts, with routing activating 16 of them per token — an expert activation ratio of about 1.79%. But note:

• Memory footprint is counted against total parameters. Even though only 16 experts are activated during inference, the weights of all 896 experts must remain in addressable storage and be fetched dynamically by the routing mechanism.
• The activation ratio affects compute consumption only, not the storage floor.
• The quantization level directly determines weight size. Even when quantization-aware training is introduced from the SFT stage (for example, 4-bit weights and 8-bit activations in floating-point formats), the overhead of the format itself still has to be accounted for.

Using a common 4-bit floating-point format as an illustration: every 32 4-bit elements carry one 8-bit scale factor, and converted according to the public open-computing specification, the effective width of a single weight element is about 4.25 bits. A trillion-parameter model therefore has a raw data floor of roughly 1,400 GB, rising to nearly 1,488 GB once scale factors are included — and this still excludes non-4-bit tensors, metadata, memory alignment, runtime workspace buffers, and the KV cache.

For a horizontal comparison: a MoE model at the hundreds-of-billions scale needs about 685 GB for its main and auxiliary parameters in 8-bit floating point, and even after 4-bit quantization it still requires roughly 335 GB of weights plus 37 GB of activations — about 400 GB in total — to run on 8 high-bandwidth accelerator cards with 640 GB of combined memory. A trillion-parameter model is more than 4 times that size, which locks the floor for a full deployment at 1.5 TB-class storage to start.

2. Why the Official Recommendation Is a Supernode

Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth. With 896 experts spread across many accelerator cards, every token must have its activations shipped to the card hosting the routed expert, then shipped back once the computation is done. Communication latency directly eats into compute; the more cards there are and the more scattered the experts, the worse the problem becomes.

The value of a supernode lies in using a high-speed interconnect protocol (such as NVLink or an equivalent-bandwidth fabric) to enlarge the communication domain, so that expert traffic never leaves a single domain. The official recommendation to build a supernode from 64 or more accelerators is, in essence, trading interconnect bandwidth for MoE routing efficiency.

3. Tier 1: Just Make It Run, Regardless of Speed

Goal: fit the nearly 1,500 GB of weights into addressable storage and get the model to emit tokens, even at single-digit tokens per second.

The most direct approach is GPU + CPU heterogeneous inference. The core logic:

• Each MoE layer has a shared attention component (KDA plus gated MLA) that is compute-intensive and stays on the GPU to exploit its high bandwidth;
• The 896 sparse experts account for the bulk of the storage but only 16 are activated at a time, so they are entirely offloaded to CPU memory on a large-memory server.

Solutions of this kind are already well established in the open-source community. As an example of public figures from one heterogeneous inference framework: a dual-socket server CPU paired with a single consumer flagship GPU running a hundreds-of-billions-parameter MoE model (about 377 GB after 4-bit quantization) achieves prefill speeds of over 200 token/s and decode of roughly 14 token/s, albeit with a constrained expert-selection strategy.

Rough cost:

• Large-memory server (1.5 TB of RAM or more): on the order of tens of thousands of RMB;
• A single high-end consumer GPU: anywhere from thousands to the tens of thousands of RMB;
• Total investment can be kept under ¥100,000 (roughly $14,000).

Trade-off: decode speeds of single digits to the low teens in token/s — suitable only for validation and light trial use.

4. Tier 2: A Single User at 25 token/s

Goal: keep decode speed stable at 25 token/s or above for a single interactive user, with quantization no lower than the 4-bit floating-point format.

At this tier, the key bottleneck shifts from “fitting the model” to “feeding it fast enough”. CPU memory bandwidth becomes the primary constraint — the expert weights do live in system memory, but the latency of every routed fetch slows down decode.

Viable paths:

1. Add more GPUs: keep the entire attention component or the hottest experts on the GPU to reduce CPU–GPU transfers;
2. Increase memory bandwidth: choose a server platform with multi-channel, high-frequency memory;
3. Optimize routing: cache frequently activated experts to lower all-to-all communication overhead.

Estimated hardware configuration:

• 4 or more high-end compute cards (256 GB or more of combined memory) to host the attention component and part of the experts;
• 1.5 TB or more of system memory to hold all expert weights;
• High-speed PCIe or NVLink interconnects.

Cost range: hundreds of thousands of RMB (roughly $30,000–$100,000). A single user at 25 token/s is achievable, but concurrency capacity remains limited.

5. Tier 3: 10 Concurrent Users, Each at 25 token/s or More

Goal: handle 10 concurrent requests with a decode speed of no less than 25 token/s for each, at a quantization level no lower than the 4-bit floating-point format.

This is the leap from “personal workstation” to “small inference cluster”. Ten concurrent requests mean total throughput must exceed 250 token/s, and the KV cache’s memory footprint grows linearly with the level of concurrency.

Architecture requirements:

• Supernode-grade interconnect: 64 or more accelerators, with high-speed links enlarging the communication domain to ensure expert traffic stays within it;
• All weights resident in VRAM or high-bandwidth memory: keep CPU memory bandwidth from becoming the bottleneck;
• Continuous batching and PagedAttention-style techniques: improve memory utilization and throughput;
• A hybrid expert-parallelism + tensor-parallelism strategy: balance communication and computation.

Cost range: starting at the ¥1 million level (roughly $140,000 and up). This is no longer “local deployment” in the usual sense — it is the construction of a small inference cluster.

6. The Limits of These Estimates and How to Verify Them

It must be stressed: all of the speed, memory, and hardware figures above — except the official architecture details and API pricing — are estimates based on public information. Once the weights are released, every item must be verified through benchmarks.

A suggested verification checklist:

1. The actual weight footprint (including scale factors, metadata, and alignment padding);
2. Decode speed curves at different quantization levels;
3. How throughput and latency change as concurrency scales from 1 to 10;
4. The difference in all-to-all communication latency inside versus outside a supernode;
5. KV cache memory usage at different sequence lengths.

7. Summary

Tier	Goal	Core Approach	Cost Level
1	Just make it run	GPU + CPU heterogeneous inference, experts offloaded to system memory	Under ¥100,000 (~$14,000)
2	One user at 25 token/s	Multiple GPUs + large memory + routing optimization	Hundreds of thousands of RMB (~$30,000–$100,000)
3	10 concurrent × 25 token/s	Supernode + high-speed interconnect + continuous batching	¥1 million level and up (~$140,000+)

Deploying a trillion-parameter MoE model locally is, at its core, a trade-off among storage capacity, memory bandwidth, and interconnect bandwidth. Tier 1 solves “fitting the model”, Tier 2 solves “feeding it fast”, and Tier 3 solves “sustaining concurrency”. Each jump in cost corresponds to breaking through one physical bottleneck.