How Much Does It Cost to Deploy a Trillion-Parameter MoE Model Locally?

1. First, Frame the Problem: Not “Buying a Few Cards,” but a Storage-Wall Engineering Problem

Before discussing how to deploy a 2.8-trillion-parameter MoE model locally, three ground rules need to be established. Every calculation below is built on them:

1. MoE memory footprint is counted on total parameters; the activation ratio only affects compute. The model has 896 experts in total, and each token routes to only 16 of them — an expert activation rate of about 1.79%. But routing depends on the input, and there is no way to know before inference which token will hit which expert, so all 896 experts must reside in addressable storage. In other words, the 1.79% activation rate saves compute, not memory. Note also that this is an “expert-count activation ratio” and is not strictly equal to a “parameter activation ratio,” because shared layers always participate in the computation and the parameter counts of individual experts may be uneven.
2. Quantization-aware training sets the deployment floor. The model went through quantization-aware training with MXFP4 weights and MXFP8 activations starting from the SFT stage. That means 4-bit deployment is not a stopgap that trades accuracy for memory — it is the officially designed path and the baseline for all estimates here.
3. Until the weights are officially released, every number is an estimate. The speeds, memory figures, and hardware configurations below are extrapolations from public architecture information; before any real deployment, each item must be verified with benchmarks.

One more piece of architectural context: the attention backbone of this new generation of models typically mixes linear attention with gated KV-compression attention and introduces attention-residual structures, so the KV cache grows far more slowly than in traditional full attention. This is good news for deployment — but it helps the KV memory in long-context scenarios, not the 1.5 TB body of weights itself.

2. The Memory Floor: About 1.5 TB Is Only the Starting Point

MXFP4 is not pure 4-bit. Under the microscaling quantization format spec, every 32 4-bit elements share one 8-bit scale, which works out to 4.25 bits per weight. Here is the arithmetic in pseudocode:

# MoE weight memory floor estimate (MXFP4 + one 8-bit scale per 32 elements)
total_params = 2.8e12
bits_per_weight = 4 + 8 / 32                  # = 4.25 bit
weights_gb = total_params * bits_per_weight / 8 / 1e9   # ≈ 1487.5 GB

That 1487.5 GB is only the floor for the bare weights. A real deployment has to stack more on top:

• Non-4-bit tensors: embeddings, the router, the various normalization layers, and the high-precision parts of attention are usually kept in FP8/BF16 — a small share in percentage terms, but a significant absolute amount;
• KV cache: per token it is roughly 2 × layers × KV heads × head dim × bytes per element. Compressed attention can push this down to the tens of kilobytes per token, but 10 concurrent streams × 32K context can still accumulate tens of gigabytes;
• Runtime overhead: activation workspaces, graph-mode execution buffers, and all-to-all communication buffers can easily add tens of gigabytes.

All things considered, plan for at least 1.6 TB of memory to be safe.

For comparison: mainstream 600–700B-class MoE models take up about 685 GB in FP8; after 4-bit quantization the weights come to roughly 335 GB, or about 400 GB with activations, which eight 80GB-class accelerators (640 GB of memory in total) can carry. The model discussed here is more than four times that size — which is exactly where the threshold for “full-scale deployment” comes from.

3. Three Deployment Targets, Three Budgets

Tier 1: Just Make It Run, Never Mind the Speed

Goal: fit the nearly 1.5 TB of weights into addressable storage and get the model to emit tokens, even at single-digit tokens per second.

Approach: heterogeneous GPU + CPU inference. The shared attention portions of each MoE layer are compute-dense, so keep them on the GPU to exploit high bandwidth; the 896 sparse experts dominate storage but only 16 activate per token, so offload all of them to CPU memory on a large-memory server. Mature open-source frameworks already implement this pattern; the core techniques are layer-by-layer scheduling and expert prefetching to hide PCIe transfer latency as much as possible. NVMe can also be used for tiered expert caching and hot-expert caching, at a further cost in speed.

Reference configuration: a dual-socket server with 2 TB of DDR5 memory, plus one or two 24–48GB-class GPUs for attention computation and scheduling.

Speed and budget: each token requires reading about 27 GB from memory (roughly 50B activated parameters × 4.25 bits). Constrained by memory bandwidth and scheduling overhead, real throughput is single-digit tokens/s, and time-to-first-token can reach the minute range. Budget for the whole machine including GPUs: about RMB 200,000–400,000 (roughly $28,000–$56,000). Suitable for technical pre-research, offline batch processing, and feasibility validation.

Tier 2: Single User, 25 tokens/s

25 tokens/s is roughly a comfortable human reading speed — the bottom line for “usable.” The key change: the weights must live in VRAM. Leaning on CPU memory as a fallback cannot sustain this speed.

A quick bandwidth calculation: each token activates about 50B parameters, which is about 27 GB of reads at 4-bit. Spread across 24 cards, each card needs to read only about 1.1 GB per token; at 25 tokens/s that corresponds to about 28 GB/s per card — far below the multi-TB/s ceiling of a single card’s high-bandwidth memory. The bottleneck is not bandwidth; it is fitting 1.5 TB into VRAM in the first place.

Reference configuration: 1487.5 GB ÷ 80 GB ≈ 19 cards is the floor for weights alone; add KV cache and runtime headroom, and you should plan for 24 80GB-class data center accelerators (three 8-card servers). If you choose 141GB-class large-memory cards, this can be compressed to 16 cards (two servers). Between nodes, provision 200G-class RDMA networking. If the model ships with multi-token prediction or speculative decoding modules, additional VRAM must be reserved for their weights and caches.

Budget: at recent market prices, one 8-card 80GB-class server costs about RMB 2–3 million (roughly $280,000–$420,000); three servers plus networking and storage bring the total investment to roughly RMB 6–10 million (about $850,000–$1.4 million).

Tier 3: 10 Concurrent Users, 25+ tokens/s Each

Aggregate decode throughput must reach at least 250 tokens/s, on top of the compute demand from multi-stream long-context prefill — this is already the shape of a small inference cluster. The official recommendation is a supernode built from 64 or more accelerators; the reason is covered in the next section. If your business sets a floor on quantization precision, the memory and card-count budgets need to be recalculated at higher precision.

Reference configuration: eight 8-card servers (64 cards) + 400G-class RDMA network and switching equipment + high-performance parallel storage + data center retrofit (power, cooling, floor loading).

Budget: hardware alone is about RMB 20–30 million (roughly $2.8–$4.2 million); with networking, storage, and facilities, total investment starts at RMB 25 million (≈ $3.5 million). Don’t forget the hidden costs: a 64-card cluster draws at least 80 kW of power; at PUE 1.3 and industrial electricity rates, annual power bills can reach hundreds of thousands of RMB (tens of thousands of US dollars), and you will need at least one dedicated operations engineer.

4. Why Does 10-Way Concurrency Require a “Supernode”?

Large-scale MoE inference is extremely sensitive to all-to-all communication bandwidth. With 896 experts spread across dozens of cards, once each token is routed, its activations must be shipped to the card hosting the target expert, computed, and then shipped back — each round trip is a global all-to-all.

• If the experts sit on machines connected by ordinary Ethernet, network latency and bandwidth will eat GPU compute directly, and the more cards you add, the worse the waste;
• The essence of a supernode is to use a high-bandwidth, low-latency interconnect to enclose dozens or even hundreds of cards in the same communication domain, so that this shuttling never leaves the domain and the communication cost drops from “cross-network” to “in-domain forwarding.”

In other words, Tier 2 is a contest of memory capacity, while Tier 3 is a contest of interconnect architecture — two entirely different engineering problems.

5. A Checklist of Hidden Costs That Are Easy to Overlook

• Storage: the weight files total about 1.5 TB, and a production environment needs multiple replicas — budget for object storage or a parallel file system;
• Networking: 400G switches and optical transceivers are consumable-style spending; hundreds of thousands of RMB (tens of thousands of dollars) can disappear quickly;
• Power and cooling: as noted above, starting at hundreds of thousands of RMB per year;
• Operations staffing: debugging, monitoring, and failure recovery for large-scale distributed inference all require dedicated people;
• Depreciation risk: models iterate fast, and the mismatch between hardware lifecycle and model lifecycle is the biggest long-term risk of self-building.

6. Self-Build or Cloud? Run the TCO Numbers Before Deciding

The three tiers map to three profiles:

Tier	Speed	Budget magnitude	Who it suits
Heterogeneous “it runs”	single-digit tokens/s	RMB 200K–400K (≈ $28K–56K)	pre-research, offline jobs
Single user at 25 tps	25 tokens/s	RMB 6–10M (≈ $850K–1.4M)	long-term, high-frequency individual use
10 concurrent users	10 × 25 tokens/s	RMB 25M+ (≈ $3.5M+)	sustained service needs and an ops team

The pragmatic advice is to proceed in stages: first rent cloud compute on demand to validate business value — cloud providers such as Baidu AI Cloud offer GPU instances ranging from single cards to multi-node clusters, dedicated resource pools, and managed LLM inference services, billed on demand with no upfront capital investment. Once concurrency, latency, and data residency requirements have been validated with real data, then decide whether to self-build. Put API call costs, cloud rental costs, and self-build TCO (including electricity, operations, and depreciation) into the same spreadsheet, and the answer will usually reveal itself.

7. Final Thoughts

• All figures are estimates based on public information; once the weights are officially released, verify each item with benchmarks;
• Be clear about what you actually need — “it runs,” “usable for one person,” or “concurrent service” — the budgets differ by an order of magnitude;
• MoE memory is accounted on total parameters, while compute is accounted on activated parameters. Get this straight, and your budget won’t be calculated in the wrong direction.