How Much Does It Cost to Deploy a Trillion-Parameter MoE Model Locally?

Introduction: Why “Can It Run?” Is a Storage Problem First

Trillion-parameter MoE models have turned “local deployment” from a conversation topic into a real engineering challenge. A quick disclaimer: except for publicly disclosed architecture details and API pricing, all figures for speed, memory, and hardware configurations in this article are estimates based on publicly available information. Once the official model weights are released, every claim should be verified against actual benchmarks.

A common misconception about MoE models is that “each token only activates a small fraction of parameters, so inference is cheap.” The correct picture has two halves:

• Activation ratio determines compute cost: each token only forward-computes through the few experts it is routed to.
• Total parameter count sets the storage floor: experts can be called in any combination, so all 896 experts must reside in addressable storage before inference begins.

Take this model as an example: 2.8 trillion total parameters, 896 experts, 16 experts activated per token, yielding an expert activation ratio of roughly 1.79%. Note this is the “expert count activation ratio,” not strictly identical to the parameter activation ratio, but close enough at the order-of-magnitude level. In plain terms, this is a problem of fitting approximately 1.5 TB of weights into memory or VRAM—even if the model only produces a few tokens per second.

At the architectural level, the backbone consists of gated incremental linear attention and latent-space compressed attention components, combined with attention residual connections. The engineering significance of this hybrid design is straightforward: the shared attention parts are compute-intensive but parameter-light, making them suitable for high-bandwidth compute units; the sparse expert blocks are parameter-heavy but only partially activated at a time, naturally fitting “large-storage, low-frequency-access” media. This characteristic will directly shape the deployment strategy later.

Storage Floor Calculation: Where Does 1.5 TB Come From?

1. Quantization Format: 4-bit Is the “Native Form,” Not Post-hoc Compression

This model underwent quantization-aware training from the SFT stage, using micro-scaling floating-point formats: MXFP4 for weights and MXFP8 for activations. This means 4-bit deployment is not a community-driven lossy compression—it is the official format aligned with the training distribution, with relatively controlled precision loss.

2. From Parameter Count to Bytes

The bare 4-bit lower bound is straightforward:

2.8T × 4 bit ÷ 8 ≈ 1,400 GB

But MXFP4 is not pure 4-bit. According to the open specification for micro-scaling quantization, every 32 4-bit elements share an 8-bit scale, so the average bit width per element is 4 + 8/32 = 4.25 bit:

2.8T × 4.25 ÷ 8 ≈ 1,487.5 GB

3. 1,487.5 GB Is Just the Lower Bound

Real-world memory footprint must also account for four categories of overhead:

• Non-4-bit tensors: embedding layers, output heads, normalization parameters, routers, and some attention components typically retain BF16/FP8 precision.
• Metadata and alignment: expert weight paging, address tables, and memory alignment all add extra consumption.
• Runtime workspace: activation buffers, all-to-all communication buffers, and intermediate computation graph results.
• KV cache: grows linearly with concurrency and context length, and cannot be ignored in long-context scenarios.

A practical engineering rule is to plan for “weights × 1.1–1.3” in total capacity—meaning at least 1.6–1.9 TB of planning.

4. A Reference Point: The 671B-Parameter Model

For comparison, the industry’s open-source 671B-parameter MoE model (including roughly 14B in multi-token prediction modules) requires about 685 GB in full FP8; after 4-bit weight quantization it is roughly 335 GB of weights plus 37 GB of activations, totaling around 400 GB. This is precisely the scale that 8 × 80 GB accelerator cards (640 GB combined) can cover—hence the origin of the “one 8-GPU machine running a full-scale model” tier.

At 2.8 trillion parameters—more than 4× larger—storage crosses the 1.5 TB threshold. The single-node 8-GPU route is completely ruled out, and the deployment paradigm must be redefined.

After You Fit It: Communication Becomes the First Bottleneck

1. The Communication Cost of Expert Parallelism

Large-scale MoE inference typically uses expert parallelism: 896 experts are spread across multiple cards, and after routing, each token’s activations must be sent via all-to-all communication to the card holding the target expert, then sent back after computation. This process is extremely sensitive to communication bandwidth—latency directly eats into compute efficiency.

The problem is a bandwidth gap: intra-machine card-to-card high-speed interconnect bandwidth is one to two orders of magnitude higher than inter-machine networking over standard fabrics. Once a token’s routing target lands in a different communication domain, a single all-to-all round-trip can take longer than the expert computation itself.

2. Why the Official Recommendation Calls for 64+ GPU Supernodes

The “supernode” approach binds a larger pool of accelerators into a single “logical machine” using high-bandwidth interconnect, keeping all-to-all traffic within one communication domain. For a 2.8-trillion-parameter model, the official guidance of 64+ accelerators per supernode is essentially about keeping the nearly 1.5 TB of weights and sufficient communication bandwidth in the same domain.

This leads to an important conclusion: chaining several 8-GPU machines together over standard Ethernet may technically have enough VRAM, but actual throughput will collapse. Cobbled-together multi-node setups are basically infeasible at the trillion-parameter scale unless you accept extremely low throughput or switch to a heterogeneous approach.

Three Viable Paths and Their Cost Ranges

Path A: Full-GPU Supernode—Full Performance, High Throughput, Tens of Millions in Investment

Approximate hardware composition:

• 8 machines × 8 cards each = 64 flagship 80 GB accelerators (5.1 TB total VRAM, leaving ample headroom for weights, KV cache, and communication buffers);
• 400G-class RDMA networking (multiple NICs per machine + low-latency switches);
• High-performance parallel storage and all-flash local disks.

Based on public market pricing, a single 80 GB flagship accelerator typically ranges from 150,000 to 250,000 RMB. The cards alone run roughly 10–16 million RMB; adding chassis, networking, racks, and power, the total lands in the 15–25 million RMB range. The full system draws roughly 60–90 kW, requiring datacenter modifications, with annual electricity costs alone in the hundreds of thousands of RMB.

Best for: organizations and inference-as-a-service providers with sustained high-concurrency demands.

Path B: CPU+GPU Heterogeneous—150,000–300,000 RMB, Single-Digit Tokens/s

This is the most realistic option on a tight budget, built on the idea of “tiered access by frequency”:

• The shared attention parts are compute-dense and parameter-light—keep them on GPU for high-bandwidth execution.
• The 896 sparse experts dominate storage but only 16 activate per token—offload all of them to CPU main memory, fed by memory bandwidth.

The key enabler is an asymmetry: per token, the actively computed parameters are roughly 2.8T × 1.79% ≈ 50 billion (plus shared attention layers at each stage), giving a compute intensity comparable to a 50B-parameter dense model—yet storage still demands 1.5 TB. “Modest compute, enormous storage” is precisely where CPU big-memory + GPU shines.

Estimated hardware:

• Dual-socket server-grade CPU platform + 2 TB memory (roughly 32 × 64 GB RDIMMs): 50,000–100,000 RMB for memory, 80,000–150,000 RMB for the full platform;
• 1–2 consumer flagship GPUs at 24 GB each for attention compute and dispatch: 30,000–50,000 RMB;
• Total: roughly 150,000–300,000 RMB (varies with memory generation and GPU tier).

Performance expectations can be extrapolated from community heterogeneous inference reports: on a 671B-parameter 4-bit model (~377 GB), a dual-socket server with a single consumer GPU achieves roughly 200+ tokens/s in prefill and about 14 tokens/s in decode—numbers that include caveats like expert-selection strategies. With 4× the storage, decode for the 2.8T model will most likely fall into single-digit tokens/s, with concurrency capacity essentially limited to a single user.

Two caveats: first, some heterogeneous frameworks apply expert pruning or selection strategies to reduce CPU-side compute, which changes model behavior—benchmark numbers should be taken with a grain of salt. Second, decode speed is primarily bottlenecked by memory bandwidth and PCIe transfer; simply adding CPU cores helps little. Prioritize memory channel count and bandwidth when selecting hardware.

Path C: Cloud Rental—Light-Asset Validation and Elastic Scaling

For most teams, building a 64-GPU supernode is impractical. The cloud path is often more sensible:

• Rent instances: cloud platforms such as Baidu AI Cloud offer GPU hosts and container instances billed by the hour or on-demand, letting you first validate the quantized model and inference pipeline at small scale, then scale as needed—avoiding a large upfront capital outlay.
• Use managed APIs: call large-model service platform APIs billed per token, with zero hardware and zero operations overhead; per-unit cost is usually far below amortized self-built costs.

Specific pricing varies widely by provider, GPU type, region, and committed usage. Rather than quoting numbers here, combine real-time quotes with your own concurrency and context-length requirements to estimate monthly spend.

Hidden Costs That Are Easy to Overlook

1. KV cache and concurrency: concurrency × context length determines KV memory. In long-context scenarios, this can consume 10–20%+ of the memory budget, directly increasing the required card count.
2. Power, cooling, and facility: the full-GPU path starts at hundreds of thousands of RMB per year in electricity, plus UPS and cooling upgrades.
3. Quantization quality verification: although the model was trained with quantization awareness from SFT (MXFP4 weights, MXFP8 activations), theoretically keeping 4-bit precision loss modest, benchmarks are still needed once weights are released.
4. Staffing and operations: distributed inference tuning (expert parallelism degree, routing load balancing, communication-compute overlap) requires dedicated engineers—a cost frequently underestimated.
5. Data compliance: for many enterprises, the real motivation for on-premises deployment is keeping data in-house. This “compliance benefit” should be factored into any option comparison.

Decision Guidance

• Individual or research experimentation, tolerating single-digit tokens/s: choose Path B—150,000–300,000 RMB for a dual-socket server with 2 TB memory plus a consumer GPU.
• Enterprise with sustained high concurrency: Path A (tens of millions self-built) or Path C (cloud elastic), depending on whether usage is steady.
• Need the model’s capability, not the deployment itself: use an API directly—lowest total cost.
• Before starting on any path, do three things: confirm storage capacity ≥ weights × 1.2; verify inter-card and inter-machine interconnect bandwidth; re-check throughput and precision with benchmarks once weights are released.

Conclusion

Locally deploying a trillion-parameter MoE model is fundamentally a systems-engineering problem of “tiered storage + communication-domain design.” First, get nearly 1.5 TB of weights into addressable storage. Then use high-bandwidth interconnect and heterogeneous scheduling to keep tokens flowing. Full-GPU supernodes solve for speed, CPU-heterogeneous setups solve for cost, and cloud rental solves for agility. All figures in this article are estimates based on publicly available information—once the weights are officially released, validate everything with benchmarks before placing any purchase order.