How Much Does It Cost to Deploy a Trillion-Parameter MoE Model Locally?

1. The Bottom Line Up Front

Using a massive Mixture-of-Experts (MoE) model with approximately 2.8 trillion total parameters and weights stored natively in 4-bit micro-scaling floating point format (MXFP4) as our analysis sample, we can state the conclusion upfront: the minimum threshold to “just run” this class of model is already a data-center-grade multi-GPU server project—not something you can solve by assembling a workstation and slotting in a few consumer flagship GPUs. Below, we break down hardware and costs across three tiers: “just make it run,” “single user at 25 tokens/s,” and “10 concurrent users at 25 tokens/s each.”

Note: This article provides data-level analysis only and does not discuss supply or channel issues for specific GPU models. All performance metrics should be validated through actual testing before going live.

2. The Hard Math of VRAM

2.1 The Key MoE Characteristic: Total Parameters ≠ Activated Parameters

The core idea behind Mixture-of-Experts architecture is “division of labor”: parameters are split into numerous expert networks, and each token is routed to only a small subset of them. Our sample model has approximately 2.8 trillion total parameters, but only about 104 billion are actually activated per token generated. This creates the classic tension on the inference side:

• Compute cost is calculated based on “activated parameters” and is relatively manageable;
• VRAM usage is calculated based on “total parameters” and cannot be reduced at all—all weights must reside in VRAM, ready for routing calls at any time.

Therefore, for local deployment, the primary constraint is not compute but capacity.

2.2 What Native 4-bit Means

The sample model’s weights are at 4-bit precision in their native state—this is not a lossy quantization forced by compressing FP16 weights. In other words, “quantization level no lower than FP4” is its natural factory state; 4-bit is the official design floor, and there is no room for “further reducing precision to save VRAM.” VRAM can be estimated directly at 4-bit:

Weight VRAM ≈ Total parameters × 4bit ÷ 8
            = 2.8 trillion × 0.5 bytes ≈ 1.4 TB
Per-instance total VRAM ≈ Weights + Activations + Runtime overhead + KV/State cache ≈ 1.5 TB

This 1.5 TB is the baseline for all subsequent configuration estimates.

2.3 Why the Consumer GPU Path Doesn’t Work

A single consumer flagship GPU typically has only 32 GB of VRAM. Just to hold 1.5 TB, you would theoretically need about 47 cards; even counting only the 1.4 TB of weights, you’d need around 44. Multi-GPU inference requires high-bandwidth coherent interconnects between cards for tensor parallelism, and the topology complexity, bandwidth bottlenecks, and debugging costs of spanning dozens of cards across multiple machines are well beyond the realm of “local deployment.” So the consumer path should be ruled out from the start—this isn’t a matter of “buying a few top-tier GPUs”; even dozens won’t work.

3. Three Requirement Tiers, Three Configuration Answers

Tier	Core Constraint	Hardware Baseline	Key Risk
Just make it run	Fit ~1.5 TB VRAM	New-gen 288GB single card ×8 (fallback: 80GB×19 or 141GB×11)	Dequantization overhead on older-gen cards
Single user at 25 tokens/s	Capacity + memory bandwidth	Same as above, cannot be reduced	Speed degradation at long context
10 concurrent × 25 tokens/s	Aggregate throughput ≥250 tokens/s	Same node + scheduling optimization	Concurrency long tail, per-stream slowdown

3.1 Tier 1: Just Make It Run, Speed Doesn’t Matter

Working backward from 1.5 TB capacity:

• Previous-generation data center GPUs (80 GB per card): approximately 19 cards needed;
• Same-generation large-VRAM version (141 GB per card): approximately 11 cards needed.

But there’s an easily overlooked detail: previous-generation data center GPUs do not natively support decoding execution for this type of 4-bit format and require runtime dequantization—the weights fit, but decoding requires continuous format conversion, and the bandwidth and compute overhead will significantly drag down actual performance. So for the “just make it run” tier, the more realistic baseline is new-generation large-capacity data center GPUs: 288 GB VRAM per card, 8 cards in a single node totaling approximately 2.3 TB, enough to hold all weights with runtime headroom on a single machine. Official deployment references from mainstream open-source inference frameworks also use 8-card parallelism as the baseline configuration.

Conclusion: Even for the most lenient “just make it run” tier, this is already a standard single-node server project, requiring supporting data center space, power, and operations.

3.2 Tier 2: Single User, Stable 25 Tokens/s

The hardware starting point should still remain at the “288GB×8” level. The reason is the same as the previous tier: the 1.4 TB weight capacity constraint has nothing to do with “how many people use it”—you can’t shrink the machine to two or three cards just because there’s only one user. Weights must reside in full; this is a rigid requirement.

Once capacity is satisfied, single-stream speed is primarily determined by memory bandwidth, decode memory access efficiency, and framework scheduling overhead. Single-stream large model inference speed typically falls in the range of “tens of tokens per second,” so 25 tokens/s is a reasonable target. However, it depends heavily on context length—the longer the context, the larger the KV cache and the heavier the memory access pressure, making speed degradation more likely. Therefore, before going live, this tier must be stress-tested at the actual maximum context length used in production; you cannot extrapolate from VRAM capacity alone.

3.3 Tier 3: 10 Concurrent Streams, 25+ Tokens/s Each

10 concurrent × 25 tokens/s means the system must reliably deliver over 250 tokens/s aggregate throughput. Interestingly, the VRAM configuration for this tier may not necessarily be higher than Tier 2: large model serving universally employs batch scheduling, where multiple requests share resident weights, and what truly grows with concurrency is the KV cache portion. According to existing industry estimates, an 8-card node at this level can hold at least around 40 concurrent requests; if production contexts are not in the million-token range, the theoretical request capacity would be even higher.

So 10 concurrent streams alone won’t force you to add a second server. The real challenge lies in the word “simultaneously”: when all 10 streams are continuously generating, can the scheduler fairly allocate compute, suppress long-tail request slowdown, and keep each stream stably above 25 tokens/s? This is a performance engineering problem at the throughput and scheduling level that can only be resolved through stress testing—there are no paper shortcuts.

4. Finally, the Money: Self-Built vs. Cloud

These GPUs are all data-center-grade products that individuals and workstation channels generally cannot obtain, so costs can only be estimated along two paths.

4.1 Self-Built Purchase

• An 8-card complete system with new-generation large-VRAM inference cards typically costs in the millions of RMB range at market prices, and often requires channel resources;
• If falling back to previous-generation 80 GB cards to make up 19 units, you’d need at least 3 complete systems (8+8+3) plus cross-machine high-speed interconnects—total cost may not be lower, and operational complexity actually increases;
• Beyond the purchase price, there are ongoing costs: an 8-card system at this level typically draws from several kilowatts to around 10 kW, and 24/7 operation means electricity, data center cooling, and hardware depreciation add up to a considerable annual expense.

4.2 Cloud Rental

A more pragmatic approach is renting GPU compute on demand: through GPU compute instances on cloud platforms such as Baidu AI Cloud, paying by usage or by subscription period, handing off data center space, power, and operations to the platform, and paying only for core compute. The cost formula is straightforward:

Monthly cost ≈ Per-card hourly rate × Number of cards × 730 hours

Based on public market rates, cloud hourly rental for previous-generation 80 GB-class single cards is roughly in the single-digit USD range, while new-generation large-VRAM cards are more expensive—an 8-card node at tens of USD per hour is normal. Specific card types, regions, and contract pricing vary widely; please refer to each platform’s official pricing page. A rough rule of thumb: for long-term high-utilization scenarios, one to two years of self-built total cost of ownership may be comparable to long-term rental, after which self-building becomes more economical; for workloads with high fluctuation, the cloud’s elasticity and managed operations offer clear advantages.

5. Pre-Launch Validation Checklist

Regardless of which path you choose, complete these three validation steps before formal investment:

1. Capacity validation: Confirm that the target card type × card count leaves at least 20% VRAM headroom under actual weights + runtime usage;
2. Single-stream stress test: Measure single-stream speed at the actual maximum context length used in production, confirming that 25 tokens/s still holds at long context;
3. Concurrency stress test: Simulate 10 or more requests generating simultaneously, observing per-stream speed distribution, long-tail behavior, and whether time-to-first-token (TTFT) is acceptable.

Conclusion

For a massive MoE model with 2.8 trillion parameters stored natively at 4-bit, local deployment starts at the “8-card data center server” level from the very lowest tier. Single-user 25 tokens/s and 10 concurrent × 25 tokens/s can share the same hardware baseline—the difference lies in scheduling optimization and depth of stress testing. On cost, self-building is a one-time investment in the millions of RMB plus ongoing operations, while cloud rental is billed hourly and can be stopped at will. Calculate the four accounts—VRAM, speed, concurrency, and cost—before committing; that is the most prudent path.