Self-Hosting a Trillion-Parameter MoE Model: VRAM Requirements, Three Deployment Tiers, and Cost Breakdown

1. The Short Answer: This Is Not a “Buy a Few Consumer GPUs” Problem

When it comes to deploying ultra-large sparse mixture-of-experts (MoE) models locally, a common misconception is that as long as the weights fit in VRAM, inference will simply work. For models with total parameter counts in the trillions, that assumption is far too optimistic.

Take a representative MoE architecture with roughly 2.8 trillion total parameters and about 104 billion activated per token. Even with the weights stored in a native 4-bit floating-point format (MXFP4-class) — note that this is the officially designed storage precision, not FP16 quantized down to 4 bits after the fact — the weights alone require:

2.8 × 10^12 parameters × 4 bits ÷ 8 bits/byte ≈ 1.4 TB

Add activations, runtime buffers, the KV cache, and state-space overheads, and a single user realistically needs about 1.5 TB of GPU memory. That number determines everything: this is no longer a workstation problem — it is a server-class project.

The rest of this article breaks the requirements down into three tiers.

2. Tier 1: It Just Needs to Run, Speed Doesn’t Matter

2.1 The theoretical capacity floor

If all you care about is “the weights fit,” you can scrape together enough capacity with previous-generation data center GPUs:

• 80 GB cards: about 19 of them
• 141 GB cards: about 11 of them

But there is a critical catch: these two GPU generations cannot natively execute MXFP4 — the weights must be dequantized at runtime. Fitting the model does not mean running it well. Dequantization adds extra compute overhead, memory-bandwidth pressure, and precision-path switching, and actual throughput can drop to unacceptable levels.

2.2 A more realistic minimum

A more pragmatic starting point is current-generation GPUs that natively support low-precision formats, and 8 GPUs in parallel is the smallest deployment unit commonly seen in practice. This also matches the official deployment recipes of mainstream inference frameworks: 8-way tensor parallelism combined with pipeline parallelism is the baseline for getting a trillion-scale MoE model to run at all.

The conclusion is clear: even “just make it run” is a server-class project. An ordinary workstation cannot carry it.

3. Tier 2: One User at 25 Tokens per Second

3.1 The bottleneck is memory capacity, not compute

Many people intuitively assume that a single user means fewer GPUs.

The answer is no. The reason: the 1.4 TB of weights must fit in memory first — a hard capacity constraint that has nothing to do with concurrency. You cannot shrink an 8-GPU node down to 2 or 3 cards just because there is only one user; the weights simply won’t fit.

So the hardware starting point remains an 8-GPU node built on current-generation GPUs.

3.2 Can 25 tokens/s actually be guaranteed?

To be honest: single-stream LLM inference typically lands in the “tens of tokens per second” range, so 25 tokens/s is a reasonable target. But it cannot be guaranteed purely on the basis of a hardware configuration, because real throughput depends on:

• Context length: long contexts significantly increase KV cache footprint and attention compute
• Batching strategy: how efficiently continuous batching is scheduled
• The implementation quality of dequantization/low-precision kernels
• Memory-bandwidth utilization

Before going live, you must benchmark with your actual context lengths rather than treating theoretical peak numbers as a promise.

4. Tier 3: 10 Concurrent Users at 25 Tokens/s Each

4.1 The aggregate throughput requirement

10 concurrent streams at 25 tokens/s each means the system must sustainably deliver 250 tokens/s of aggregate output.

4.2 Memory does not necessarily need to double

Interestingly, the memory configuration for this tier is not necessarily higher than Tier 2, because:

• In capacity terms, a single 8-GPU node can host roughly 40 concurrent resident requests
• If your contexts are not in the million-token range, the theoretical number of resident requests is even higher

So 10 concurrent users by itself will not force you onto a second server.

4.3 The real risk is scheduling and throughput

The problem shifts to a different dimension: when 10 streams generate at the same time, can each one hold 25 tokens/s?

This is a throughput and scheduling performance question, determined by:

• The inference framework’s batching and preemption policies
• Paged KV cache management and fragmentation control
• How saturated each GPU’s compute and memory bandwidth become

Again, this must be measured — it cannot be inferred backward from memory capacity.

5. Cost Estimate: Based on Cloud Reserved Pricing

Because these GPUs are all data-center-class products, most users will realistically have to rent them from a cloud platform. Taking the 12-month reserved pricing for current-generation accelerators from one major cloud provider — about $4.76 per GPU per hour — an 8-GPU node works out to:

$4.76 per GPU/hour × 8 GPUs = $38.08 per hour

Converted to a monthly figure:

$38.08 per hour × 24 hours × 30 days ≈ $27,418 per month

And that covers only the GPU compute cost. Not yet included:

• High-speed interconnects (intra-node NVLink/Infinity Fabric-class links)
• Storage and model-loading bandwidth
• Network egress and load balancing
• Operations, power, and cooling (if you build it yourself)

6. Decision Guide: When to Self-Host and When to Use an API

Putting all of the above together, the decision framework is straightforward:

Scenario	Recommendation
Just experimenting / validating output quality	Use a cloud API — lowest cost
Data compliance requires private deployment	Start with an 8-GPU node; budget tens of thousands of USD per month
Up to 10 concurrent users, moderate context	A single 8-GPU node may be enough, but benchmark it first
High concurrency, long context	Multi-node distributed inference; costs scale linearly

The core conclusion: for trillion-scale MoE models, the deployment barrier is set by weight capacity, not compute. The 1.4 TB weight floor locks this firmly into server-class territory. Before you buy or rent anything, run a round of load testing with your real workload — it is worth more than any theoretical estimate.