How Much Does It Cost to Self-Host a Trillion-Parameter MoE Model?

1. The VRAM Math: Where the Barrier Comes From

The mixture-of-experts (MoE) architecture is the dominant design for today’s trillion-parameter models, characterized by “massive parameters, sparse activation.” Take a typical open-source trillion-scale MoE model: about 2.8 trillion total parameters, yet each token activates only around 104 billion of them during the forward pass — less than 4%. This produces the most common misconception: “Only 4% is activated, so I only need 4% of the VRAM, right?”

The answer is no, and the reason lies in how routing works: each token is dynamically routed to different experts, and any expert can be hit, so every expert’s weights must stay resident in VRAM. That leads to two basic conclusions:

• Compute demand is determined by activated parameters: the per-token compute cost is roughly on par with a hundred-billion-parameter dense model;
• VRAM demand is determined by total parameters: the weight size sets the memory floor, regardless of the activation ratio.

The VRAM estimate fits in one line: weight memory = total parameters × bits per weight ÷ 8.

total_params = 2.8e12      # total parameter count
bits_per_weight = 4        # native 4-bit weights
weight_tb = total_params * bits_per_weight / 8 / 1e12
print(f"Model weights alone require: {weight_tb:.1f} TB")   # ≈ 1.4 TB

It’s worth emphasizing that the 4-bit precision here is typically the model’s native design precision — trained and released in a low-bit floating-point format rather than quantized down from 16-bit after the fact — so accuracy loss is manageable. But it imposes matching requirements on hardware and the inference stack: only accelerators with native low-bit floating-point compute can fully realize the capacity and bandwidth benefits; otherwise the weights must be dequantized at runtime, which negates the point.

On top of the 1.4 TB of weights, you must add activations, inference framework runtime overhead, and the KV cache (which grows linearly with context length and concurrency, and can be estimated as 2 × layers × KV heads × head dimension × bytes per element × context length × concurrency). All in, the realistic VRAM requirement for a single-user scenario is about 1.5 TB — the baseline for every plan that follows.

2. Three Deployment Tiers: From “It Runs” to “It Serves”

Tier 1: It just has to run

Matching the 1.5 TB requirement on pure capacity, two combinations exist on paper:

• Previous-generation flagship accelerators (80 GB per card): roughly 19 cards;
• Same-generation large-memory variants (141 GB per card): roughly 11 cards.

But there’s an easily missed pitfall: previous-generation hardware lacks native compute support for 4-bit low-bit floating point, forcing runtime dequantization of the weights. “It fits” and “it runs decently” are two different things, and much of the bandwidth advantage goes to waste.

The more realistic minimum is the current generation of data center accelerators: 288 GB-class cards, where 8 cards in parallel hold all the weights with headroom to spare for the KV cache and runtime overhead. In fact, the official deployment documentation of mainstream inference frameworks recommends the same 8-card setup. In other words, even if the goal is merely “it runs,” this is already a standard multi-GPU inference server project — an ordinary workstation is completely out of the picture.

Tier 2: A single user at 25 tokens/s

The hardware starting point should still be 8× 288 GB-class cards, for two reasons:

1. VRAM is a hard constraint: the 1.4 TB of weights has to fit first — there is no “shrink to two or three cards because only one person uses it” option;
2. Decode speed is bound by weight bandwidth: autoregressive generation reads the activated weights in full for every token, so the higher the aggregate memory bandwidth, the faster the output.

Is 25 tokens/s realistic? Single-stream inference on a model of this scale typically lands in the “tens of tokens per second” range, so 25 tokens/s is a reasonable target. But the actual figure depends heavily on context length, KV cache occupancy, and batching strategy — it must be confirmed by load testing against your real workload before launch, not extrapolated from paper bandwidth specs.

Tier 3: 10 concurrent users at 25 tokens/s each

This tier requires the system to sustain 250 tokens/s of aggregate output. Interestingly, the VRAM configuration doesn’t necessarily need to exceed Tier 2: judging by this class of model’s activation footprint and KV cache behavior, a single 8-card node can theoretically host around 40 concurrent requests — and more if contexts aren’t in the million-token range.

So 10 concurrent users won’t by itself force a second server. The real challenge shifts to software and scheduling:

• Continuous batching and request scheduling: keeping batching efficiency high as requests arrive dynamically with varying lengths;
• Paged KV cache management: reducing memory fragmentation and raising the number of resident requests;
• Tail latency control: when 10 streams generate simultaneously, can each hold a steady 25 tokens/s — a throughput and scheduling metric that can only be verified by measurement, never inferred from VRAM capacity.

3. The Cost: Cloud Pricing Is the Only Realistic Benchmark

All of the accelerators above are data center products with no retail channel, so availability and cost can only be assessed through cloud platforms. Take, for example, the 12-month reserved price of a major global cloud provider’s current-generation large-memory accelerator: roughly $4.8 per card per hour, or about $38 per hour for an 8-card node. Run flat out 24/7:

• Monthly: 38 × 730 ≈ $28,000;
• Yearly: roughly $330,000.

Comparable capacity is available from other providers as well — for instance, Baidu AI Cloud offers GPU cloud servers and bare metal instances with pay-as-you-go and reserved (monthly/annual) billing, which can serve as additional reference points when comparing prices across providers and choosing a deployment region. When selecting, focus on four things: per-card VRAM specification and actually usable capacity, inter-card interconnect bandwidth, native support for the low-bit weight format your model uses, and the maturity of official images and inference framework support.

A few ways to keep the cost down:

1. Assess real utilization first: for low-frequency usage, reserved instances can actually cost more — pay-as-you-go combined with scheduled start/stop is often the cheaper route;
2. Context trimming and KV cache compression directly reduce VRAM pressure and push back the scaling point;
3. Trade latency for cost: if the workload tolerates queuing, a fixed number of cards can carry a higher batching granularity;
4. Leverage discounts and regional price differences: for stable long-term loads, prefer reserved instances, and compare quotes across regions.

4. A Pre-Launch Load Testing Checklist

Whichever tier you choose, complete the following validation before going live:

1. Load test: measure weight loading time and peak VRAM watermark; reserve 10–15% headroom against fragmentation;
2. Single-stream baseline: fix the context length and measure time-to-first-token (TTFT) and time-per-output-token (TPOT);
3. Concurrency ladder: ramp from 1 stream up to the target concurrency, plot the throughput–latency curve, and locate the performance knee;
4. Long-context cases: use your workload’s real longest inputs to verify the KV cache budget and resident concurrency;
5. Stability soak: run continuously for several hours while watching for VRAM leaks, fragmentation, and drift in latency percentiles.

5. The Bottom Line

The whole article in three sentences:

• Hardware baseline: start from 1.5 TB of VRAM; the realistic minimum is 8× 288 GB-class current-generation accelerators. All three tiers (it runs / single user at 25 tokens/s / 10 concurrent × 25 tokens/s) sit at essentially the same VRAM level — the differences show up in scheduling and throughput validation;
• Cost scale: a cloud 8-card node runs about $38 per hour; around the clock, that’s roughly $28,000 per month or $330,000 per year — institutional budget territory;
• Applicability: private deployment only pays off when data compliance requirements are strict, call volume is large enough, and you have multi-GPU cluster operations capability. For individuals and small teams, managed inference services or APIs are the more rational path.