Deploying a Trillion-Parameter MoE Model Locally: VRAM Requirements, Concurrency, and Cost Analysis

The Bottom Line: This Is Not a Consumer-GPU Problem

Many developers see “open weights” and assume a few high-end consumer graphics cards will be enough to run the model at home. For dense models below a few hundred billion parameters, that logic can work with certain quantization methods. But when we are talking about trillion-parameter Mixture-of-Experts (MoE) architectures with tens of billions of active parameters per token, the situation is fundamentally different:

Even the “minimum viable” setup requires server-grade hardware, not workstation-grade hardware.

This article focuses on one thing: calculating the VRAM, concurrency, and cost requirements for this class of model. Whether you can purchase these accelerators or fit them in your facility is outside the scope of this discussion.

---

The VRAM Calculation: Why the Weights Are a Hard Barrier

Total Parameters vs. Active Parameters

The core feature of an MoE architecture is a large total parameter count, but only a fraction of parameters are activated per token. Take a typical configuration:

• Total parameters: approximately 2.8 trillion
• Active parameters per token: approximately 104 billion

This leads to a common misconception: if only 104 billion parameters are active, shouldn’t VRAM requirements follow that number?

No. Active parameters determine compute requirements, but all weights must reside in VRAM. The router can select a different combination of experts for every token. Since you cannot predict which experts will be needed next, all expert weights must be accessible at all times.

FP4 Weights: The Native Precision, Not Post-hoc Quantization

The good news is that this class of model natively uses MXFP4 (4-bit floating point) as the weight precision. This is crucial:

• It is not a forced 4-bit quantization of FP16 weights after training — it is designed and released at 4-bit precision;
• Precision loss is therefore relatively controlled, and this represents the “officially supported deployment format.”

The weight VRAM calculation is straightforward:

Weight VRAM ≈ Total Parameters × Bits per Parameter ÷ 8
            = 2.8 × 10^12 × 4 ÷ 8
            ≈ 1.4 TB

Just the weights alone require 1.4 TB of VRAM.

Don’t Forget Runtime Overhead

Beyond weights, a real deployment also requires:

Overhead Item	Description
Activations	Intermediate tensors during forward propagation
KV Cache	Grows linearly with context length and concurrency
State Space / Runtime Buffers	Framework scheduling and communication buffers

In total, a single-user deployment realistically needs around 1.5 TB of GPU memory.

What does this number mean in practice? Here are some reference points:

• Accelerators with 80 GB of VRAM: approximately 19 units in theory
• Accelerators with 141 GB of VRAM: approximately 11 units in theory

But note: “theoretical capacity” is not the same as “runs well.” Previous-generation accelerators cannot natively execute MXFP4 — they require runtime dequantization, which consumes compute and bandwidth, significantly hurting actual throughput.

A more realistic minimum configuration is 8 next-generation accelerators with native MXFP4 support. This is not a single vendor’s opinion — major inference frameworks list 8-GPU parallelism as the starting point in their official deployment guides.

---

Three Configuration Tiers: From “It Runs” to “Ten Concurrent Users”

Let’s break down the requirements into three tiers and examine the hardware starting point and real bottleneck for each.

Tier 1: It Runs, Speed Not Guaranteed

Goal: Load the model and generate tokens; speed is not a metric.

Hardware starting point: 8 next-generation accelerators (with 180 GB+ VRAM per card), or an equivalent high-memory setup.

Bottleneck analysis:

The core constraint at this tier is capacity, not compute. The 1.4 TB of weights must fit. If they don’t fit, nothing else matters. While previous-generation cards may have sufficient theoretical capacity (e.g., 11×141 GB or 19×80 GB), they lack native MXFP4 support and require dequantization paths, resulting in a worse experience.

Even “just running it” is already a server-grade project.

Tier 2: 25 Tokens per Second for a Single User

Goal: Interactive use for one user with acceptable output speed.

Hardware starting point: Still 8 next-generation accelerators.

Why can’t you use fewer cards?

You might think: with only one user, why not use fewer cards?

The answer is no. The reason is simple:

VRAM capacity is a hard constraint, independent of concurrency.

You cannot compress 1.4 TB of weights into two or three cards just because only one person is using the model. The weights must be fully resident — this is a physical requirement.

Is 25 tokens per second reliably achievable?

To be clear, no benchmark data is available here, so no guarantee can be made. However, a reasonable judgment is:

• Single-stream large model inference typically lands in the range of “tens of tokens per second”;
• Therefore, 25 tokens per second is a reasonable target, but not a promise;
• Stress testing must be done with your actual context length before deployment, since long contexts significantly slow down decoding.

Tier 3: 10 Concurrent Users at 25 Tokens per Second Each

Goal: 10 users simultaneously, each maintaining 25 tokens per second.

Aggregate output requirement: At least 250 tokens per second of sustained throughput.

Does VRAM need to double?

Not necessarily. Here is a counterintuitive conclusion:

• A single 8-GPU node can handle roughly 40 concurrent requests at a minimum;
• If the context is not at the million-token level, the number of resident requests can be even higher.

In other words, 10 concurrent users alone do not force you to add a second server.

So where is the real bottleneck?

The bottleneck shifts from “capacity” to “throughput and scheduling”:

• With 10 simultaneous generation streams, GPU compute, memory bandwidth, and KV Cache management all become contention points;
• Whether each stream can sustain 25 tokens per second is a throughput and scheduling performance metric;
• This must be measured in practice — it cannot be derived from VRAM capacity alone.

In short: “fits in memory” ≠ “runs fast” ≠ “stable under concurrency” — these are three separate problems.

---

The Cost Picture: Why Cloud Platforms Are the Only Realistic Option

Self-Hosting vs. Cloud Rental

These accelerators are data-center-grade products, not readily available on the consumer market. For most teams, cloud rental is the only practical path.

The cost calculation is straightforward:

Hourly cost = Per-card hourly rate × Number of cards

Using one cloud provider’s GPU instance as an example, a next-generation high-memory accelerator at 12-month reserved pricing costs approximately $4.76 per card per hour. Then:

8-card node hourly cost = 4.76 × 8 ≈ $38.08 per hour

At this rate:

Usage Duration	Estimated Cost
1 day	~$914
1 month	~$27,000
12 months (reserved)	~$330,000

Note: This is a single public pricing example from one cloud provider. Prices vary significantly across platforms, reservation terms, and billing models (on-demand, reserved, or spot). When making selection decisions, compare GPU instance pricing across multiple cloud providers and choose a billing mode that fits your usage pattern.

Directions for Cost Optimization

1. On-demand vs. reserved: For feasibility validation, on-demand instances are more cost-effective. For confirmed long-term use, reserved pricing significantly lowers the unit cost.
2. Context length control: KV Cache grows linearly with context. Limiting context length directly reduces VRAM pressure and cost.
3. Batch scheduling: Proper batching improves GPU utilization and lowers the per-token cost.
4. Quantization strategy: Choose hardware with native support for the target precision to avoid dequantization overhead.

---

Summary Table: Three Configuration Tiers

Tier	Goal	Hardware Starting Point	Core Bottleneck
1	It runs	8 next-gen accelerators	VRAM capacity (1.4 TB weights)
2	Single user at 25 tokens/s	8 next-gen accelerators	Capacity + single-stream decode speed
3	10 concurrent users × 25 tokens/s	8 accelerators, possible expansion depending on context	Throughput and scheduling; requires practical testing

---

Practical Recommendations for Developers

1. Calculate capacity first, then talk about performance. The first barrier for a trillion-parameter MoE is always VRAM, not compute.
2. Don’t apply consumer-GPU thinking to server-grade models. At the trillion-parameter scale, the hardware form factor changes entirely.
3. Concurrency does not equal the number of servers. With sufficient capacity, a single 8-GPU node can handle dozens of concurrent requests. The real test is throughput stability.
4. Measure before going live. Context length, concurrency patterns, and output length significantly affect real-world performance. No paper calculation can replace stress testing.
5. Calculate cost based on usage patterns. Use on-demand for short-term validation, reserved for long-term — don’t lock into long-term contracts prematurely.

Deploying a trillion-parameter MoE model locally is essentially an integrated engineering problem of capacity planning, performance tuning, and cost control. Only by getting these three calculations right can you make a rational technical decision.