The Bottom Line: This Is Not a Workstation Project

If you are picturing a high-end workstation with a few consumer GPUs, stop there. Using a typical next-generation trillion-parameter MoE (Mixture-of-Experts) model as the reference point — roughly 2.8 trillion total parameters with about 104 billion activated per token — even the bare minimum of “it runs” already means a data-center-class multi-GPU server. This article does not discuss whether you can actually get your hands on the cards. It focuses on three calculations: how much VRAM you need, how the hardware options break down, and roughly what it costs.

1. VRAM Sizing: Why the Starting Point Is 1.5TB

1.1 Sparse Activation Does Not Reduce VRAM Requirements

The selling point of MoE architectures is that only a small fraction of parameters is activated per step: about 104 billion activated parameters determines the per-step compute load and energy consumption, which looks far lighter than a dense model of the same total size. But routing is dynamic, and the inference system cannot predict which experts the next token will hit. As a result, all 2.8 trillion parameters’ worth of weights must reside in VRAM at all times. Placing expert weights in CPU memory or on NVMe and loading them on demand is theoretically possible, but every expert switch incurs PCIe transfers — or worse — and throughput collapses to unusable levels. So the first-principles constraint is: VRAM capacity ≥ all weights + runtime overhead.

1.2 Native 4-bit Is the Official Precision, Not Post-Hoc Quantization

One design choice that helps deployment: these models natively use a 4-bit floating-point weight format (such as MXFP4-style microscaling block floating point). 4-bit is the precision at which the model is trained and released, not a forced quantization of 16-bit weights down to 4-bit. This means two things. First, VRAM sizing can safely be computed directly at 4-bit. Second, the accuracy loss is within the official design envelope, so you do not need to fall back to 8-bit or 16-bit deployment to “preserve accuracy.”

1.3 The Math: 1.4TB of Weights + Runtime ≈ 1.5TB

There is a general formula for weight VRAM:

Weight VRAM ≈ total parameters × bits per parameter ÷ 8

Plugging in this example: 2.8 trillion × 4 bits ÷ 8 ≈ 1.4TB.

Then add three overhead components: activation values and temporary tensors, inference framework runtime and communication buffers, and KV cache (which grows linearly with context length and concurrency). For a single-user scenario, the actual VRAM budget should be planned at about 1.5TB.

total_params = 2.8e12      # total parameters
bits_per_weight = 4        # native 4-bit weights
weight_tb = total_params * bits_per_weight / 8 / 1e12   # ≈ 1.4 TB
budget_tb = weight_tb * 1.05 + 0.03                     # rough estimate for runtime + activations
print(f"Weight VRAM ≈ {weight_tb:.2f} TB")
print(f"Single-user VRAM budget ≈ {budget_tb:.2f} TB")

As for KV cache, it can be estimated separately with a general formula: KV VRAM ≈ 2 × layers × KV heads × head dimension × bytes per element × context length × concurrency. Once context length climbs into the hundreds of thousands or millions of tokens, KV cache becomes the dominant VRAM consumer — which is the key variable behind whether concurrency can scale, as discussed later.

2. Three Hardware Tiers: From “It Runs” to “It’s Enough”

2.1 Tier 1: It Runs, Speed Irrelevant

Doing pure capacity division against a 1.5TB budget:

• With previous-generation 80GB data-center cards: 1500 ÷ 80 ≈ 19 cards;
• With 141GB large-VRAM cards: 1500 ÷ 141 ≈ 11 cards.

Theoretically the capacity fits, but there is a common pitfall: neither of these GPU generations natively supports 4-bit floating-point formats. After the weights are loaded, they typically must be dequantized to 8-bit or 16-bit at runtime to participate in computation — VRAM usage inflates after dequantization, effective capacity shrinks, and precious high-bandwidth VRAM is wasted. “It fits” and “it runs decently” are two entirely different things.

The more realistic minimum configuration is 8 of the current-generation 288GB flagship data-center cards: 2.3TB total, which leaves about 0.8TB of headroom for KV cache and runtime after accommodating 1.4TB of weights. The official deployment documentation of mainstream inference frameworks also recommends 8-way parallelism — so 8 cards is not an aggressive configuration but an industry-consensus starting line. Conclusion: even “it just runs” is already a server-class project, unrelated to ordinary workstations.

2.2 Tier 2: 25 Tokens/s for a Single User

The hardware starting point is still 8 current-generation large-VRAM cards. The reason is simple: the bottleneck is first and foremost fitting the 1.4TB of weights. You cannot shrink the server to two or three cards just because “only one person is using it” — the result of shrinking is that it simply does not fit, not that it runs a bit slower.

Why is the bottleneck memory bandwidth rather than compute? During decoding, every token generated requires streaming the roughly 104 billion activated parameters through memory once — on the order of 52GB at 4-bit. Single-stream speed is fundamentally constrained by memory bandwidth, which is exactly why data-center inference cards push HBM bandwidth so high.

Is 25 tokens/s a reasonable target? Empirically, single-stream decoding for large models typically lands in the “tens of tokens per second” range, and 25 is a conservative, reasonable target. But note: public materials do not provide measured data for this exact configuration, so no guarantee can be made. Before launch, you must benchmark with your real context lengths — the longer the context, the greater the attention and KV read/write overhead, and single-stream speed will drop noticeably.

2.3 Tier 3: 10 Concurrent Users at 25 Tokens/s Each

That means aggregate output of at least 250 tokens/s.

Interestingly, the VRAM configuration does not necessarily need to be higher than Tier 2. Weights are shared across all requests; rising concurrency adds KV cache and activations, not a second copy of the weights. Based on a rough estimate of KV usage per request, a single 8-GPU node can theoretically hold about 40 concurrent requests; if your workloads use contexts far shorter than a million tokens, the resident count can be even higher. So 10 concurrent users alone does not force you onto a second server.

The real challenge is scheduling and throughput: when 10 streams are generating simultaneously, can each sustain 25 tokens/s? This depends on the inference framework’s continuous batching, KV management, and VRAM scheduling efficiency — metrics that must be measured, not inferred from VRAM capacity. In practice, also watch for head-of-line blocking when short and long requests are mixed, KV eviction policies, and VRAM fragmentation.

3. The Money Question: Based on Cloud Platform Pricing

These GPUs are all data-center products; individuals cannot buy them, and retail versions do not exist. The pragmatic approach is to look directly at cloud platform instance pricing. Taking one overseas cloud platform’s publicly listed GPU instance pricing as an example (current-generation large-VRAM cards, 12-month reserved pricing):

• About $4.76 per GPU per hour;
• An 8-GPU machine ≈ $38.08/hour;
• At 730 hours per month ≈ $27,800/month;
• Roughly $330,000 per year.

A few reminders:

1. Reserved pricing is usually the lowest tier, but it comes with a longer lock-in period; on-demand pricing is typically significantly higher, and prices vary widely across vendors and regions. The numbers above are only for order-of-magnitude judgment. Cloud platforms both inside and outside China (including Baidu AI Cloud) offer on-demand and reserved billing models; it is worth comparing multiple vendors before committing.
2. There are hidden costs beyond the GPU: a weight file starting at 1.4TB requires object storage and download bandwidth, and instances usually need to be paired with large CPU memory, local disks, and internal network bandwidth — all of which show up on the bill.
3. If you are considering building your own data center, you must factor in the full machine purchase, power and cooling, colocation, and operations staffing, and compare that against cloud reserved pricing over the full lifecycle before deciding.
4. A pragmatic path: rent an 8-GPU node by the hour to complete benchmarking first (single-stream speed, concurrent throughput, maximum context length). Once you have real data, decide whether to buy a long-term reservation — avoiding the trap of locking in a year only to find you cannot hit the target.

4. Pre-Launch Checklist

• VRAM headroom: reserve 20%–30% headroom for KV growth and peak usage;
• Format support: confirm the inference framework has native acceleration for the native 4-bit floating-point format, to avoid falling into the inefficient “dequantize-at-runtime” mode;
• Benchmark metrics: TTFT (time to first token), TPOT (time per output token), and aggregate concurrent throughput — all three must meet target;
• Context tiering: benchmark short and long contexts separately, and set service targets for each;
• Supporting infrastructure: orchestrate inference services with a container platform, keep weight files in object storage, and integrate logging and monitoring/alerting;
• Cost guardrails: set budget alerts and auto-release policies to prevent on-demand instances from running away.

5. Summary

Local deployment of a trillion-parameter MoE model is fundamentally a combined problem: “VRAM capacity comes first, bandwidth determines speed, and scheduling determines concurrency.” 1.4TB of weights makes 8 current-generation large-VRAM cards the starting line; streaming activated weights per token means single-stream speed must be measured; weight sharing means the first bottleneck to concurrency scaling is scheduling, not VRAM. On cost, an 8-GPU reserved instance runs about $38/hour, roughly $330,000 per year, plus hidden storage and network overhead. For most teams, short-term rental and benchmarking first, then deciding on a reservation, is the most reliable path to production.