Local Deployment of Trillion-Parameter Models: VRAM Thresholds, Concurrency, and Cost Estimation

1. Defining the Problem: Breaking “Deployment” into Three Quantifiable Tiers

When discussing local deployment of trillion-scale MoE (Mixture of Experts) models, the most common pitfall is conflating “can it fit” with “does it run well.” A sound approach is to first define tiers by business objective, then derive hardware constraints for each tier.

This article uses three tiers:

1. Minimum usable tier: the model can load and complete a full inference pass; throughput and latency are not a concern;
2. Single-user smooth tier: a single user achieves roughly 25 tokens/s output speed, approaching a usable interactive experience;
3. Multi-concurrency tier: 10 concurrent streams, each still maintaining 25 tokens/s or above, i.e., aggregate throughput of at least 250 tokens/s.

The constraints differ across tiers: the first tier is limited by VRAM capacity, the second by VRAM bandwidth and single-stream decoding efficiency, and the third by scheduling and aggregate throughput. Each is examined below.

2. First Constraint: Weights Must Reside Fully in VRAM

2.1 Distinguishing Total Parameters from Activated Parameters

Consider an MoE architecture with roughly 2.8 trillion total parameters and about 104 billion activated parameters per token. A common misconception is that “fewer activated parameters means lower VRAM requirements.” This is wrong: MoE sparsity applies to the computation path, not to storage. The routing mechanism determines that any given token passes through only a few experts, but the weights of all experts must be accessible whenever needed, so all weights must still reside permanently in high-speed VRAM (or at least in a tier that can be swapped in quickly).

2.2 Estimating Weight Size in Bytes

If the model natively uses 4-bit precision (an official design precision such as MXFP4, rather than post-hoc quantization from FP16), weight volume can be estimated as:

Weight volume ≈ Total parameters × Bytes per parameter
             = 2.8e12 × (4 / 8)
             = 1.4e12 Bytes
             ≈ 1.4 TB

One key distinction deserves emphasis: native low-precision design and post-hoc quantization differ fundamentally in deployment difficulty. The former converges in a low-precision format during training and executes directly at inference without dequantization; the latter requires restoring weights to higher precision at runtime before computation, which carries both precision-loss risk and extra memory-access and compute overhead.

2.3 Runtime Overhead Cannot Be Ignored

1.4 TB covers only the weights. Actual deployment must also account for:

• Activation buffers: growing with batch size and context length;
• KV Cache / state space: proportional to concurrency and sequence length, growing significantly in long-context scenarios;
• Runtime footprint of the inference framework itself: graph capture, temporary buffers, communication buffers, etc.

Taken together, actual VRAM requirements for a single-user scenario land around the 1.5 TB mark. This number is the starting point for all subsequent hardware selection.

3. Minimum Usable Tier: It Fits, but May Not Run Well

3.1 Theoretical Capacity Solutions and Their Flaws

If one only sums VRAM capacity, it is theoretically possible to pool more than 1.5 TB using a larger number of small-VRAM cards. But this approach has two hard limitations:

• No native low-precision execution: some previous-generation accelerators cannot directly execute 4-bit matrix operations and require runtime dequantization, which pushes VRAM usage back up and reduces throughput;
• Insufficient interconnect bandwidth: expert parallelism requires frequent all-to-all communication, and inadequate interconnect bandwidth becomes a severe bottleneck.

So there is a clear gap between “it fits” and “it runs.”

3.2 A More Realistic Starting Point: Single-Node 8-GPU Parallelism

Common industry technical approaches generally recommend a single node with 8 GPUs as the minimum deployment unit, preferring newer-generation accelerators that natively support low-precision formats and offer larger per-card VRAM. At roughly 140 GB VRAM per card:

8 × 140 GB ≈ 1.12 TB

A single node still cannot hold the full 1.5 TB requirement, so practical deployments typically need to either:

• adopt multi-node + expert parallelism, distributing different experts across nodes;
• or accept partial weight residency in host memory / tiered storage, trading bandwidth for capacity.

Either way, the conclusion is the same: even for “it runs” alone, this is a server-class project, not a workstation-class one. It involves high-speed interconnect, multi-machine communication, and distributed inference framework tuning, with operational complexity far beyond single-machine deployment.

4. Single-User Smooth Tier: The Real Bottleneck Behind 25 Tokens/s

4.1 Why You Cannot Cut Costs by Reducing GPUs

An intuitive question: if only one person is using it, why not scale down from 8 GPUs to 2 or 3?

The answer is that capacity constraints take priority over compute constraints. A single user does not reduce weight volume—the 1.4 TB of weights cannot be reduced by a single byte. Reducing the GPU count directly means the model will not fit, not that it “runs a bit slower.” So the hardware starting point for the single-user tier is essentially the same as the minimum tier: still a single node at the 8-GPU level.

4.2 What 25 Tokens/s Represents

In magnitude terms, typical single-stream autoregressive decoding output for large models falls in the “tens of tokens per second” range. 25 tokens/s is a reasonable but empirically validated target, not something that can be guaranteed on paper. Factors affecting it include:

• VRAM bandwidth: decoding is a classic memory-bound workload, reading a large amount of weights for every token generated;
• Load balancing of expert routing: uneven routing in MoE causes some GPUs to wait while others are overloaded;
• Context length: KV Cache growth competes for bandwidth and capacity, and throughput typically drops noticeably with long contexts;
• Batching strategy: the scheduling efficiency of continuous batching directly affects single-stream latency.

The correct engineering practice is therefore: run load tests at realistic context lengths before launch, rather than inferring performance from VRAM capacity.

5. Ten-Concurrency Tier: Enough Capacity Does Not Mean Enough Throughput

5.1 Aggregate Throughput Target

10 streams × 25 tokens/s = 250 tokens/s aggregate output. This is a throughput metric, not a capacity metric.

Interestingly, this tier does not necessarily require more VRAM. Based on the capacity and compute of a single 8-GPU node, one node can theoretically handle dozens of concurrent requests; if contexts are not extremely long (e.g., not at the million-token level), even more requests can reside simultaneously. In other words, 10 concurrent requests alone usually will not force you onto a second server.

5.2 The Real Risk

The risk is whether each of the 10 streams can maintain 25 tokens/s during simultaneous generation. This involves:

• Scheduling fairness: whether the batching scheduler starves long requests in favor of short ones, or vice versa;
• VRAM bandwidth contention: more concurrent streams amplify bandwidth pressure, and per-stream speed may drop;
• Communication amplification from expert parallelism: higher concurrency means more frequent all-to-all communication and a greater chance the interconnect becomes the bottleneck;
• Tail latency: meeting the average does not mean meeting P99, and interactive experience depends more on tail metrics.

The conclusion is clear: concurrency capability must be measured, not extrapolated from VRAM capacity alone. A recommended validation method is to fix the context-length distribution, gradually increase load to 10 streams, and observe mean and P99 tokens/s per stream.

6. Cost Estimation: From Per-GPU Hourly Pricing to Monthly Budget

6.1 Why Cloud Pricing Is the Only Practical Reference

Data-center-class accelerators of this kind are essentially not sold through retail channels. Actual self-built costs include complete server procurement, data center retrofitting, power and cooling, spare parts, and operations labor—hard to summarize in a single number. A more realistic estimation basis is therefore cloud on-demand or reserved pricing, used as an upper-bound reference for self-built costs.

6.2 A Reusable Estimation Template

Assume an 8-GPU instance from a cloud provider with a per-GPU reserved price of about $4.76/hour:

8-GPU instance hourly price = 4.76 × 8 = $38.08/hour
Monthly (730 hours) ≈ 38.08 × 730 ≈ $27,798
Annual ≈ 38.08 × 8760 ≈ $333,581

Note that the above is only a derivation template based on an example unit price; actual prices vary by region, billing mode (on-demand / reserved / spot), and contract term. Readers should substitute the unit price with quotes they can obtain.

6.3 Cost Differences Across the Three Tiers Are Actually Small

A counterintuitive but important conclusion: the hardware configurations for the first and second tiers are nearly identical, because capacity constraints dominate; the third tier, if no additional nodes are added, is also roughly cost-neutral. The real cost watershed appears at the step of “whether multiple nodes are needed”—once you cross nodes, interconnect equipment, rack space, and operational complexity all rise in step changes.

So the focus of cost optimization is not “reducing GPUs” but rather:

• Improving single-node utilization: through batching and scheduling optimization, letting the same machine carry more concurrency;
• On-demand elasticity: for non-24/7 services, use on-demand instances instead of long-term reservations;
• Context length governance: controlling average context length directly reduces KV Cache pressure, effectively raising the concurrency a node can carry.

7. Pre-Deployment Checklist

Before actually placing orders, confirm each item in the following order:

1. Capacity accounting: weights + activations + KV Cache + runtime headroom—is there 15%–20% redundancy?
2. Precision path confirmation: does the accelerator natively support the model’s target low-precision format, and is dequantization required?
3. Parallelism strategy design: does the combination of tensor parallelism, expert parallelism, and pipeline parallelism match the interconnect topology?
4. Load-test plan: fix the context-length distribution, measure single-stream tokens/s and 10-stream aggregate throughput separately, and record P50/P99;
5. Cost model: choose the billing mode based on actual service hours, and convert monthly cost to cost per thousand tokens;
6. Degradation contingency: queuing, rate limiting, and short-context degradation strategies under concurrency overload.

8. Summary

Local deployment of trillion-scale MoE models is fundamentally an engineering problem of capacity first, throughput validated, cost last. VRAM capacity determines the hardware floor, and that floor does not drop just because there are fewer users; throughput targets must be confirmed through load testing under realistic context conditions; cost estimation should anchor on cloud instance pricing, recognizing that the cost difference between the first two tiers is actually small. Only by clearly separating these constraints can you avoid the common mistake of “buying GPUs by compute and failing on capacity.”