The True Cost of Locally Deploying a Trillion-Parameter LLM

1. The Bottom Line: It’s Not Just About “Adding a Few GPUs”

Running a trillion-parameter Mixture-of-Experts (MoE) model in your own environment is fundamentally a server- or cluster-level infrastructure project. You cannot simply solve it by plugging a few consumer-grade flagship GPUs into a workstation. Even if the goal is just to “get it running,” the hardware threshold already exceeds the existing equipment capabilities of most teams.

This article does not discuss procurement channels or supply feasibility. Instead, it focuses on laying out the VRAM and cost calculations clearly. The analysis sample is set to an MoE model with 2.8 trillion total parameters, approximately 104 billion activated parameters per token, and native 4-bit precision storage. The estimation methodology is universally applicable to similar architectures.

2. The First Calculation: How Much VRAM is Actually Needed?

2.1 Total Parameters Dictate the VRAM Floor, Activated Parameters Dictate Compute

There is a common misunderstanding regarding MoE architectures: although each token only activates about 104 billion parameters, this solely affects the computational workload, not the VRAM footprint. The router distributes different tokens to different experts, and any token might hit any expert. Therefore, during inference, the weights of all experts must fully reside in VRAM and remain on standby.

Thus, the baseline for VRAM estimation is the total parameter count, not the activated parameter count—this is the most common misconception in capacity planning.

2.2 Native 4-bit vs. Post-Training Quantization are Different

The sample model natively adopts MXFP4 (Microscaling Float 4-bit, an open industry low-precision data format). The 4-bit weights are the official designed precision, not a forced compression from FP16 weights after training. This distinction is crucial for deployment quality:

• Native low precision: Precision loss is digested during the training phase, ensuring inference quality.
• Post-training quantization: Converting FP16 to 4-bit inevitably loses information. It typically requires remedial measures like calibration datasets or quantization-aware training, and the results vary by model.

2.3 From Weights to Actual Footprint: 1.4TB is Just the Starting Point

The VRAM requirement for the model weights alone is:

2.8 trillion × 4bit ÷ 8 ≈ 1.4TB

However, beyond the weights, there are three other types of overheads that must be accounted for:

1. Activations: Temporary usage for intermediate inference results.
2. KV Cache / State Space: Grows linearly with context length and concurrency; highly significant in long-context scenarios.
3. Runtime overhead: The inference framework itself, communication buffers, VRAM fragmentation, etc. It is recommended to reserve 5%–10%.

Altogether, the actual VRAM requirement for a single-user scenario is approximately 1.5TB. This is the foundation for all subsequent configuration derivations.

2.4 Reusable Estimation Formula

# Weight VRAM
weight_bytes = total_params * bits_per_weight / 8
# General approximation formula for KV Cache 
# (specific values depend on the model's public architecture)
# 2(key/value) × num_layers × num_kv_heads × head_dim × seq_len × batch_size × dtype_bytes
kv_bytes = 2 * num_layers * num_kv_heads * head_dim \
           * seq_len * batch_size * dtype_bytes
# Total requirement = Weights + KV Cache + Activations + Runtime margin (10%~20%)

Values like the number of layers and KV heads must be referenced from the model’s public configuration. Models utilizing attention variants like GQA or MLA can significantly compress KV Cache overhead.

3. The Second Calculation: Choosing Among Three Deployment Tiers

3.1 Tier 1: Just Getting It Running, Speed is Not a Concern

Reverse-calculating based on the 1.5TB actual requirement:

• Previous generation data center GPUs with 80GB VRAM: Requires about 19 cards.
• Large VRAM variants with 141GB: Requires about 11 cards.

It might seem like just “getting enough VRAM” is sufficient, but there is a critical generational issue: the above two generations of GPUs do not have native compute support for 4-bit formats like MXFP4. Although the weights can fit into VRAM in 4-bit form, they must be de-quantized at runtime before participating in matrix calculations. This consumes extra compute power and misses out on the throughput dividends of native FP4 compute units. “Fitting it in” and “running it well” are two different things.

Therefore, a more realistic minimum configuration is 8 next-generation data center accelerators with 288GB VRAM per card (aggregating 2.3TB). This generation of hardware natively supports 4-bit formats, allowing weights to participate in calculations directly at their original precision. In fact, the official deployment guidelines from mainstream open-source inference frameworks for such models also specify 8-card parallelism, corroborating that 8 cards is the de facto threshold in current engineering practice.

Conclusion: Even just to “get it running,” this is already a server-grade project, completely ruling out ordinary workstations.

3.2 Tier 2: Single User, Targeting 25 Tokens/s

The hardware starting point remains 8 next-generation large-VRAM cards. The reason is straightforward: the primary contradiction for such models is not a lack of compute, but the fact that the 1.4TB weights must fit first. VRAM capacity is a hard constraint and does not change with the number of users. You cannot scale down from 8 cards to two or three just because only one person is using it.

As to whether 8 cards can stably achieve 25 Tokens/s: public data does not provide corresponding real-world test results, so we cannot responsibly guarantee it. An empirical reference is that single-stream LLM inference typically sits in the “tens of tokens per second” magnitude, making 25 Tokens/s a reasonable target. However, decoding speed is influenced by multiple factors including VRAM bandwidth, tensor parallelism efficiency, and expert activation patterns. You must run stress tests using your actual context length before going live.

3.3 Tier 3: 10 Concurrent Users, Each at 25 Tokens/s

This tier implies the system must stably provide an aggregate output of 250 Tokens/s.

Interestingly, the VRAM configuration may not need to be increased compared to Tier 2. According to estimates from public sources, an 8-card node can host approximately 40 concurrent requests at a medium context length. If the context is not at the million-token level, the hostable quantity will be even higher. In other words, 10 concurrent users alone will not force you to deploy a second server.

The real challenge lies in scheduling and throughput: when 10 streams are generating simultaneously, can you guarantee that each stream maintains 25 Tokens/s? This depends on continuous batching efficiency, request scheduling strategies, and VRAM bandwidth contention. These are performance metrics rather than capacity metrics and cannot be deduced purely from VRAM capacity; they must be tested. During stress testing, it is recommended to monitor:

• Aggregate throughput (Tokens/s);
• Time To First Token (TTFT);
• Average and tail latency (P99) per token.

A summary of the three tiers is as follows:

Tier	Core Requirement	Hard Constraint	Starting Configuration	Primary Risk
Just Running	Fit the weights	Capacity ≥1.5TB	8× next-gen large-VRAM cards	Older cards require de-quantization, degrading performance
Single user 25 tok/s	Streaming experience	Capacity + Bandwidth	Same as above, cannot be scaled down	Lacks empirical data; requires stress testing
10 concurrent × 25 tok/s	250 tok/s aggregate	Throughput and scheduling	Single node can host ~40 concurrent requests	Batch scheduling; requires stress testing

4. The Third Calculation: How to Calculate the Costs

These data-center-grade GPUs generally do not go through retail channels. For most teams, the realistic option is hourly renting via cloud platforms. Estimating using common industry pricing levels (numbers are examples only; refer to actual cloud platform quotes at the time):

• Assume a next-generation accelerator card has a 12-month reserved price of $4.76 per card per hour;
• An 8-card node then costs $38.08 per hour;
• Calculating continuous operation at 720 hours per month: $38.08 × 720 ≈ $27,400 per month, or about $330,000 annually.

And this is just the compute bill itself. Storage, networking, inter-machine communication, and operations staff must be accounted for additionally.

Calculating the reverse is equally important: if API call volume is not extremely high and there are no strict data privacy mandates, directly using cloud-based managed model services (billed per token) is almost always more cost-effective. Self-hosting only becomes viable when the call volume is sufficiently large or when data must remain strictly on-premise. For instance, cloud providers like Baidu AI Cloud typically offer both paths: GPU cloud servers (for self-deployment) and managed model inference services (billed per API call). The cost structures are completely different, making it worthwhile to calculate both in parallel during the project initiation phase before making a decision.

5. Pre-Project Checklist

Converging the methodology of this article into a universal checklist, applicable to the private deployment evaluation of any large model:

1. Check the total parameter count and native precision to calculate weight VRAM;
2. Check the model structure (layers, KV heads, attention variants) to estimate KV Cache;
3. Define target context length and concurrency, reserving a 10%~20% margin;
4. Verify if the GPU generation natively supports the target precision (e.g., FP4/FP8); beware of “can fit but can’t run well” solutions;
5. Cross-reference the official deployment guidelines of mainstream inference frameworks to confirm the starting card count;
6. Stress test aggregate throughput, Time To First Token, and tail latency using real business workloads;
7. Calculate cloud rental costs and managed service costs in parallel to choose the path with the lowest total cost of ownership.

6. Conclusion

The local deployment of trillion-parameter models is essentially an equation with four variables: “VRAM Capacity × Precision Support × Throughput Scheduling × Budget.” The VRAM calculation determines if it can run; precision support determines how well it runs; scheduling and stress testing determine if concurrency promises can be fulfilled; and the cost calculation determines if the project is worth doing. Only after calculating all four accounts clearly should you proceed—otherwise, you might find that the real problems have just begun after renting that 8-card server.