How to Deploy a Trillion-Parameter LLM Locally: Hardware and Cost Analysis

1. The Bottom Line

Running the current generation of trillion-parameter MoE (Mixture-of-Experts) LLMs locally is not as simple as buying a few consumer-grade flagship GPUs. Even if the goal is merely to “get an output,” the hardware threshold is firmly at the server level. The model weights alone require approximately 1.4TB of VRAM. Factoring in runtime overhead, activations, and KV Cache, a single-user scenario demands a budget of around 1.5TB of VRAM.

This implies two things: first, this is an entry-level problem for 8-card data center servers, not a configuration question for high-end workstations; second, the core of the decision is not “which GPU to choose,” but rather calculating the VRAM, performance, and cost budgets first. Let’s break this down into these three steps.

2. The First Calculation: How 1.4TB VRAM is Derived

1. Sparse Activation Saves Compute, Not VRAM

These models utilize a Mixture-of-Experts (MoE) architecture: the total parameter count is roughly 2.8 trillion, but only about 104 billion parameters are activated per token. Many developers mistakenly infer from this that “VRAM is only needed for the activated portions.”

The router dynamically selects experts for each token, meaning any expert could be hit by the next request. To provide online inference services, all expert weights must remain resident in VRAM. Otherwise, if routed to an unloaded expert, weights must be temporarily moved from system memory or even disk, causing latency to instantly spiral out of control. Therefore, the true benefit of sparse activation is that the compute per token is reduced to the level of a 100-billion-class dense model, while VRAM usage is still calculated based on the total parameter count.

2. Native 4-bit Precision is Good News

This new generation of models natively uses the microscaled floating-point 4-bit format (MXFP4) for weight storage. Note that this is the official design precision, with training and inference pipelines optimized around it, rather than a “second-hand quantization” where FP16 weights are compressed to 4-bit after the fact. This offers two main advantages:

• Controlled precision loss: Output quality aligns with official benchmarks, eliminating the risk of tuning quantization parameters yourself.
• Hardware efficiency: If the inference hardware natively supports FP4 computation, it bypasses the de-quantization step, saving both VRAM bandwidth and compute power.

3. From Weights to Available VRAM

Beyond weights, space must be reserved for activations, runtime overhead, and the KV Cache. The KV Cache footprint can be estimated using a general formula: approximately “2 × number of layers × hidden dimension × context length × precision bytes × concurrency.” While the exact value depends on the model architecture, it is certain that longer contexts and higher concurrency increase this overhead. In long-context scenarios, it is not uncommon for this to reach the hundred-GB level.

A schematic estimation code is as follows:

# VRAM estimation (schematic, decimal units)
total_params = 2.8e12                     # Total parameter count
weight_TB = total_params * 4 / 8 / 1e12   # ≈ 1.4 TB (weights only)
# On top of this, add:
# + Runtime and VRAM fragmentation: approx. 5%~10%
# + Activations and KV Cache: scales with context length and concurrency
# Actual single-user requirement ≈ 1.5 TB

The 1.5TB figure is the cornerstone for deriving all subsequent configurations.

3. Three Deployment Tiers: From “Runnable” to “Usable”

Tier 1: Just Run It, Ignore Speed

Theoretical Capacity Solution: Use previous-generation data center cards to maximize capacity. You would need about 19 cards of the 80GB version or about 11 cards of the 141GB large-VRAM version to cover the 1.5TB requirement.

However, “fitting the model” does not mean “running it well.” Previous-generation cards do not support native FP4 computation. MXFP4 weights must be de-quantized to FP8/BF16 at runtime before participating in calculations. De-quantization consumes additional compute and VRAM bandwidth, and intermediate tensors squeeze the already tight capacity, significantly discounting effective performance. This path works, but the cost-to-performance ratio is poor.

A More Realistic Solution: The current generation of flagship data center accelerators—featuring roughly 288GB of VRAM per card with native FP4 support. Eight cards provide approximately 2.3TB of VRAM, comfortably accommodating the 1.5TB baseline while leaving room for concurrency and long contexts. Mainstream inference frameworks’ publicly available reference deployments also start with 8-card parallel setups, corroborating this judgment.

As for the “extreme budget” route: offloading part of the weights to CPU memory or NVMe SSDs, transferring them in batches over PCIe. This can indeed make a standard server “run,” but decoding speed typically drops to single-digit tokens/s. It is only suitable for offline verification and holds no value for production services.

Conclusion: Even if you only want it to “run,” this is a server-grade project. Consumer-grade GPUs offer only 24~32GB of VRAM each. Even if you fill an 8-card workstation chassis, the total capacity is less than 300GB—an order of magnitude less than the 1.4TB weight requirement. They are directly disqualified.

Tier 2: Single-User at 25 token/s

The hardware starting point remains 8 current-generation flagship cards. The reason is straightforward: for a model of this scale, the first bottleneck is not compute, but the fact that the 1.4TB weights must physically fit. You cannot scale down from 8 cards to 2 or 3 just because “only one person is using it”—the weights simply won’t fit.

Can 8 cards stably output 25 token/s? Here is a reality check: based on public community tests, the single-stream decoding speed for models of this scale generally sits in the “tens of tokens per second” range. 25 token/s is a reasonable target, but actual performance depends heavily on three variables:

• Context length: During decoding, generating each token requires reading the entire KV Cache. Under long contexts, speed will noticeably drop.
• Precision path: The bandwidth difference between native FP4 computation and runtime de-quantization directly impacts decoding speed.
• Inference framework scheduling: Kernel optimization and memory management differences across frameworks can cause multi-fold throughput gaps.

Therefore, before going live, you must stress-test using your actual context length. Do not extrapolate long-context experience from short-context benchmarks.

Tier 3: 10 Concurrent Streams × 25 token/s

Translated into requirements, the system must stably output an aggregate throughput of 250 token/s.

Interestingly, the VRAM configuration may not need to be higher than Tier 2. Public benchmarks suggest that an 8-card node can theoretically host around 40 concurrent requests. If your scenario does not involve million-token long contexts, the hostable number will be even higher. In other words, 10 concurrent streams alone will not force you to buy a second server.

The real challenge lies in throughput and scheduling: when 10 streams are generating simultaneously, can each maintain 25 token/s? This is a performance metric, not a capacity metric, and depends on:

• The scheduling efficiency of continuous batching: whether it can efficiently batch new and old requests.
• Paged management and fragmentation control of the KV Cache: avoiding the awkward situation of “having capacity left, but unable to form a contiguous block.”
• Queuing and preemption strategies when mixing long and short requests: preventing one ultra-long request from dragging down global latency.
• Whether inter-card interconnect bandwidth can support the communication overhead of tensor parallelism and expert parallelism.

These factors must be tested empirically; they cannot be deduced from VRAM capacity. During stress testing, it is recommended to monitor three metrics simultaneously: aggregate throughput, single-stream token/s, and P99 latency. If any fails to meet the standard, roll back and tune. If concurrency continues to grow or higher rates are needed, you will have to move towards multi-node interconnects (a combination of expert parallelism + tensor parallelism + pipeline parallelism), which demands lossless inter-node networks and significantly doubles the cost.

Three-Tier Quick Reference Table:

Tier	Requirement	VRAM Baseline	Recommended Configuration	Key Risks
1	Just run it	1.5TB	8× current-gen large-VRAM cards (~2.3TB)	Old card solutions suffer de-quantization loss
2	Single-user 25 token/s	1.5TB	Same as above, starting at 8 cards	Speed drops under long contexts
3	10 streams × 25 token/s	1.5TB	Single 8-card node hosts ~40 streams	Scheduling and aggregate throughput require testing

4. The Second Calculation: Evaluating Costs

1. Why Cloud is the Only Option

The configurations mentioned above all utilize data center-grade accelerators. These are not sold to individuals at retail, and standard teams lack the facilities, power supply, and operational capabilities to host them. Thus, the only realistic path is renting cloud GPU instances. When reviewing prices, note the difference between pay-as-you-go and reserved instances, as the price gap is usually substantial.

2. Converting Hourly Rates to Monthly and Annual Bills

Taking the publicly available reserved instance pricing from an overseas cloud platform as an example: a current-generation large-VRAM card costs about $4.80/card/hour, making an 8-card node approximately $38/hour. Calculated by different usage intensities:

• Low-frequency usage (8 hours/day, 22 working days/month): ~$6,700/month. Shut down when finished to keep costs controlled.
• 7×24 constant deployment: ~$27,000/month, translating to roughly $330,000/year.

These numbers illustrate two points: First, low-frequency usage is acceptable, but you must configure automatic start/stop scripts; otherwise, idle time is pure money burning. Second, 7×24 constant deployment is extremely expensive. Before committing, think clearly about whether your business truly needs to exclusively occupy a whole machine.

Don’t forget hidden costs: the 1.4TB weight files require object storage for distribution and version management. During stress testing and runtime, you need monitoring, alerting, and logging services as a fallback. These correspond to standard categories in general cloud product ecosystems and must be included in the budget.

3. Build, Rent, or Call an API Directly?

• Data must remain on-premises, high-frequency calls: Self-build or use dedicated instances. Start with an 8-card node. For long-term loads, negotiate reserved pricing.
• Data-sensitive but low-frequency calls: Use cloud pay-as-you-go instances with scheduled start/stop to reduce idle costs to zero.
• No strict compliance constraints: Directly call cloud-hosted LLM services and bill by token. This is almost always an order of magnitude cheaper than self-hosting and lets you focus your energy on the business logic.

5. Action Checklist Before You Start

1. Calculate the VRAM budget: Weights (total params × 4bit ÷ 8) + runtime overhead + activations and KV Cache. Leave an additional 20% margin overall.
2. Verify hardware precision support: Ensure the target GPU natively supports FP4/MXFP4 computation to avoid de-quantization losses.
3. Capacity ≠ Performance: Stress test with your actual context length and concurrency model. Simultaneously monitor aggregate throughput, single-stream speed, and P99 latency.
4. Optimize scheduling before scaling concurrency: The gains from optimizing continuous batching, paged caching, and queuing strategies often outweigh simply adding more machines.
5. Estimate costs based on actual usage hours: Calculate pay-as-you-go and reserved prices separately. Negotiate reserved pricing for long-term constant deployment, and always use automatic shutdown for low-frequency usage.

In summary: For local deployment of trillion-parameter models, the first step is not selecting a GPU, but calculating the 1.4TB VRAM budget, the 25 token/s performance budget, and the monthly cost of tens of thousands of dollars. Only proceed when all three budgets check out.