How to Deploy a Trillion-Parameter LLM Locally: Hardware and Cost Breakdown

1. Why Is the Deployment Barrier So High?

The new generation of flagship large language models generally adopts the Mixture-of-Experts (MoE) sparse architecture. Taking the model discussed in this article as an example: the total parameter count reaches 2.8 trillion, but each token actually activates only about 104 billion parameters. This design of “massive total parameters, sparse activation” brings two direct consequences:

1. Compute pressure is diluted, but capacity pressure is not. Each token only needs to compute about 4% of the parameters, meaning the compute requirement for feedforward calculations is much lower than that of a dense model of the same scale. However, during inference, any single request might be routed to any expert. Therefore, the weights of all 2.8 trillion parameters must permanently reside in VRAM and cannot be paged from disk on demand—otherwise, decoding latency would spiral out of control.

2. Precision design determines the memory baseline. The good news is that this model natively uses 4-bit floating-point weights (a block floating-point format). 4-bit is the official release precision, not a “patch-style quantization” that compresses FP16 down to 4-bit post-hoc, ensuring numerical quality. Based on this, we can calculate the initial cost:

   Model weights alone ≈ 2.8 trillion × 4 bit ÷ 8 ≈ 1.4 TB

However, 1.4TB is just the lower limit. On top of this, you must add:

• Activations and runtime overhead: The inference framework itself, communication buffers, and driver contexts usually require tens of GBs to be reserved.
• KV Cache: Its size ≈ 2 (K and V) × number of layers × number of KV heads × head dimension × sequence length × concurrency × bytes per element. The longer the context and the higher the concurrency, the faster this expands.
• Redundancy from parallelism strategies: Certain sharding and replication schemes will consume additional VRAM.

A comprehensive estimate shows that a single-user scenario actually requires about 1.5TB of VRAM. Remember this number; it is the anchor for all subsequent hardware sizing discussions.

2. Three Tiers of Requirements, Three Deployment Strategies

Tier 1: As Long As It Runs, Speed Is Not a Concern

Let’s do a theoretical capacity deduction first:

• Using the previous generation of flagship data center cards with 80GB per card: 1.5TB ÷ 80GB ≈ 19 cards minimum.
• Using the large memory version with 141GB per card: 1.5TB ÷ 141GB ≈ 11 cards minimum.

However, “fitting the model” does not mean “running it well.” There is an easily overlooked pitfall here: previous generation cards do not support native FP4 computation. Although the weights are stored in 4-bit, they must be de-quantized back to higher precision for matrix multiplication during execution. This de-quantization consumes extra bandwidth and compute, significantly discounting actual throughput. In other words, the theoretically feasible 19-card or 11-card setups offer poor cost-effectiveness.

A more realistic minimum configuration is an 8-card node of the new generation of data center GPUs (with around 288GB of VRAM per card, and native support for FP4 data formats). This is not just the author’s judgment—in the official deployment documentation of mainstream inference frameworks, the reference configuration for models of this scale is likewise 8-card parallelism.

Conclusion: Even if the goal is just “to make it run,” this is already a server-grade project. Cabinet power supply, cooling, and high-speed interconnects between cards must meet data center standards. Ordinary workstations are completely out of the question.

Tier 2: Single-User Output at 25 token/s

The hardware starting point remains the 8-card new generation node. The reason is straightforward: the bottleneck is primarily VRAM capacity, not compute. The 1.4TB of weights must fit first; you cannot shrink the node down to two or three cards just because “only one person is using it”—if it doesn’t fit, it doesn’t fit.

Is 25 token/s realistic? We can do a rough calculation based on the physics of decoding: during the decoding phase, to generate each token, the approximately 104 billion activated parameters (about 52GB in 4-bit) must be read entirely from VRAM. The aggregated VRAM bandwidth of an 8-card node is on the order of tens of TB/s, so the theoretical upper limit is “tens of tokens per second.” After deducting scheduling, communication, and de-quantization overhead, a single stream falling to tens of tokens per second is a reasonable expectation. Therefore, 25 token/s is a “reachable” target, not an aggressive metric.

But it must be emphasized: always perform stress testing with real business context lengths before going live. The longer the context, the heavier the attention computation and KV reads. Speeds in long-text scenarios can differ several-fold from short-text scenarios.

Tier 3: 10-Way Concurrency, Each at 25 token/s or Above

This means the system needs to stably output an aggregated throughput of 250 token/s. Intuitively, one might think doubling concurrency tenfold requires adding more machines. Interestingly: not necessarily.

• Capacity dimension: Based on VRAM estimation, an 8-card node can theoretically host around 40 concurrent requests. If the business context is not at the million-token level, the number of hosted concurrent streams will be even higher. 10-way concurrency is far from hitting the ceiling, so there is no need to bring in a second server for this reason.
• Throughput dimension: The real issue is whether, when 10 streams are generating simultaneously, you can guarantee that each stream maintains 25 token/s. This is a scheduling and throughput metric, dependent on the inference engine’s continuous batching efficiency, KV Cache management strategies (paged management, prefix caching), and the load balancing of expert routing. This cannot be estimated solely by VRAM capacity; it must be tested empirically.

You can conduct concurrency stress testing using the following approach (pseudocode):

for concurrency in [1, 5, 10, 20]:
    Simultaneously initiate N concurrent requests, using the average context length of the real business as input
    Record per-stream: First Token Latency, end-to-end output speed
    Pass criteria: All concurrent stream speeds ≥ 25 tps, and First Token Latency P99 meets the target

In terms of engineering, you can also implement: request queuing and rate limiting to protect tail latency; reusing system prompt prefix caching; monitoring per-stream speeds and auto-scaling upon degradation.

3. The Money Issue: How to Calculate Cloud Rental

All the aforementioned GPUs are data center-grade products. Individuals and SMEs generally do not have procurement channels, making cloud rental the only realistic path. Mainstream cloud platforms (including Baidu AI Cloud and other domestic and international providers) offer GPU cloud hosts/bare-metal instances. Billing is divided into two tiers: pay-as-you-go and reserved/annual/monthly discounts. Reserved prices are typically significantly lower than pay-as-you-go prices.

Here is a calculation framework (numbers are examples only; actual rates depend on real-time quotes from each platform):

Assuming the reserved price for a new generation large-memory card is $4.76 per card per hour for 12 months,
8-card node = 4.76 × 8 ≈ $38.08 per hour;
If running 24/7: 38.08 × 24 × 365 ≈ $333,000 per year.

Several ideas for cost reduction:

1. On-demand start/stop: If only used for R&D and testing, there is no need to run it 24/7. Phased usage can compress the annual cost by an order of magnitude.
2. Spot/Preemptible instances: Suitable for fault-tolerant stress testing and offline batch processing, but you must accept that instances can be reclaimed at any time.
3. ROI comparison: Self-hosted vs. Managed API: If concurrency doesn’t max out over the long term, directly calling a cloud-hosted LLM service (billed per token) is likely much cheaper than maintaining an 8-card node yourself. Self-hosting only makes sense when concurrency scale, data privacy, or customization needs genuinely justify it.
4. Do not forget hidden costs: The storage and distribution of the 1.4TB weights (using object storage to temporarily hold model files), node operations and maintenance, and downtime windows for version upgrades all cost real money.

4. Pre-Deployment Checklist

Let’s distill the above into an actionable checklist:

1. Capacity accounting: Weights 1.4TB + Activations/KV/Runtime ≈ 1.5TB. Recheck the KV Cache increment based on target concurrency.
2. Hardware selection: Prioritize current-generation data center cards that natively support FP4. Start with 8 cards. Avoid forcing previous generation cards without FP4 support to handle de-quantization.
3. Parallelism strategy: Confirm that the inference engine supports a combination of tensor parallelism and expert parallelism, and leave room for parameter tuning.
4. Stress testing validation: Use both short contexts and the longest business context to test single-stream speed, First Token Latency, and per-stream speed under concurrency.
5. Observability: Integrate monitoring and alerting (GPU utilization, VRAM watermarks, per-stream output speed, queue length) and set degradation thresholds.
6. Cost review: Recalculate based on “reserved price × number of cards × actual runtime,” and compare it with managed service quotes before initiating the project.

5. Final Thoughts

Returning to the initial three-tier question, the answers can be compressed into three points:

• To run it: Starts with an 8-card new generation data center GPU node. This is a server-grade project, not a workstation setup.
• Single-user 25 token/s: The same configuration works. The key is to stress test with real business contexts before going live.
• 10 concurrency × 25 token/s: VRAM capacity is highly likely sufficient. Success depends on scheduling and throughput; empirical testing has the final say.

Budget-wise, the reserved cost of an 8-card node is on the order of tens of dollars per hour, reaching hundreds of thousands of dollars annually if run 24/7. Define the SLA first—how many concurrent streams, what speed, what context length—and then reverse-engineer the hardware and budget. This is the only correct approach to avoid making blind project decisions.