Cost Analysis of Deploying Trillion-Parameter Models Locally: From VRAM Accounting to Three-Tier Configuration

1. The Bottom Line: This Is Not a Workstation Task

When discussing whether a trillion-parameter model can run locally, a common first reaction is to “buy a few flagship consumer GPUs.” This intuition completely falls apart when facing trillion-scale MoE (Mixture-of-Experts) models. Even if we set the bar at its absolute lowest—just wanting it to “run, regardless of speed”—the hardware threshold still firmly falls within the realm of server-grade nodes.

This article does not discuss procurement channels; instead, it focuses on a pure capacity and performance estimation, providing a reusable capacity planning methodology. Once you understand this framework, you can apply it to evaluate the deployment of any large model with similar precision and architecture.

2. The First Principle: All Weights Must Reside in VRAM

A common misconception about trillion-scale MoE models is: since each token only activates a small fraction of the parameters, does the VRAM only need to hold that activated portion?

No. Sparse activation reduces compute, not storage. The routing mechanism means that any token could be distributed to any expert, so all expert weights must permanently reside in accessible VRAM. Taking a configuration with 2.8 trillion total parameters and approximately 104 billion activated parameters per token as an example, the activation ratio is about 3.7%, but the storage demand is still calculated based on the full 2.8 trillion.

Fortunately, there is a key premise in the precision design: this type of model natively uses the 4-bit floating-point (MXFP4) format. 4-bit weights are the official designed precision, not a forced post-training quantization compression from FP16. This means the theoretical lower limit of the weight volume can be calculated directly using 4 bits:

Weight VRAM ≈ 2.8e12 × 4 bit ÷ 8 bit/Byte ≈ 1.4 TB

This 1.4 TB is solely for the weights themselves. Actual deployment must also account for activations, runtime overhead, KV Cache, and state space. A comprehensive estimate suggests that a single-user scenario requires approximately 1.5 TB of GPU VRAM in practice. This number is the starting point for all subsequent configuration discussions and the decisive variable for the entire cost structure.

3. Three Deployment Tiers and Corresponding Configurations

By dividing the requirements into three tiers, it becomes clearer that “capacity” and “throughput” are two different types of problems.

Tier 1: Just Run It, Ignore Speed

From a pure capacity perspective, one could theoretically stack 1.5 TB using previous-generation high-capacity cards: about 19 x 80GB cards, or 11 x 141GB cards. However, there is an easily overlooked pitfall here—previous-generation GPUs do not natively support MXFP4 execution and require runtime de-quantization. Fitting the model into the VRAM does not mean it will run efficiently; the extra overhead and precision path introduced by de-quantization must be evaluated.

A more realistic starting point is a new-generation 8-card node that supports this precision format: 8 x 192GB-class accelerator cards, providing a total capacity of about 1.5 TB, which exactly covers the weights plus runtime overhead. This is also the parallelization scheme given in the official deployment documentation of mainstream inference frameworks—an 8-card parallel setup is the standard starting point for models of this scale.

The conclusion is clear: even just to “make it run,” this is already a server-grade project, not a standard workstation configuration.

Tier 2: Single User at 25 Tokens/s

The hardware starting point for this tier is basically the same as Tier 1, and the reason is still capacity rather than compute: the 1.4 TB of weights must be accommodated first. You cannot shrink the node down to two or three cards just because there is only a single user.

It must be emphasized that whether an 8-card node can stably achieve 25 Token/s cannot be guaranteed without empirical testing data. The typical magnitude for single-stream large model inference is “tens of tokens per second,” making 25 Token/s a reasonable target. However, before going live, stress testing with real context lengths is mandatory. The longer the context, the larger the KV Cache footprint, and the more easily the effective throughput will be dragged down.

Tier 3: 10 Concurrent Users, Each at 25 Tokens/s

This means the aggregate output must stably reach at least 250 Token/s. Interestingly, the VRAM configuration may not need to be higher than Tier 2: under conditions where the context is not extremely long, an 8-card node can typically host dozens of concurrent requests. 10 concurrent users alone does not force you to deploy a second server.

The real problem shifts to the throughput and scheduling level: when 10 streams are generating simultaneously, can you guarantee that each stream maintains 25 Token/s? This depends on batching efficiency, scheduling strategies, and VRAM bandwidth. These are performance metrics that must be tested empirically and cannot be reverse-calculated simply based on VRAM capacity. The capacity accounting determines “whether it fits,” while the throughput accounting determines “whether it runs”—these two must be verified separately.

4. Cost Estimation: Why Cloud Pricing Is the Only Benchmark

These accelerator cards are all data-center-grade products, and retail channels generally do not target individuals. Therefore, the logical path for cost estimation is hourly cloud platform billing. Taking the GPU instances of a major cloud provider as an example, calculated at a 12-month reserved price of approximately $4.76/GPU/hour:

8-card node hourly cost ≈ 4.76 × 8 ≈ $38.08 / hour
Monthly cost ≈ 38.08 × 730 ≈ $27,800 / month
Annual cost ≈ 38.08 × 8760 ≈ $334,000 / year

By this metric, the annual reserved cost is in the range of over three hundred thousand dollars. During actual selection, it is important to note that pricing varies greatly depending on the provider, billing model (on-demand/reserved/spot), and card model. The table above only provides a calculation framework, and actual numbers should be based on real quotes. If using on-demand billing, the cost for short-term stress testing is controllable, but the long-term steady-state cost will be significantly higher than the reserved price.

5. A Reusable Evaluation Checklist

By abstracting the above analysis into a methodology, you can follow these steps to evaluate any local large model deployment:

1. Calculate the weight lower limit: Total parameters × native precision bits ÷ 8 to get the weight volume. Be careful to distinguish between “native low precision” and “post-training quantization”—the latter introduces precision and runtime overhead.
2. Add runtime overhead: On top of the weights, add activations, KV Cache, and framework overhead. A single-user scenario typically requires reserving more than 10%.
3. Verify precision support: Confirm whether the target hardware natively executes that precision format; otherwise, evaluate the overhead of the de-quantization path.
4. Define configurations by tier: The capacity tier looks at total VRAM, the throughput tier looks at tested Token/s, and the concurrency tier looks at aggregate throughput and scheduling efficiency.
5. Calculate based on cloud pricing: Use hourly unit price × number of cards × duration to calculate on-demand and reserved rates respectively, then decide whether to use steady-state or elastic deployment.

6. Conclusion

The local deployment of trillion-scale MoE models is essentially a capacity problem where storage takes precedence over compute: 1.4 TB of weights is the hard threshold, 1.5 TB of VRAM is the realistic requirement for a single user, and an 8-card server node is the bare minimum starting point. Among the three deployment tiers, the first two are constrained by capacity, while the third is constrained by throughput and scheduling, with the latter verifiable only through stress testing. On the cost side, using cloud platform hourly rates as the baseline, the annual steady-state expense is in the range of hundreds of thousands of dollars. Crunching these numbers clearly is far more meaningful than simply discussing “how many cards to buy.”