Evaluating LLM Local Deployment Costs: From VRAM Calculation to Architecture Selection

1. Why “Local Deployment” is Back on the Agenda

In recent years, the capabilities of open-source Large Language Models (LLMs) have improved rapidly, with top-tier models expanding from a few billion parameters in the early days to hundreds of billions today. For enterprise engineering teams, the appeal of local deployment lies in keeping data in-house, ensuring predictable long-term inference costs, and enabling deep customization. However, the barrier to entry has risen just as fast—a small model that once ran on a single consumer-grade graphics card now often requires an inference cluster costing millions.

This presents a very practical challenge: when a team or management proposes deploying an open-source LLM in a private data center, how can technical leads quickly and rigorously answer: “How much will it cost, how many GPUs do we need, and is it worth it?” This article provides a reusable evaluation framework to turn the vague feeling of “it seems expensive” into justifiable numbers.

2. The First Calculation: Estimating VRAM Requirements

VRAM is the starting point for all cost estimations. A comprehensive calculation must include at least three components:

1. Model Weights: The basic estimation formula is VRAM ≈ Parameter Count × Bytes per Parameter. Loaded in FP16/BF16 precision, each parameter takes 2 bytes; INT8 quantization takes 1 byte; INT4 quantization takes 0.5 bytes. For example, a 10-billion parameter model requires about 20GB of VRAM just for weights in FP16.
2. KV Cache: During inference, the attention key-value pairs must be cached for each request. Its size is directly proportional to the concurrency level and context length. In long-context scenarios, the KV Cache overhead can approach or even exceed the weights themselves, making it the most easily underestimated part of VRAM planning.
3. Runtime Overhead: The inference framework itself, driver context, and activation values require an additional 10% to 20% buffer.

A sample calculation code snippet is shown below:

def estimate_vram_gb(params_b: float, bits: int = 16,
                     num_layers: int = 0, num_kv_heads: int = 0,
                     head_dim: int = 0, ctx_tokens: int = 8192,
                     concurrency: int = 1) -> float:
    # 1) Weight VRAM: Parameter count × bytes per parameter
    weight_gb = params_b * (bits / 8)
    # 2) KV Cache: 2(K/V) × num_layers × num_kv_heads × head_dim × num_tokens × bytes
    kv_per_token = 2 * num_layers * num_kv_heads * head_dim * (bits / 8)
    kv_gb = kv_per_token * ctx_tokens * concurrency / 1024**3
    # 3) Runtime buffer approx. 15%
    return (weight_gb + kv_gb) * 1.15

It is worth emphasizing that current flagship open-source models generally adopt the MoE (Mixture of Experts) architecture: the total parameter count is massive, but only a subset of parameters is activated per token. The implication for deployment is that weight VRAM must be allocated based on the total parameter count (all experts must reside in memory), while compute consumption is closer to the activated parameter count. This explains why such models “can handle the inference compute but cannot fit into VRAM.” Taking a MoE model with hundreds of billions of total parameters as an example, just the weights in FP16 require hundreds of GBs of VRAM. A single GPU cannot accommodate this, necessitating multi-GPU or even multi-node tensor splitting.

3. The Second Calculation: Scaling from Single GPU to Cluster

Cluster size is determined by two constraints, and you must take the larger of the two:

• Fits in Memory: The total amount of (Weights + KV Cache) divided by the VRAM of a single GPU gives the minimum number of GPUs required. Concurrently, the communication and redundancy overhead introduced by tensor parallelism and expert parallelism means actual usable VRAM is discounted.
• Runs Fast: The concurrency required by the business, along with Time-to-First-Token (TTFT) and throughput metrics, dictates the actual number of GPUs needed. For production environments facing real traffic, the required GPU count is typically significantly higher than the theoretical minimum of “fitting in memory.”

Additionally, high-speed interconnect networks between nodes are indispensable for cross-machine deployments. The cost of switches and optical modules accounts for a significant portion of the overall budget and is easily overlooked.

4. The Third Calculation: Total Cost of Ownership (TCO)

Many teams making a budget for the first time only calculate the cost of the GPUs. The actual TCO must include at least:

• Compute Hardware: Entire GPU servers (including CPU, system memory, high-speed storage, etc.);
• Network and Storage: High-speed switches, optical modules, distributed or object storage;
• Data Center and Power: High-power racks, cooling, power redundancy—AI servers have significantly higher single-unit power consumption than standard servers;
• Software and O&M: Inference framework tuning, monitoring and alerting, driver and firmware maintenance, dedicated engineering labor;
• Depreciation and Iteration: GPU generations update quickly, typically depreciated over 3 to 5 years, and new model generations may demand higher configurations.

Based on public industry discussions and common estimation standards, there can be a multiple-fold difference between the “minimum viable configuration” (e.g., low-precision quantization, low concurrency, validation use only) and the “recommended production configuration” (real traffic-facing, with redundancy and high availability) for flagship open-source models. These two budget tiers often fall into the hundreds of thousands and millions of dollars, respectively. This is exactly why rigorous calculation must precede project initiation: underestimate, and it won’t run; overestimate, and it won’t be approved.

5. On-Premise vs. Cloud: A Decision Checklist

Major cloud providers (such as Baidu AI Cloud) typically launch hosted inference services for popular open-source models shortly after their release, sometimes achieving “Day 0” adaptation—meaning they are ready for API calls on the day of release. For most enterprises, it is recommended to make decisions using the following checklist:

Consideration	Leans On-Premise	Leans Cloud
Data Compliance	Strict offline requirements, end-to-end autonomous control	Can choose isolated options like dedicated resources
Usage Scale	Large-scale, stable, long-term	Small to medium scale, highly fluctuating
Team Capability	Has a dedicated infrastructure team	Lacks operations personnel
Financial Form	Can absorb heavy upfront capital expenditure	Prefers pay-as-you-go, elastic spending
Iteration Speed	Model versions are relatively fixed	Needs to frequently keep up with new models

The advantage of cloud solutions lies in elastic scaling, zero-maintenance (fully equipped with container platforms, monitoring, logging, etc.), and the ability to quickly adopt new models. The advantage of on-premise deployment is the amortized unit cost for long-term, large-scale inference and complete data control. A pragmatic path is: first validate business value on the cloud at a smaller scale, then decide whether to sink stable traffic to a self-built cluster or adopt a hybrid “cloud elasticity + on-premise baseline” architecture long-term.

6. Post-Deployment: Objective Model Evaluation

Deployment is just the beginning. Teams still need an objective evaluation method to avoid the trap of “great benchmarks, terrible business performance”:

1. General Benchmarks: Use community-recognized evaluation sets covering reasoning, coding, math, and long-text dimensions as a horizontal reference.
2. Custom Business Evaluation Sets: Extract hundreds to thousands of samples from real business scenarios and annotate them. This reflects actual performance better than public leaderboards.
3. Online Canary Release and Human Review: Validate with a small fraction of traffic, focusing on engineering metrics such as hallucination rate, instruction following, and output format stability.
4. Cost-Performance Curve: Record latency and throughput under different concurrency and precision levels to find the optimal balance between quality and cost.

7. Budget Justification Strategy for Management

If you need to request a budget from the company, a persuasive proposal typically includes: requirements and scale forecasting, two to three tiers of configuration options (minimum viable / recommended / redundant), detailed TCO and depreciation metrics, comparative calculations with cloud alternatives, and a phased implementation plan (start with small-scale validation, then scale as needed). Reframing “how much money do we need” into “what capabilities each plan buys, what the risks are, and what our fallback options are” will significantly increase the approval rate.

8. Conclusion

Deploying LLMs locally is not merely a question of “do we have GPUs,” but a systematic engineering effort encompassing VRAM calculation, cluster design, TCO evaluation, and business validation. Calculate the costs clearly first, then choose the deployment path, and finally validate the business value. This approach ensures that every penny of the budget is spent where it matters most.