Deploying Open-Source LLMs Locally: From VRAM Estimation to a Multi-Million Dollar Budget

Open Source Does Not Mean Free: The Real Cost of Running the Model

The recent wave of trillion-parameter Mixture-of-Experts (MoE) models from the open-source community has narrowed the capability gap with closed-source flagships. Many enterprises are now tempted to download the weights and deploy locally. However, it is crucial to recognize that open-source licenses waive licensing fees, not compute costs. Deploying a model with weights exceeding 1 TB involves a complete supply chain of GPUs, high-speed networking, and data center power.

Consider a representative scenario: a flagship open-source model released in low precision (FP8) requires about 1.5 TB of weight storage. A current top-of-the-line eight-GPU server typically offers just over 1 TB of total VRAM. This means a single server cannot even hold the weights, let alone reserve space for the KV cache, activation values, and framework overhead during inference. A rough calculation suggests the minimum viable investment for local deployment is around 8 million RMB (approximately $1.1 million USD), while a recommended production-grade configuration could reach 30 million RMB (approximately $4.2 million USD). The following sections break down this analysis.

First Principles: Calculating VRAM Requirements for Your Model

Total VRAM demand can be strictly defined with a simple inequality:

Total VRAM > Weights + KV Cache + Activations + Framework Overhead

1. Weights. The memory for weights is calculated as: Total Parameters × Bytes per Parameter. FP8 precision uses 1 byte per parameter, while BF16 uses 2 bytes. A 1.5 TB weight footprint implies a model with approximately 1.5 trillion parameters. A key characteristic of MoE architectures is that while total parameter count is massive, only a small subset is activated for each token. This explains why such models consume significant memory but are computationally fast.

2. KV Cache. This is often the most underestimated component. During autoregressive decoding, the Key and Value tensors for each layer must be cached for every token. The formula is:

KV Cache ≈ 2 × Num Layers × Num KV Heads × Head Dimension × Bytes per Precision × Sequence Length × Concurrency

This grows linearly with “context length × concurrency”: doubling the concurrency or context length doubles the cache requirement, independent of model size.

3. Activations and Framework Overhead. Inference frameworks require memory for activation tensors, communication buffers, and fragmentation, typically reserving an additional 10% to 20% of the total.

Here is a reusable estimation script:

def estimate_memory_gb(weights_gb, num_layers, kv_heads, head_dim,
                       seq_len, concurrency, kv_bytes=1, overhead=0.15):
    kv_per_token = 2 * num_layers * kv_heads * head_dim * kv_bytes
    kv_gb = kv_per_token * seq_len * concurrency / 1e9
    total = (weights_gb + kv_gb) * (1 + overhead)
    return {"weights": weights_gb, "kv_cache": round(kv_gb, 1),
            "total": round(total, 1)}
# Example: weights 1500GB, 120 layers, 8 KV heads, head dim 128,
# 32K context, 32 concurrent requests, KV cache in FP8
print(estimate_memory_gb(1500, 120, 8, 128, 32768, 32))
# => {'weights': 1500, 'kv_cache': 257.7, 'total': 2021.4}

The conclusion is direct: a model with 1.5 TB of weights, under 32 concurrent requests with a 32K context, requires approximately 2 TB of total VRAM. To support hundreds of concurrent requests and longer contexts, a baseline of 3-4 TB is a more realistic planning target.

When a Single Node Isn’t Enough: Parallelization and “Super Pods”

When single-node VRAM cannot accommodate the total requirement, the model must be distributed across multiple machines. Mainstream inference frameworks typically combine three strategies:

• Tensor Parallelism (TP): Splits weight matrices across GPUs so each card computes a portion. It requires the most frequent communication and is generally confined to the high-speed interconnect within a single node.
• Pipeline Parallelism (PP): Divides the model by layers, with each segment residing on a different machine. It is simpler to implement but suffers from utilization loss due to pipeline bubbles.
• Expert Parallelism (EP): The standard for MoE models, distributing different experts across devices. Tokens are routed via All-to-All communication between devices.

Cross-node parallelism introduces a new bottleneck: communication. Every step of TP/EP involves significant collective communication. If machines are connected only by standard Ethernet (e.g., 200Gbps), communication time can exceed compute time, leaving the cluster underutilized. This has driven the evolution of “super pods”: within a single rack or high-bandwidth domain, caching-coherent interconnects or high-speed switching fabrics link a dozen or more GPUs into a “logical single server.” This allows TP/EP to operate within the high-bandwidth domain, directly impacting economics. The effective utilization of a super-pod configuration is generally significantly higher than a standard multi-node setup with the same number of GPUs, although the exact margin depends on model architecture, framework implementation, and network topology.

Based on these factors, we can derive the hardware scale:

1. 1.5 TB of weights plus inference overhead requires at least 2TB VRAM, meaning a minimum of two 8-GPU servers just to store the model.
2. Production environments cannot run at 100% VRAM utilization; they require throughput headroom and expansion capacity. A minimum viable cluster should be planned with 4-6 nodes.
3. To meet production SLAs (high concurrency, multiple replicas, canary deployments, and fault tolerance), a cluster of 12-16 nodes is a starting point.

The Complete Cost Picture: Capital and Operational Expenditures

Capital Expenditure (CapEx):

Item	Description	Cost Reference
GPU Servers	Minimum of 4-6 nodes	Largest cost driver; a single 8-GPU flagship server typically costs 1.5-2.5 million RMB ($210k-$350k USD)
Network Equipment	High-performance switches (e.g., InfiniBand) and optical modules for collective communication	Approximately 8-12% of total hardware cost
Storage	Object storage and parallel file systems for weights, datasets, and logs	Approximately 3-5% of total
Data Center Modifications	Power upgrades, liquid/air cooling, racks, and cabling	5-15% depending on existing infrastructure

Operational Expenditure (OpEx):

1. Electricity: A fully loaded 8-GPU server draws roughly 10kW. With a PUE of 1.3 and an average industrial electricity price, the annual power cost per server is around 80,000 RMB (~$11k). An 8-node cluster will incur about 600,000-700,000 RMB (~$85k-$100k) per year in electricity, assuming typical inference workloads rather than 24/7 maximum load.
2. Personnel: At least 1-2 engineers specializing in distributed inference frameworks and GPU operations, plus network and data center support, representing a significant annual salary cost.
3. Depreciation and Spare Parts: GPU servers are typically depreciated over 3-5 years. High-end accelerators have a non-zero failure rate and lead time for replacement, so you should budget 5-10% of the total hardware purchase price annually for spare parts and maintenance.

Summing these items: a minimum viable configuration (4-6 servers + networking + data center modifications) falls in the 8-12 million RMB (~$1.1M-$1.7M) range. A production-recommended configuration (12-16 servers + redundant network + capacity headroom) naturally reaches 25-35 million RMB (~$3.5M-$5M). This is why public estimates place the local deployment threshold for such models at “minimum 8 million RMB, recommended 30 million RMB”—it is a reflection of production-grade availability requirements, not an exaggeration.

Self-Hosting, Renting GPUs, or Using an API: Choosing the Right Path

Dimension	On-Premises Deployment	Cloud GPU Instances	Managed API
Initial Investment	Millions of RMB	Hourly or reserved instance billing	Near zero
Data Compliance	Full control	Controllable within a private VPC	Must assess data egress risks
Elasticity	Poor; scaling requires procurement cycles	Good; scale up/down in minutes	Good; pay-per-use
Operational Complexity	High	Medium; requires managing framework and scheduling	Low
Best For	High concurrency, strict compliance, deep customization	Moderate concurrency, compliance requirements, rapid validation	Small to medium workloads, fast time-to-market

Practical guidance:

• If your daily token consumption is not in the hundreds of millions and you have no hard compliance constraints, a managed API is almost always more cost-effective.
• If you need private deployment but have fluctuating loads, cloud GPU resources are preferred. Major cloud providers offer mature solutions for GPU compute, container orchestration, object storage, and monitoring, allowing you to convert a large capital expense into operational expenditure and reduce the cost of trial and error.
• On-premises deployment only begins to make financial sense in a 3-year Total Cost of Ownership (TCO) analysis if you meet at least two of the following three conditions: stable, high concurrency; a hard requirement for data never leaving your premises; and a need for continuous fine-tuning and architectural modifications.

Budget Proposal Recommendations for Decision-Makers

To present a budget to management, argue from first principles rather than “everyone else is buying one.”

1. Start with business metrics: Define the target QPS, average input/output token length, and acceptable latency. These are the inputs for all calculations.
2. Derive resource scale: Use the methods and scripts in the VRAM section to calculate memory requirements and the number of servers, linking each server to specific business metrics.
3. Present two tiers: Distinguish between “minimum viable” and “recommended” configurations, explaining the specific differences in concurrency, availability, and fault recovery to avoid the need for emergency budget approvals later.
4. Attach a 3-year TCO comparison: Include electricity, personnel, depreciation, and maintenance across self-hosting, renting, and API routes.
5. Propose phased funding: Start with a small PoC, move to a scaled pilot, and then to production. Using measured data from each phase to justify the next investment makes the proposal easier to approve.

Conclusion

Open-source large models have made capability free, but they have made the cost of running that capability explicit. For enterprises, the real question is not “should we deploy an open-source model?” but “which path matches our load profile and compliance boundaries?” Calculating the VRAM, cluster size, power, and personnel line-by-line before deciding how much to spend is far more grounded in production reality than relying on any impressive demo.