How Much Does It Cost to Self-Host an Open-Source LLM? A Practical Cost-Estimation Framework

    When people ask what it costs to self-host an open-source large language model, the GPU price is almost always the first number they reach for—and almost always the wrong place to start. This article walks through a practical, bottom-up method for estimating the true cost: how to calculate weight and KV Cache VRAM requirements, why card count does not scale linearly, why memory bandwidth rather than FLOPS is the real inference bottleneck, and how to roll hardware, power, datacenter, network, and labor into a three-year TCO. It also covers the hidden costs most budgets miss, a utilization-based framework for choosing between self-built clusters and the cloud, and a tiered budgeting approach for getting spend approved. Two ready-to-use Python scripts for VRAM and TCO estimation are included.

    1. The GPU Bill Is Only Part of the Total

    When people ask “how much does it cost to self-host a large model,” almost everyone’s first answer is the price of the accelerators. But anyone who has taken a project through a full approval cycle knows that before the procurement list goes up, you need to answer five questions: how many cards, where they sit, where the power comes from, who operates them, and whether the card will still count as advanced three years from now. Those five questions map directly to five cost categories: compute hardware, datacenter and power, storage and network, labor and operations, and the sunk risk of depreciation and generational turnover.

    Laid out simply:

    Cost category Typical content Expense nature
    Compute hardware Accelerators, servers, CPU/memory, NICs, switches One-time (CapEx)
    Datacenter & power Racks, power redundancy, cooling, electricity, rack rental Ongoing (OpEx)
    Storage & network Model repository, training/inference data, vector store, logs, object storage Ongoing
    Labor & ops Inference optimization, platform engineering, SRE, on-call Ongoing, often the single largest item
    Iteration & depreciation Hardware depreciation, model-generation turnover, idle rate Hidden

    One pattern keeps proving itself: in a mid-sized self-built cluster, accelerators usually account for 50–65% of the three-year total cost of ownership (TCO). The remaining 35–50% is spread across power, datacenter, network, and labor. A budget that reports only the card price will almost certainly get additional funding requests during implementation.

    2. The First Hard Constraint: How Much VRAM Do You Actually Need

    2.1 Estimating the Weight Footprint

    Whether a model runs at all comes down to whether it fits in VRAM, and the weight estimate is deliberately simple:

    1. VRAM (weights) ≈ parameter count × bytes per parameter

    Bytes per parameter depends on the loading precision: BF16/FP16 is 2 bytes, FP8 is 1 byte, and 4-bit quantization is roughly 0.5 bytes (in practice you also add quantization metadata and dequantization overhead). A 70-billion-parameter model loaded in half precision needs about 140 GB for the weights alone—a single 80 GB card simply cannot hold it, no matter what. This is not a performance problem but a physical constraint, and it can only be solved by splitting across multiple cards or by quantizing.

    2.2 The Most Underestimated Factor: KV Cache

    More troublesome than the weights is the KV Cache. During autoregressive decoding, the Key/Value tensors for every token in every layer of the generated sequence must be cached, or every step would have to recompute the entire sequence. Its footprint scales with context length and with the number of concurrent requests, which makes it completely unlike the fixed footprint of the weights—it is the dynamic part that “eats” the remaining VRAM:

    1. KV Cache ≈ 2 × layers × KV heads × head_dim × sequence length × concurrency × bytes per element

    The key insight is that long context and high concurrency are two multipliers, not addends. When the context grows from 4K to 128K and concurrency goes from 1 to 32, the KV Cache can expand by three to four orders of magnitude and easily exceed the weights themselves. This is exactly why so many teams see “the model loads fine, but we OOM the moment concurrency kicks in.”

    2.3 A Drop-In Estimation Script

    The code below depends only on the standard library. It folds the two formulas above into a single tunable function so you can run quick estimates inside a planning document:

    1. def estimate_vram_gb(param_count_b, bytes_per_weight=2.0, layers=0,
    2. kv_heads=0, head_dim=0, seq_len=8192,
    3. batch_size=1, kv_bytes=2.0, overhead=0.15):
    4. """Rough estimate of the VRAM (GiB) needed for single-instance inference.
    5. Fill in layers / KV heads / head_dim according to the target model's architecture."""
    6. gib = 1024 ** 3
    7. weights = param_count_b * 1e9 * bytes_per_weight / gib
    8. kv_cache = (2 * layers * kv_heads * head_dim * seq_len * batch_size * kv_bytes) / gib
    9. # overhead: activation buffers, CUDA context, framework cache, and fragmentation reserve
    10. return round((weights + kv_cache) * (1 + overhead), 2)
    11. if __name__ == "__main__":
    12. # Example: 70B parameters, half precision, 80 layers, 8 KV heads, head_dim=128
    13. print(estimate_vram_gb(70, 2.0, 80, 8, 128, seq_len=32768, batch_size=16))

    Always keep a 10–20% margin. VRAM fragmentation, communication buffers, and the temporary tensors created during sampling are not in these formulas, and planning right at the theoretical value is one of the most common causes of production incidents.

    3. The Second Constraint: Card Count Doesn’t Scale Linearly

    Dividing the VRAM requirement by the VRAM per card gives you the lower bound on card count, not the optimal count. Multi-card parallelism adds very real overhead:

    • Tensor parallelism splits the weights of a single layer across cards, and every layer then needs a cross-card reduction. The finer the split, the higher the communication share and the more clearly per-card compute utilization drops. In practice, 4–8 cards within a single machine is a comfortable range.
    • Pipeline parallelism splits by layer and can scale across machines, but it introduces pipeline bubbles. Under load imbalance, overall throughput is dragged down by the slowest stage.
    • Interconnect sets the ceiling on parallelism. Intra-machine high-bandwidth links and ordinary cross-machine Ethernet differ by an order of magnitude. Once a model is large enough to require cross-machine tensor parallelism, the network becomes the primary bottleneck, and high-speed NICs and switches have to be added to the total cost.
    • Mixture-of-Experts (MoE) architectures carry larger total parameter counts at the same activated compute, so the VRAM pressure falls mainly on the weights—and expert parallelism makes all-to-all communication particularly sensitive.

    The conclusion is direct: you cannot skimp on network and interconnect spending, because that spending determines whether the cards you bought can actually be used.

    4. The Real Bottleneck Is Memory Bandwidth, Not FLOPS

    Inference has two phases, and their bottlenecks are completely different:

    • Prefill (processing the input) handles the entire prompt in parallel at once. It is compute-intensive, bound by FLOPS, and it determines time to first token (TTFT).
    • Decode (generating token by token) computes just one token per step, yet it must move all the weights and the KV Cache from VRAM to the compute units. It is memory-intensive, bound by VRAM bandwidth, and it determines time per output token (TPOT) and the typewriter-like smoothness of the output.

    From this, three budget-relevant conclusions follow:

    1. Don’t select hardware on nominal compute alone. For long-text batch generation, VRAM bandwidth and VRAM capacity often determine real throughput more than peak TFLOPS do.
    2. Concurrency is the main lever for throughput, but it comes at a cost. A larger batch amortizes memory access and significantly raises total tokens per unit time, but it also lengthens per-request decode latency. Before budgeting, confirm the service-level objective (SLO) with the business side: is this an acceptable “offline batch job,” or “online interaction” that requires first token within one second? The two cases can differ severalfold in card count.
    3. Quantization delivers two benefits and carries two risks. Dropping weights to 8-bit or 4-bit saves both VRAM and bandwidth, and in theory improves both concurrency and speed. But the precision loss is uneven across tasks—math reasoning, code generation, and structured extraction are especially sensitive to quantization. Before launch you must run a comparison on your own domain evaluation set rather than trusting generic leaderboards.

    Engineering optimizations—continuous batching, paged KV-cache management, prefix-cache reuse, speculative decoding—can all deliver meaningful throughput gains on the same hardware. Conversely, the absence of these optimizations can inflate your hardware budget by more than 30% out of thin air, and the labor to implement them is itself a cost line.

    5. Converting a Hardware Quote into a Three-Year TCO

    The quote for buying cards is a one-time number; TCO is a three-year one. Here is a calculation skeleton you can paste straight into a planning document:

    1. def three_year_tco(node_price, node_count, node_kw, pue, kwh_price,
    2. residual_rate=0.2, opex_rate=0.15):
    3. """Estimate the three-year TCO (currency unit matches the input).
    4. node_price covers the whole server, memory, NICs, etc.; node_kw is the whole-machine power draw (kW)."""
    5. capex_net = node_price * node_count * (1 - residual_rate) # deduct residual value
    6. annual_kwh = node_kw * pue * 24 * 365
    7. power_cost = annual_kwh * kwh_price * 3
    8. opex = capex_net * opex_rate * 3 # labor, rack space, network, spares, software subscriptions
    9. total = capex_net + power_cost + opex
    10. return {
    11. "Net compute investment": round(capex_net),
    12. "Three-year electricity": round(power_cost),
    13. "Three-year ops amortization": round(opex),
    14. "Three-year total": round(total),
    15. "Electricity share": f"{power_cost / total:.1%}",
    16. }
    17. if __name__ == "__main__":
    18. print(three_year_tco(800_000, 8, 10, pue=1.3, kwh_price=0.7))

    Two points about the parameters are worth calling out:

    • pue is the data center’s Power Usage Effectiveness, which captures the extra power drawn by cooling, power distribution, and other supporting systems. Air-cooled facilities typically run at 1.3–1.6; liquid cooling and efficient designs can go significantly lower. If you use colocation instead of building your own racks, this cost shows up directly in the rack-rental quote.
    • opex_rate is the operations amortization coefficient, converting platform engineering, SRE, on-call, spares, monitoring, and software subscriptions into an annualized ratio. It is the easiest item to leave out, and it is often the fastest-growing one over three years.

    Here is what the numbers feel like. Take an 8-card server priced at 800,000 yuan with a whole-machine power draw of about 10 kW: three-year electricity runs into the low hundreds of thousands of yuan on its own. For ten such machines, the electricity bill alone is enough to buy another whole server. And in real projects, the 8-card machine is usually just the entry ticket: a 671B-parameter model with FP8 weights needs about 671 GB for the weights alone—before KV Cache and framework overhead—which already will not fit into a single machine’s 8 × 80 GB = 640 GB of VRAM. This is all the more true for 4-bit schemes like GPTQ that compress only the weights and not the KV Cache, which under long-context, high-concurrency conditions are not necessarily more economical than FP8. Deployments at this scale therefore usually require multi-machine, multi-card setups, and electricity and network costs climb in steps as a result.

    6. Five Hidden Costs That Are Easy to Miss

    1. Datacenter and supporting infrastructure. High-density racks impose thresholds on power capacity, floor loading, and cooling capacity, and retrofitting an old facility can cost more than the equipment itself. When air cooling can no longer keep up, you have to move to liquid cooling—and both the piping and the operations regime have to be rebuilt.
    2. The three-tier storage structure. The model repository and data lake typically need object storage; vector retrieval needs a vector database; and logging, alerting, and tracing are the “dashboard” for the inference service. Without observability, no throughput drop in production can be diagnosed, and that hidden loss far exceeds the price of the tools themselves.
    3. Inference-optimization talent. Engineers who can do operator optimization, batch scheduling, quantization tuning, and parallelism-strategy selection are generally harder to hire and more expensive than ordinary backend engineers. For many teams, the real bottleneck is not too few cards but no one who can push those cards to their limit.
    4. Model iteration and asset impairment. The generation cycle of open-source models has already shrunk to a few months. The cards you buy today may no longer be the best cost-performance choice in two years, and a 3–5-year depreciation schedule means you have to absorb asset devaluation along the way. In AI infrastructure, the idea that hardware “holds its value” basically does not hold.
    5. Idle rate. This is the most concealed and the most fatal of all. Buying to peak capacity while average utilization sits at 20% is equivalent to multiplying your effective unit cost by five. Any budget that does not account for utilization is an underestimated budget.

    7. Self-Build vs. Cloud: Decide by Utilization, Not Sentiment

    The real advantages of self-building are data sovereignty, a lower unit cost under stable long-term load, and deep customization with controllable latency. The costs are upfront capital expenditure, rigid iteration, and the operations burden. The advantages of cloud elasticity are zero operations overhead, pay-as-you-go billing, and the ability to adopt new hardware at any time. The costs are a higher marginal cost under sustained high load and additional network and data-compliance requirements.

    A practical decision framework:

    Dimension Lean self-build Lean cloud elasticity
    Average utilization Long-term above 60% Long-term below 30%
    Load shape Steady and predictable Sharp peaks, very deep troughs
    Data compliance Data strictly cannot leave its domain Cloud is acceptable under compliance constraints
    Team capability Has an inference-platform engineering team No dedicated operations staff
    Iteration pace Needs to stay locked to one model generation long-term Needs frequent switching and head-to-head evaluation

    For most mid-sized and large organizations, the realistic answer is a hybrid: put the core, stable, and sensitive workloads on a self-built cluster and run them at full utilization, while placing peak overflow, disaster recovery, and early evaluation of new models on the cloud. Cloud providers such as Baidu AI Cloud, with their elastic compute and containerized platforms, can take on exactly this “overflow and validation” role—letting the self-built portion be sized against average load rather than peak load. That is one of the most effective ways to bring TCO down.

    8. Tiered Budgeting Recommendations for Approval

    If your goal is to put a budget in front of decision-makers, don’t give a single number—give three tiers and a validation path:

    • Exploration / validation tier (tens of thousands to hundreds of thousands of yuan): a single machine with a few cards, consumer- or professional-grade accelerators, a mid-sized open-source model, and 4-bit quantization, to validate feasibility and establish an effectiveness baseline. The goal is to answer “is this worth doing?”
    • Small-scale production tier (millions of yuan per year): a single machine with multiple cards, redundancy, a basic observability stack, and dedicated operations, to support an internal pilot or limited external traffic. The goal is to answer “can it serve reliably?”
    • Scale tier (tens of millions of yuan per year): a multi-machine cluster, high-speed networking, liquid cooling or a high-density facility, a platform-engineering team, and full monitoring, alerting, and disaster recovery. The goal is to answer “can the unit cost beat calling an API?”

    Three procurement-discipline recommendations go with these: first, run a 2–4 week POC to measure real throughput, first-token latency, and the concurrency ceiling, then back out the card count from those numbers; second, put electricity and labor in the same budget table rather than two separate ones; third, reserve a 20–30% buffer budget for the re-tuning that model iteration will require.

    9. Conclusion

    The cost of self-hosting a large language model is fundamentally a capacity-planning problem under VRAM and bandwidth constraints, not a GPU quote sheet. Work it through in this order: first calculate the weight and KV Cache footprints, then set concurrency and card count against your latency target, then fold power, datacenter, network, and labor into a three-year TCO, and finally use average utilization rather than peak to decide how much to self-build. Follow that sequence, and the resulting number will usually differ from your initial back-of-the-envelope guess by a factor of two to three—and that two-to-threefold gap is precisely why most projects end up seeking additional funding during implementation. Calculating it up front is the single biggest saving you can capture in this budget.