Self-Hosting Open-Source LLMs: How to Calculate the Real Cost

    Self-hosting an open-source large language model is not a one-time GPU purchase. This article breaks down the total cost of ownership (TCO) into five layers — hardware, capacity headroom, power and cooling, operations, and engineering time — and shows how to estimate VRAM requirements first, since memory capacity decides which model you can run and how many accelerators you need. It covers a rough per-parameter formula for weight memory, the KV cache that grows with context length and batch size, and a Python script for a quick order-of-magnitude estimate; the trade-offs between FP16/BF16, INT8, and INT4 quantization; the latency and throughput constraints that shape cost per request; and when self-hosting beats cloud API calls and vice versa. It closes with practical ways to surface the costs teams usually overlook, so the decision rests on numbers rather than intuition.

    1. Why self-hosting deserves a proper cost model

    Over the past year, the capability ceiling of open-source large language models has kept rising, and more teams are seriously evaluating running models on their own infrastructure. The motivations are usually a few familiar ones: compliance and security requirements that keep data inside the internal network, predictable control over inference latency, and lower unit cost per call over the long run.

    A common mistake in evaluating these deployments is to ask only “can it run?” Loading a model successfully is the starting line, not the finish line. What actually determines whether a deployment holds up is the total cost of ownership (TCO). You can look at inference speed, you can look at memory footprint, and you can compare the per-thousand-token price against a hosted service — but once you put everything back on the cost axis, it becomes clear that self-hosting an open-source model is not as simple as “buying a card.” It is a systemic investment spanning hardware, power, operations, and people.

    The goal of this article is to break that cost down into pieces and give you an estimation methodology you can actually apply and re-run.

    2. The core cost components: don’t just look at the GPU

    When most people start estimating cost, their first thought is “how much does one accelerator cost?” That is only the tip of the iceberg. A complete cost model has at least five layers:

    • Hardware acquisition: accelerators (GPU/NPU), memory capacity, CPU, system memory, and system/data disks.
    • Capacity headroom: supporting high concurrency or fine-tuning usually requires horizontal scaling across multiple cards and multiple machines.
    • Power and cooling: the ongoing electricity draw of high-wattage equipment, plus data center cooling and power-delivery upgrades.
    • Operations and stability: monitoring and alerting, failure recovery, model updates, and driver/framework maintenance.
    • People: engineers who can debug distributed inference and tune performance — their time is routinely and severely underestimated.

    Power and people are the two items most often left out. A high-wattage inference server running at full load can cost a meaningful share of its own purchase price in electricity over a year. And a single engineer fluent in inference optimization can, over the long run, cost more than the hardware itself.

    3. Estimating VRAM requirements: the first question in cost modeling

    Memory capacity determines both “how large a model can you run” and “how many cards do you need,” which makes it the starting point of any cost estimate. VRAM usage comes from three parts: model weights, the KV cache (the key/value cache used during inference), and activations plus framework overhead.

    3.1 Model weights

    Weight memory scales directly with parameter count and quantization precision. The rule of thumb:

    1. Weight memory ≈ parameters × bytes per parameter
    2. FP16/BF16 : 2 bytes per parameter
    3. INT8 : 1 byte per parameter
    4. INT4 : ~0.5 bytes per parameter

    Take a 7B-parameter model as an example: at FP16 precision, the weights occupy roughly 14GB, while INT4 quantization compresses that to about 3.5GB. This is exactly why quantization has become the key lever for cutting cost and improving efficiency in self-hosted deployments — it directly determines how large a model you can run on how small and cheap a card.

    3.2 KV cache

    During inference, the model caches the key and value of each token so that prior context can be reused. This memory grows linearly with context length and batch size. With long documents or high concurrency, it can even exceed the weights themselves.

    3.3 A quick estimation script

    The script below is an illustrative Python snippet for a fast back-of-the-envelope check, so you have an order-of-magnitude sense of memory requirements before you commit to hardware. (The results are engineering estimates; treat actual framework measurements as authoritative.)

    1. def estimate_vram(params_b, precision_bits, context_len,
    2. layers, hidden_dim, batch=1):
    3. """Estimate inference VRAM requirements. Returns values in GB."""
    4. gi = 1024 ** 3
    5. bytes_per_param = precision_bits / 8
    6. # 1) Model weights
    7. weights_gb = params_b * 1e9 * bytes_per_param / gi
    8. # 2) KV cache (simplified estimate for typical multi-head attention)
    9. # Each token and each layer needs both K and V cached
    10. kv_per_token = 2 * layers * hidden_dim * bytes_per_param
    11. kv_cache_gb = kv_per_token * context_len * batch / gi
    12. # 3) Activations and framework runtime overhead
    13. # Empirically 15%-25% of the two figures above
    14. overhead = (weights_gb + kv_cache_gb) * 0.20
    15. total = weights_gb + kv_cache_gb + overhead
    16. return round(weights_gb, 2), round(kv_cache_gb, 2), round(total, 2)
    17. # Example: 7B model, FP16, 4K context, 28 layers, hidden dim 4096
    18. w, kv, total = estimate_vram(
    19. params_b=7, precision_bits=16,
    20. context_len=4096, layers=28, hidden_dim=4096, batch=1
    21. )
    22. print(f"Weights≈{w}GB KV≈{kv}GB Total≈{total}GB")

    Once you have the total memory requirement, you can work backwards to the number and class of accelerators you need, and from there to a hardware budget. One practical principle: memory capacity first, peak compute second. If memory is insufficient, the model simply won’t load; if compute is insufficient, it’s just slower.

    4. Precision and quantization: trading cost against quality

    Quantization is the most effective lever for bringing cost down, but more aggressive is not automatically better:

    • FP16/BF16: the smallest accuracy loss and the largest memory footprint. Suitable for use cases sensitive to output quality.
    • INT8: halves memory usage with accuracy loss that is usually manageable — a common sweet spot for cost-effectiveness.
    • INT4: compresses memory to roughly a quarter, significantly lowering the hardware barrier, but may degrade quality on complex reasoning tasks and needs to be validated against your actual workload.

    When choosing a quantization scheme, compare quality on a small sample first, then settle on the precision. Avoid trading away the output consistency the business actually depends on just to save money.

    5. Performance constraints: throughput, latency, and concurrency

    Cost depends not only on “can it run” but also on “how fast, and how many users can it serve at once.” Two metrics matter most:

    • Time to first token (TTFT): determines the feel of an interactive experience, and is noticeably affected by compute in long-context scenarios.
    • Throughput (tokens/s): determines how many requests can be served per unit of time, and directly drives how much concurrency a single card can handle.

    Under high concurrency, memory usually becomes the bottleneck before compute does — because every concurrent request consumes additional KV cache. So if you’re aiming for high concurrency at low cost, batching strategy and memory utilization are where the optimization effort should go.

    6. Self-hosting vs. cloud API calls: which is cheaper?

    Self-hosting is not always the cheaper option. The answer depends on usage volume and stability:

    • Self-hosted: a good fit when call volume is sustained and stable, data sensitivity is high, or you need deep customization or fine-tuning. At large enough scale, the amortized unit cost advantage is clear.
    • Cloud API calls: a good fit when call volume fluctuates, during early validation, or when you have no operations team. It converts hardware, operations, and scaling costs into pay-as-you-go spending, with a low starting barrier and good elasticity.

    To decide, plug your projected monthly call volume into two formulas — “self-hosted amortized monthly cost” versus “cloud unit price × call volume” — and factor in data compliance requirements. For many teams, a hybrid approach of “validate in the cloud, then serve core steady-state traffic locally” balances flexibility against long-term cost.

    7. Making hidden costs visible

    Operations and power are the costs most easily overlooked. A few concrete recommendations:

    1. Power estimation: estimate annual electricity cost as full-load system power × runtime × electricity price, add cooling overhead, and fold the result into TCO.
    2. Resource monitoring: use monitoring, alerting, and log services to continuously observe memory usage, GPU utilization, and latency, so waste and bottlenecks surface early.
    3. Elasticity and storage: keep model weights, logs, and vector data in centralized object storage; run training or batch jobs through a container platform with elastic scheduling so compute is allocated on demand.
    4. Capturing human knowledge: document deployment scripts, tuning experience, and incident runbooks to reduce dependence on any single engineer — itself a form of hidden cost optimization.

    8. Conclusion: know the numbers before you choose

    The cost of self-hosting an open-source large model is fundamentally a combined question of “scale × precision × operations.” The core takeaways come down to three points:

    • Memory is the starting point: estimate your memory requirement first, then work backwards to hardware specs and quantity.
    • Quantization is the lever: find the precision balance point between quality and cost that your business can accept.
    • Scale is the dividing line: self-hosting pays off when calls are stable and voluminous; the cloud is more flexible during fluctuation or early validation.

    Once you put all four cost items — hardware, power, operations, and people — on the table and compare them against the cloud’s unit price, the decision stops being “deploy on a hunch” and becomes a budget and hardware choice you can defend. That accounting view gets closer to the reality of production than any single performance number.