Self-Hosting Open-Source LLMs: A Complete Cost Assessment Guide, from Hardware Costs to Budget Proposals

1. The Problem: When “Open Source” Meets Capital-Intensive Infrastructure

Open-source model weights can be downloaded for free, but a wide infrastructure gap separates “getting the model to run” from “running it well.” For a trillion-parameter sparse architecture, even when the weight files are publicly available, local deployment involves VRAM capacity, interconnect bandwidth, power and cooling, and data center retrofit costs that far exceed most teams’ first instinct.

The figures that circulate in the industry — an entry point of roughly $1 million, with recommended configurations running to several million dollars or more — are not alarmism. They follow from the physical characteristics of the model architecture. This article tries to break the bill down: where the money goes, which parts can be trimmed, which parts cannot be skipped, and how to present a reproducible methodology when submitting a budget request to management.

One caveat: this article covers general methodology only and does not target any specific vendor’s products or pricing. All figures are illustrative estimates based on public architecture parameters; actual procurement should follow official vendor quotes.

2. The Four Layers of Total Cost

The total cost of ownership (TCO) of a self-hosted large model breaks down into four layers. They constrain one another and cannot be optimized in isolation.

2.1 The Compute Layer: Accelerators and VRAM

This is the bulk of the cost. Even with low-precision quantization, a trillion-parameter sparse model needs hundreds of gigabytes to several terabytes of VRAM just to load the weights. With 8-bit quantization, you need dozens of accelerators with 80GB of VRAM each to fit the weights plus the KV cache; higher precision or larger concurrent context windows push the card count up further.

The estimation formula can be simplified to:

Required accelerators ≈ (weight VRAM + peak KV cache + activation overhead) / usable VRAM per card × redundancy factor

The redundancy factor is typically 1.2–1.5, covering memory fragmentation and load spikes. At per-card prices of several thousand dollars and up, dozens of accelerators alone push the bill toward the million-dollar mark — consistent, in order of magnitude, with the widely cited “roughly $1 million minimum” for deployments of this class.

2.2 The Interconnect and Server Layer: Underestimated Hidden Costs

Multi-card inference requires high-speed interconnect between cards. If interconnect bandwidth is insufficient, tensor-parallel communication becomes the bottleneck and accelerator utilization stays below 50% for long stretches — effectively paying twice for the same throughput. The cost of high-speed interconnect switches, purpose-built server backplanes, and high-bandwidth NICs therefore must be budgeted; it typically runs 15–25% of total hardware spend.

In addition, servers built around high-power accelerators can draw tens of kilowatts each, and the cost of power distribution, cooling, and rack retrofits is not negligible. Air cooling runs into thermal limits at high per-rack power density, at which point liquid cooling enters the conversation — and retrofit costs are especially steep in older facilities.

2.3 The Software and Adaptation Layer: Engineering Time Is a Cost Too

Inference framework selection, operator adaptation, parallelism strategy tuning, and quantization validation all require engineers with distributed-systems experience. A mid-sized team typically needs anywhere from a few weeks to a few months to go from environment setup to a stable service, and this labor should be counted in TCO as person-months.

This line item is often missing from budget spreadsheets, yet it is exactly what determines whether the hardware you bought can actually be put to work. A good practice is to earmark 10–15% of the budget for software adaptation and tuning.

2.4 The Operations and Power Layer: Recurring Spend

Over a three-year horizon, electricity, colocation, spare parts, and operations staffing can add up to 30–50% of the hardware purchase cost. TCO must be evaluated over a three-year period, not just the one-time purchase price.

3. Tiered Configurations: Three Budgets for Three Goals

Teams’ goals vary widely, and budgets should not be one-size-fits-all. Work backward from the goal to the configuration.

3.1 Entry Tier: Validation and Prototyping (roughly $1M)

The goal is to “get it running, demo it, and run small-scale evaluations.” Configuration notes:

• Pick a mid-size model or a high-sparsity variant and apply 4-bit quantization;
• Use a single multi-GPU server to avoid cross-node interconnect complexity;
• Prioritize VRAM capacity over interconnect bandwidth;
• Accept low concurrency (single digits) and longer time to first token.

This tier suits technical validation and feasibility studies. It is not built to carry production traffic.

3.2 Standard Tier: Small-Scale Production (several million dollars)

The goal is to “support internal business trials and limited concurrency.” Configuration notes:

• Multi-server, multi-card, combining tensor parallelism with pipeline parallelism;
• Deploy request queuing and dynamic batching to raise accelerator utilization;
• Introduce KV cache reuse and prefix caching to eliminate repeated computation;
• Establish basic monitoring, alerting, and capacity observability.

This tier demands genuine distributed-inference tuning skills, and the software share of the budget rises noticeably.

3.3 Recommended Tier: Production at Scale (upwards of $5M)

The goal is to “serve multiple business lines with a stable API.” Configuration notes:

• Multiple replicas behind load balancing to guarantee availability;
• Disaggregate prefill and decode and pool resources for each phase;
• Mature canary releases, rate limiting and graceful degradation, and cost-allocation metering;
• Reserve headroom for expansion and run quarterly capacity planning.

At this tier the core challenge shifts from “can it run” to “can we drive down the cost per token,” and the optimization focus moves to scheduling and batching efficiency.

4. Quantitative Evaluation: Let the Data Speak

When requesting budget from management, the most persuasive approach is not an itemized shopping list but a verifiable input–output estimate.

4.1 Key Metric Definitions

• Throughput (tokens/s): total output per unit of time; determines service capacity.
• Unit cost (USD per million tokens): TCO divided by total tokens over the period; the core metric for apples-to-apples comparison.
• Time to first token (TTFT) and time per output token (TPOT): determine user experience.
• Accelerator utilization: reflects investment efficiency; sustained utilization below 40% signals an imbalanced configuration.

4.2 The Estimation Workflow

# Illustrative estimation framework (pseudocode)
def estimate_tco(cards, card_price, host_cost, power_3y, labor_3y):
    hardware = cards * card_price + host_cost
    return hardware + power_3y + labor_3y
def unit_cost(tco, throughput_tokens_per_s, utilization, hours=24*365*3):
    total_tokens = throughput_tokens_per_s * utilization * 3600 * hours
    return tco / (total_tokens / 1_000_000)

Run small-scale load tests to measure single-replica throughput, multiply by the replica count and target utilization to project total tokens over the period, then divide TCO by that number. This figure is more persuasive than any qualitative description.

4.3 The Build-vs-Buy Comparison

A budget proposal should include a comparison: if you do not self-host, what would three years of pay-as-you-go hosted inference cost for the same token volume? Self-building tends to win when utilization is high and load is stable; when load is spiky or initial volume is small, hosted services put far less pressure on cash flow. Plot both curves on the same chart and the decision basis becomes obvious.

5. Practical Advice for Technical Leads

1. Load-test before you buy. Benchmark throughput on a small number of cards and extrapolate to the full configuration; do not budget off theoretical peak numbers.
2. Do not cut the software budget. Put tuning labor in the spreadsheet; otherwise utilization never climbs and the hardware is effectively wasted.
3. Decide on three-year TCO. The one-time purchase price is misleading; power and operations are the real long-term burden.
4. Leave room to evolve. Models iterate fast; power and interconnect design should carry headroom.
5. Build metering in from day one. Record token-level usage from the start to lay the data foundation for cost allocation and optimization later.

6. Conclusion

For open-source large models, “free” applies only to the weights. Self-hosting is, at its core, a capital-intensive infrastructure investment. Break the cost into four layers — compute, interconnect, software, and operations — and use throughput and cost per token as the common yardstick, and the vague question of “a million dollars or ten?” becomes an engineering decision that can be debated and verified. A budget proposal’s persuasiveness ultimately comes from data, not adjectives.