Self-Hosting Large Open-Source LLMs: Hardware Requirements and Cost Estimation

    Debate over large open-source LLMs tends to split between complaints about latency and per-call cost and praise for their capability ceiling — but both views describe the same underlying situation: the models are capable enough for real workloads, and nearly all the remaining friction is engineering-side, namely deployment thresholds, hardware overhead, concurrency, and unit cost. This article defines what "self-hosting" actually involves across four layers (weights, inference runtime, serving gateway, and observability), explains why GPU memory is the only hard gate — including capacity and bandwidth math with a reusable estimation script — lays out four typical deployment tiers, and breaks cost into CAPEX and OPEX with a unit-cost formula. It also covers when self-hosting is the wrong choice, the metrics to validate on, and a pre-launch checklist. The goal is a practical answer to two questions: what hardware do you need, and how much will it cost?

    Large open-source models have been getting a lot of attention lately, and opinions are sharply divided: some complain about slow responses and high per-call cost, while others point to the capability ceiling. Put the two views side by side and they turn out to describe two sides of the same thing — the model’s capability density is already enough to support real workloads, and almost all of the remaining debate lands on the engineering side: deployment thresholds, hardware overhead, concurrency, and unit cost. So the technical question worth asking isn’t “which model is strongest,” but something more down-to-earth: if you want to run a large open-source model on your own machines, what hardware do you need, and how much will it cost?

    1. Defining “self-hosting” precisely

    A lot of cost arguments come from vague concepts. Self-hosting actually has four layers, and it won’t run if any one of them is missing:

    1. Model weights: set the floor for GPU memory and determine how much room you have for quantization.
    2. Inference runtime: handles paged KV Cache management, continuous batching, prefix-cache reuse, and other scheduling — this directly determines throughput.
    3. Serving gateway: authentication, rate limiting, queuing, context assembly, and per-business tagging/billing.
    4. Observability: GPU memory usage, time-to-first-token, queue time, and token consumption must be trackable — otherwise cost is always a muddle.

    “Self-hosted” also comes in two forms: single-node (one workstation or one multi-GPU server) and a private cluster (multiple nodes, multiple replicas, failover). The two differ by an order of magnitude in cost, so be clear which tier you’re in before you start the discussion.

    2. GPU memory is the only hard gate: capacity, bandwidth, card count

    2.1 Capacity estimation

    GPU memory is consumed by two things: weights and KV Cache.

    1. Weight memory ≈ params P × bytes per param
    2. FP16/BF16 → 2 bytes; INT8 → 1 byte; INT4 → 0.5 byte
    3. (quantization metadata usually makes this slightly higher)
    4. KV Cache ≈ 2 × layers L × KV heads × head_dim × sequence length S × concurrency B × bytes per value
    5. (with grouped-query attention, KV heads are far fewer than query heads,
    6. so this term drops substantially)

    A reusable estimation script:

    1. def estimate(weights_b, bytes_per_param, layers, kv_heads, head_dim,
    2. seq_len, concurrency, kv_bytes=2, overhead=1.2):
    3. weights = weights_b * 1e9 * bytes_per_param
    4. kv = 2 * layers * kv_heads * head_dim * seq_len * concurrency * kv_bytes
    5. return (weights + kv) * overhead / 1024**3 # result in GiB
    6. # Example: 70B-param model, INT4 weights, 32 layers, 8 KV heads, head dim 128,
    7. # 32K context, 8-way concurrency
    8. print(estimate(70, 0.5, 32, 8, 128, 32768, 8))

    The key takeaway: context length and concurrency are the only variables in the memory requirement that you actually control. For the same model, pushing context from 4K to 128K and concurrency from 1 to 32 can multiply the memory demand several times over. Most “it won’t run” cases aren’t about the weights failing to fit — it’s the KV Cache eating up all the remaining headroom.

    2.2 Bandwidth determines perceived speed

    The decode phase processes one token per step, which is a classic memory-access-bound workload. Its speed ceiling is roughly set by GPU memory bandwidth, not by compute peak. This is why “the weights fit” and “it runs smoothly” are two completely different problems.

    2.3 For MoE models, distinguish two numbers

    Large sparse (MoE) models have a huge total parameter count but activate fewer parameters per token. What they reduce is compute pressure — but all expert weights still have to stay resident in GPU memory, so the capacity threshold does not go down. Estimating memory from “activated parameters” is a common pitfall.

    3. Four typical tiers and their cost ranges

    Tier Hardware Models it can carry (illustrative) Use case
    A — Validation A single 24–48GB accelerator card 7B–14B params, INT4 Personal experiments, effect validation
    B — Shared team A single 80GB-class card or dual cards 30B INT4 / 14B FP16 R&D productivity for teams under 10
    C — Production workhorse Single node, 4–8 80GB-class cards ~70B dense, or hundreds-of-billions-param MoE (INT4/FP8) Tens to hundreds of concurrent requests
    D — High-availability cluster Multiple nodes + multiple replicas + prefill/decode disaggregation Scaled replicas of Tier C models A company-wide unified inference platform

    Tier A overall investment is typically in the low thousands to about ten thousand dollars; Tier B is on the order of tens of thousands; Tier C runs from tens of thousands up into the low hundreds of thousands; and Tier D is an ongoing-investment mode. These are orders of magnitude, not quotes, because accelerator-card prices swing heavily with market supply, demand, and sales channels. Any estimate must be re-run with the latest actual quote at the time you buy.

    4. Splitting cost into CAPEX and OPEX

    4.1 One-time investment (CAPEX)

    • Compute cards: the biggest chunk, usually 60%–75%.
    • Host and CPU/memory: provision memory to 1–2× the GPU memory so that weight loading and paging don’t become the bottleneck.
    • Local storage: model weights run to tens or hundreds of GB, so you need fast SSDs to keep load times reasonable.
    • Inter-card and inter-node interconnect: single-node multi-GPU setups depend on high-bandwidth interconnect; cross-node scaling needs high-speed networking.
    • Supporting modifications: power, cooling, racks, and UPS. In an older office environment, this is very easy to leave out of the budget.

    4.2 Ongoing investment (OPEX)

    • Electricity: annual electricity = total system power (kW) × 24 × 365 × electricity rate. A fully loaded 6 kW multi-GPU server draws about 52,560 kWh per year; plug in your local rate — at a modest assumed ~$0.10/kWh that comes to roughly $5,000 a year, and that still excludes the added cost of cooling.
    • Personnel: inference-service tuning, capacity planning, model-upgrade regression, and on-call coverage. This is the most underestimated item, usually equivalent to 30%–50% of a full-time engineer.
    • Depreciation: straight-line over 3 years, so annual depreciation = total hardware cost ÷ 3.

    4.3 Unit-cost formula

    1. Annual TCO = annual depreciation + annual electricity + personnel (amortized) + ops / rack space
    2. Cost per 1M tokens = Annual TCO ÷ effective annual token output

    The denominator is decisive. GPU idle time is the biggest hidden waste in self-hosting: if a million-dollar-class cluster averages only 15% utilization, its unit cost comes out an order of magnitude higher than pay-as-you-go. So capacity planning must be load-tested against “target concurrency × 1.5 headroom,” not procured on a guess at the peak.

    Conversely, low-traffic scenarios aren’t a good fit for self-hosting: a few hundred to a few thousand calls a day can’t amortize the fixed costs, and in that case pay-as-you-go managed inference from a cloud provider is usually more economical. On the model side, you can also evaluate general-purpose compute and managed-inference offerings from providers such as Baidu AI Cloud.

    5. When self-hosting is the wrong choice

    • Traffic has clear peaks and troughs, and the peaks are unpredictable.
    • The model iterates frequently and hardware depreciation cycles can’t keep up with model evolution.
    • The team lacks inference-ops experience and can’t handle P99 latency or failure fallback.
    • Each task demands the capability ceiling and you need to switch between different model sizes often.

    Conversely, workloads where data can’t leave the intranet, that require physical network isolation, and that have long-term stable, predictable concurrency are where self-hosting truly pays off. A hybrid architecture is often the more pragmatic answer: route sensitive data to a small local model and offload complex tasks to managed services.

    6. Validate on four metrics, not just benchmarks

    1. Time to first token (TTFT): drives the interactive feel, and is most affected by context length and queue depth.
    2. Per-token generation speed: a memory-access-bound metric, strongly correlated with GPU memory bandwidth.
    3. End-to-end throughput: total token output under concurrency, reflecting batch-scheduling quality.
    4. P99 latency and effective concurrency: the real dividing line for user experience.

    On the quality side you must also build your own evaluation: prepare a 200–500-case set that closely matches your business, compare results before and after quantization, and specifically test mid-context information recall under long contexts. Quantization’s precision loss varies by task — code, math, and long-document reasoning are more sensitive to low-bit quantization, so validate with your own eval set and don’t copy others’ conclusions.

    7. Pre-launch checklist

    • Reserve at least 20% GPU memory headroom to avoid paging at peaks.
    • Set context length to real business needs; don’t pay for extreme lengths.
    • Enable prefix caching — system prompts and fixed templates can greatly cut prefill overhead.
    • Evaluate acceleration options such as speculative decoding and small draft models.
    • Make sure the gateway supports rate limiting, queuing, and fallback to smaller models.
    • Make sure observability covers memory, throughput, and queue length, with per-business token-cost tagging.
    • Load-test to 1.5× the target concurrency and record the inflection point.

    Back to the opening judgment: the capability of these large open-source models holds up, and “slow” and “expensive” are, more often than not, not model defects but the combined result of GPU-memory configuration, context strategy, batch scheduling, and utilization. So the answer to self-hosting is no longer “can it run,” but three more concrete questions — can you afford to run it, can you run it stably, and can you account for it clearly.