LLM Local Deployment: GPU Selection and Cost Estimation
A practical guide to deploying large language models on-premise. It explains why self-hosting matters, why GPU memory (VRAM) rather than raw compute is the first bottleneck in inference, how to estimate VRAM requirements, how to choose GPUs based on memory capacity and interconnect bandwidth, how to compare the CAPEX of building your own hardware against the OPEX of cloud rental, how to factor in utilization, and which hidden costs to budget for. Includes a reusable VRAM estimation script and deployment/tuning recommendations.
1. Start Here: Why Deploy Locally (On-Premise)?
Before discussing how much it will cost, you need to confirm the motivation for deploying, because that motivation directly determines the scale of compute and the cost tier. Common drivers include:
- Data compliance and security: sensitive data cannot leave its boundary, so inference must run within your own or a dedicated environment.
- Network and stability: production pipelines cannot depend on public internet calls; you need controllable latency and availability.
- Deep customization: you may need to fine-tune, quantize, or otherwise adapt the model privately, which hosted services often cannot satisfy.
- Long-term cost considerations: when call volume is extremely high and sustained, the marginal cost of self-hosted compute per unit can be lower than pay-per-call API pricing.
Once the motivation is clear, you can judge whether you need a “just make it run” validation environment or a “high-concurrency, low-latency” production cluster—and the investment between the two often differs by an order of magnitude.
2. The Core Bottleneck: VRAM in Inference, Not Raw Compute
Many beginners focus only on a GPU’s floating-point throughput (TFLOPS) when selecting hardware, but overlook a more critical fact: the first gate for LLM inference is usually VRAM capacity. The model weights must be fully loaded into VRAM; once they no longer fit, the model cannot start, regardless of compute power.
Estimating Weight VRAM
Parameter count and VRAM usage are roughly linear, and the core formula is very simple:
Weight VRAM (GB) ≈ Parameters (B) × Bytes per parameter
Where “bytes per parameter” depends on numerical precision:
- FP16 / BF16: about 2 bytes per parameter
- INT8: about 1 byte per parameter
- INT4: about 0.5 bytes per parameter
For example, a 70-billion-parameter (70B) model under FP16 needs 70 × 2 = 140 GB of VRAM for weights alone—already exceeding the VRAM of the vast majority of single cards, which means you must rely on multi-GPU parallel splitting or quantization to compress it.
The Full VRAM Breakdown
The VRAM actually required is far more than just the weights. You should also account for:
- Weight VRAM: as described above.
- KV Cache: positively correlated with sequence length, number of layers, hidden dimension, and concurrent batch size; its share is significant in long-context scenarios.
- Activations and runtime overhead: intermediate tensors from the forward pass, VRAM fragmentation, and the inference framework’s own overhead—generally reserve another 10%–20%.
Here is a rough estimation snippet you can reuse directly:
def estimate_vram(num_params_b, precision_bytes, seq_len,num_layers, hidden_size, batch_size=1, overhead=0.15):"""Estimate the VRAM required during inference (in GB).num_params_b : number of parameters (in B, i.e. billions)precision_bytes : bytes per parameter (FP16=2, INT8=1, INT4=0.5)seq_len : context / sequence lengthnum_layers : number of model layershidden_size : hidden layer dimensionbatch_size : concurrent batch sizeoverhead : reserve ratio for VRAM fragmentation and runtime overhead"""weights_gb = num_params_b * precision_byteskv_cache_gb = (2 * num_layers * hidden_size * seq_len * batch_size* precision_bytes) / (1024 ** 3)total_gb = (weights_gb + kv_cache_gb) * (1 + overhead)return round(total_gb, 2)# Example: 70B model, FP16, single 4096-token context# print(estimate_vram(70, 2, 4096, 80, 8192))
3. GPU Selection: Prioritize VRAM Capacity and Interconnect Bandwidth
Combined with the estimation above, you can follow these principles when selecting hardware:
- Prioritize data-center-class GPUs: these accelerators offer large VRAM (typically tens of GB), high memory bandwidth, and high-speed inter-card interconnect, making them a sensible choice for large models. Consumer-grade GPUs may have decent compute, but their VRAM capacity and interconnect capabilities are often limiting factors.
- Decide the card count by total VRAM: divide the estimated total VRAM by the usable VRAM per card to get the minimum number of cards. For example, if total demand is about 300 GB and usable VRAM per card is about 64 GB, you would theoretically need more than 5 cards; considering splitting overhead and redundancy, practice often goes straight to a full 8-GPU node.
- Multi-GPU parallelism needs high-speed interconnect: tensor parallelism splits a single layer’s computation across multiple cards and is extremely sensitive to inter-card bandwidth, so a single-node multi-GPU setup (intra-node interconnect) is generally better than multi-node multi-GPU.
- Use quantization to reach a feasible configuration: if VRAM is tight, evaluate INT8/INT4 quantization schemes, trading acceptable accuracy loss for significantly lower VRAM usage and thereby shrinking the overall machine size.
4. Cost Estimation: Two Paths—Self-Build (CAPEX) vs. Rental (OPEX)
1. Self-Built Hardware: Primarily Upfront Investment
Under the self-built model, cost is dominated by capital expenditure (CAPEX): you buy servers and accelerators upfront and then amortize them via depreciation. Its advantages are low per-unit cost under long-term full load and full control over data; its disadvantages are high upfront investment, slow scaling, and the need to cover operations and data-center costs yourself.
2. Cloud Rental: On-Demand or Reserved Postpaid
For data-center-class accelerators, the common practice is to check a cloud provider’s rental pricing. The industry generally uses a per-GPU-hour billing model, split into:
- On-demand billing: spin up and release at any time; flexible but the highest unit price.
- Reserved / committed-term billing: commit to a longer term (e.g., annual) in exchange for a lower unit price, suitable for long-term, stable workloads.
Cost Estimation Example (illustrative, not a specific quote): assume a data-center-class GPU on some cloud platform has a reserved price of a certain number of dollars per hour per card. Then the hourly cost of an 8-card node is “unit price × 8”. If that node runs at full load 12 hours per day, the daily cost = unit price × 8 × 12, and the monthly cost is roughly that multiplied by about 30 days. This linear, computable structure makes it easy for a team to work backward from a budget to the affordable runtime and card count.
Note: the numbers above are examples used to illustrate the calculation logic. Real prices vary by vendor, region, model, and contract term—please refer to the current official pricing of your chosen platform.
3. The Key Variable: Utilization
Whether self-built or rented, the real crux of cost estimation is utilization. If an 8-card node spends only 20% of its time on inference and sits idle the rest of the time, the cost per unit of effective compute is multiplied several times over. Therefore, you should introduce an “effective utilization” factor into your estimation:
Effective actual cost ≈ Nominal cost / Target utilization
5. Don’t Ignore Hidden Costs
Counting only the accelerators themselves can severely underestimate total investment. The following items should be included in the budget as well:
- Power and cooling: high-power accelerators demand a lot of the power supply and data-center cooling; long-term electricity and cooling costs are not trivial.
- Network and storage: loading model weights, multi-GPU communication, and persisting the KV Cache all depend on high-bandwidth networks and fast storage.
- Operations and personnel: environment setup, monitoring and alerting, fault handling, and version upgrades all require engineering effort.
- Depreciation and residual value: self-built hardware carries the risk of rapid devaluation due to technology iteration.
6. Deployment and Tuning Recommendations
- Choose your inference runtime: prefer inference frameworks that are mainstream in the community and friendly to quantization and parallelism, to avoid reinventing the wheel.
- Batching and concurrency: use techniques such as continuous batching to improve throughput, directly improving utilization and per-unit cost.
- Quantization and distillation: without hurting business outcomes, quantization, pruning, and distillation can all effectively lower the VRAM and compute thresholds.
- Observability: wire up monitoring and logging, and establish baselines for VRAM usage, throughput, and latency—only then can you continuously surface cost and performance bottlenecks.
7. Summary
The cost of local LLM deployment is essentially a chain of multiplication: “VRAM demand → card count → compute tier → pricing model × utilization”. First use VRAM estimation to lock down the hardware scale, then weigh self-build against rental according to your workload characteristics, and finally don’t forget hidden expenses like power, operations, and depreciation. Once you’ve worked this chain out clearly, you can produce a credible budget at the project-approval stage and avoid discovering compute or cost overruns only after going live.
