How to Choose a GPU for Local LLM Deployment: A VRAM and Bandwidth Selection Guide

    This guide walks through how to choose a GPU for running large language models on-premises. It lays out the right order of decisions—VRAM capacity first, then memory bandwidth, then compute and software ecosystem—and breaks down the three main consumers of VRAM: model weights, KV cache, and runtime overhead. It includes a corrected Python script for estimating both VRAM usage and the single-stream decode speed ceiling, then explains how bandwidth, compute, MoE architectures, and quantization affect real-world performance. It also covers how to read the key GPU specs, which VRAM tiers suit which model sizes, where the cost-performance sweet spot sits, the risks of modified or memory-upgraded cards, local versus cloud tradeoffs, and common pitfalls followed by a practical three-step selection process.

    1. How Do You Choose a GPU for Local LLM Deployment?

    Common reasons for deploying large language models on-premises include keeping data inside your network, gaining more control over inference latency, and avoiding the long-term cost of renting cloud compute. In these setups, the GPU is usually the most important hardware investment. It determines whether the model fits, how long a context you can support, how much concurrency you can handle, and—to a large degree—how fast generation runs.

    People make two mistakes when picking a card: looking only at VRAM and assuming “bigger VRAM is always better,” or looking only at compute and assuming “higher floating-point throughput means faster.” A more sensible order of evaluation is:

    • VRAM capacity: determines whether the model can run at all;
    • VRAM bandwidth: sets the ceiling on single-stream generation speed;
    • Compute and precision support: affects long inputs, batching, and high-concurrency throughput;
    • Software ecosystem: determines whether quantization, attention implementations, and framework support work smoothly;
    • Cost and power: determine whether long-term deployment is economical and maintainable.

    To summarize in one line: when choosing a GPU for local LLM deployment, first check that VRAM is sufficient, then check that bandwidth is fast enough, then check that compute and ecosystem match your workload.

    2. VRAM Is the First Hurdle: Model Weights, KV Cache, and Runtime Overhead

    1. The three main consumers of VRAM

    During inference, VRAM is used mainly by three components:

    1. Model weights: number of parameters × bytes per parameter.

      • FP16/BF16: 2 bytes, roughly 14GB for a 7B model;
      • INT8/FP8: 1 byte, roughly 7GB for a 7B model;
      • INT4: about 0.5 bytes, roughly 3.5GB for a 7B model.

      A note of caution: INT4 usually refers to an integer format with no floating-point mantissa, while FP4 still keeps exponent bits, so the two should not be treated as equivalent. After real quantization there are also scaling factors, alignment, and grouping overheads, so actual usage is slightly higher than the theoretical “parameters × bytes” figure.

    2. KV cache: the cached historical Key/Value pairs kept during autoregressive generation. It grows with context length, concurrency, batch size, and the actual sequence lengths. In long-context scenarios, the KV cache can approach or even exceed the weights themselves.

    3. Runtime overhead: framework memory pools, communication buffers, temporary activations, and so on. As a rule of thumb, reserving 10%–20% is a safe choice.

    2. A VRAM estimation script you can use directly

    Below is a corrected version of the estimation script. It separates two key parameters:

    • max_seq: reserves VRAM for the maximum context;
    • avg_seq: the average sequence length used when estimating real decode speed;
    • batch: the number of concurrent requests;
    • kv_dtype: the storage precision of the KV cache;
    • efficiency: a decode-efficiency coefficient, typically estimated at 0.5–0.8.
    1. BYTES_PER_PARAM = {"fp16": 2, "bf16": 2, "fp8": 1, "int8": 1, "int4": 0.5}
    2. def _bpp(dtype):
    3. key = str(dtype).lower().replace("torch.", "").replace("_", "")
    4. if key not in BYTES_PER_PARAM:
    5. raise ValueError(f"Unsupported dtype: {dtype}")
    6. return BYTES_PER_PARAM[key]
    7. def estimate_vram_gb(params_b, dtype="int4", layers=32, kv_heads=8,
    8. head_dim=128, max_seq=4096, batch=1,
    9. kv_dtype="fp16", reserve=1.15):
    10. """Estimate single-GPU inference VRAM: max_seq reserves the maximum context, batch is the number of concurrent requests."""
    11. if params_b <= 0:
    12. raise ValueError("params_b must be a positive number")
    13. if max_seq <= 0 or batch <= 0:
    14. raise ValueError("max_seq and batch must be positive numbers")
    15. bpp = _bpp(dtype)
    16. kv_bpp = _bpp(kv_dtype)
    17. weights = params_b * 1e9 * bpp / 1024**3
    18. kv = 2 * layers * kv_heads * head_dim * max_seq * batch * kv_bpp / 1024**3
    19. return weights, kv, (weights + kv) * reserve
    20. def decode_speed_ceiling(bw_gbs, params_b, dtype="int4",
    21. efficiency=0.7, **kv):
    22. """Estimate the single-stream decode speed ceiling using only the KV cache under avg_seq."""
    23. weights = params_b * 1e9 * _bpp(dtype) / 1024**3
    24. avg_seq = kv.get("avg_seq", 1024)
    25. kv_avg = (2 * kv.get("layers", 32) * kv.get("kv_heads", 8) *
    26. kv.get("head_dim", 128) * avg_seq * kv.get("batch", 1) *
    27. _bpp(kv.get("kv_dtype", "int8"))) / 1024**3
    28. return bw_gbs / (weights + kv_avg) * efficiency
    29. if __name__ == "__main__":
    30. for name, p in [("7B", 7), ("14B", 14), ("32B", 32), ("70B", 70)]:
    31. w, kv, total = estimate_vram_gb(
    32. p, dtype="int4", layers=80 if p == 70 else 32,
    33. max_seq=4096, kv_dtype="fp16")
    34. print(f"{name:>3} | weights {w:5.1f} GB | KV peak {kv:5.1f} GB | total {total:5.1f} GB")
    35. print("7B INT4 decode speed ceiling: %.0f tok/s" % decode_speed_ceiling(
    36. 760, 7, dtype="int4", efficiency=0.7,
    37. layers=32, kv_heads=8, head_dim=128,
    38. avg_seq=1024, batch=1, kv_dtype="int8"))

    Example output might look like this:

    1. 7B | weights 3.3 GB | KV peak 0.5 GB | total 4.3 GB
    2. 14B | weights 6.5 GB | KV peak 0.5 GB | total 8.1 GB
    3. 32B | weights 14.9 GB | KV peak 0.5 GB | total 17.7 GB
    4. 70B | weights 32.6 GB | KV peak 1.2 GB | total 38.9 GB
    5. 7B INT4 decode speed ceiling: 160 tok/s

    Notes:

    • The estimates are usually a bit higher than what the framework actually reports—this is intentional safety headroom;
    • The 70B KV cache depends on the number of layers, key/value heads, and per-head dimension, so it varies across model configurations;
    • Decode speed is only a theoretical ceiling; in practice it is also affected by kernel implementations, memory clock, framework scheduling, quantization format, and model structure;
    • If VRAM is insufficient, first check context length, concurrency, KV cache precision, and the quantization scheme, rather than relying directly on CPU offloading.

    3. KV cache: the hidden cost in the long-context era

    The per-token KV cache overhead can be broken down as:

    1. KV bytes per token = 2 × layers × kv_heads × head_dim × bytes per element

    The factor of 2 accounts for the two caches, Key and Value. For a model with 32 layers, 8 key/value heads, a per-head dimension of 128, and FP16 storage, each token needs:

    1. 2 × 32 × 8 × 128 × 2 = 128KB

    So a 4096-token context needs roughly 0.5GB of cache, while a 32K context needs roughly 4GB. With 8 concurrent streams, the cache overhead is amplified further.

    Common reduction techniques include:

    • GQA (grouped-query attention): reduces the number of key/value heads;
    • KV cache quantization: storing key/value pairs in INT8/FP8 can halve the overhead;
    • Paged cache management and prefix cache reuse: reduces fragmentation and redundant computation.

    When selecting a card, reserve cache space based on “maximum context × concurrency,” not just the size of the weights.

    3. What Determines Speed: VRAM Bandwidth, Compute, and Efficiency

    1. Bandwidth sets the single-stream speed ceiling

    LLM inference usually has two phases:

    • Prefill: processes input tokens in one pass; compute-intensive and mainly constrained by compute;
    • Decode: generates token by token, repeatedly reading all weights and the existing KV cache—a classic VRAM-bandwidth-bound task.

    A rough estimate is:

    1. Single-stream decode speed ≈ VRAM bandwidth ÷ (weight bytes + average-context KV bytes) × efficiency coefficient

    The efficiency coefficient is typically 0.5–0.8 and is affected by kernel implementations, quantization format, memory clock, framework scheduling, and model structure. The longer the context and the higher the concurrency, the larger the share of KV reads, and the more noticeably speed drops.

    VRAM bandwidth can be roughly understood as:

    1. VRAM bandwidth = bus width × effective data rate ÷ 8

    For example, a 320-bit, 19Gbps spec gives a bandwidth of roughly:

    1. 320 × 19 ÷ 8 ≈ 760 GB/s

    So you can’t look only at VRAM capacity—bus width and memory clock matter just as much. Two other common pitfalls:

    • Unified memory architecture: the CPU and GPU share physical memory, so the capacity looks large, but bandwidth is usually clearly lower than a discrete GPU, and generation speed may suffer;
    • VRAM overflow: once the model exceeds VRAM, some weights may be offloaded to host memory and transferred over PCIe, causing a sharp drop in speed.

    2. Compute affects concurrency and input throughput

    Single-stream conversation is not especially demanding on compute, but the following scenarios tax it more:

    • Heavy prefill input such as long-document summarization or code comprehension;
    • Batch offline inference;
    • Multi-stream concurrent serving;
    • Complex inputs such as high-resolution multimodal content or long audio.

    Here, performance depends not only on the nominal floating-point peak but also on low-precision matrix-unit efficiency and how well matrix-multiply and attention kernels are optimized. A quick rule of thumb: for single-stream, look at bandwidth; for high concurrency, look at the combination of compute and bandwidth.

    3. MoE and quantization: new variables in selection

    • MoE (mixture-of-experts) models: they have many total parameters but activate only a subset per token. They demand large VRAM, while bandwidth needs are closer to the activated-parameter scale. A card with large VRAM and moderate bandwidth may be more suitable;
    • Quantization formats: INT4, INT8, and FP8 each have their own use cases. When selecting, confirm that the inference framework, GPU architecture, and quantization scheme are compatible—don’t just look at the theoretical compression ratio.

    4. Four Dimensions: Reading the Key GPU Specs

    Dimension Key spec Impact on local deployment
    Capacity VRAM capacity Determines how large a model and how long a context you can fit
    Speed Bus width, effective clock, VRAM bandwidth Determines the single-stream generation speed ceiling
    Compute & precision Floating-point throughput, low-precision matrix units Affects high-concurrency throughput and long-input prefill speed
    Ecosystem Architecture generation, quantization support, attention-kernel support Determines whether software runs and runs well

    A few easily overlooked details:

    1. Architecture generation: newer architectures usually support low-precision formats such as BF16 and FP8 better, and may also offer features like memory bandwidth compression;
    2. Efficient attention implementations: some memory-efficient attention kernels rely on newer architecture instructions; older architectures may fall back to naive implementations, increasing VRAM usage and slowing long-context speed;
    3. Power and cooling: high-end cards may draw 250–350W or more, so plan the whole-system power supply and cooling accordingly;
    4. Multi-GPU coordination: multiple cards can scale capacity, but inter-card communication overhead eats into the gains. Consumer cards usually lack dedicated high-speed interconnects, so under the same VRAM budget, a single large-VRAM card is often simpler to manage, especially at low-to-moderate concurrency.

    5. Which Common VRAM Tiers Should You Choose?

    The table below assumes INT4 or low-precision quantization, single-stream, short context as the baseline. For long context, high concurrency, or complex batching, you need to reserve more VRAM.

    VRAM tier Model sizes to consider Typical use
    8–12GB 3B and below; 7B INT4 Lightweight Q&A, personal assistants
    16GB 7B–14B INT4; some 7B INT8 Everyday chat, short-context inference
    20–24GB 32B INT4; 14B INT8 (control context and concurrency) Main chat, document processing, local coding assistance
    48GB and up 70B INT4 (single-stream or short-to-medium context) High-quality generation, batch offline inference

    Two additions:

    • Both 20GB and 24GB can fit 32B INT4; the difference is mainly in concurrency headroom and context budget;
    • The table considers inference only. If you do parameter-efficient fine-tuning such as LoRA locally, you also need to add overhead for gradients and optimizer states, which raises VRAM needs.

    If you go beyond these ranges, consider a cloud GPU instance or a professional accelerator card instead of forcing consumer hardware to handle training, full-parameter fine-tuning, or large-scale offline batch processing.

    6. Running the Numbers: Where Is the Cost-Performance Sweet Spot?

    Hardware investment usually follows diminishing marginal returns:

    • Going from 8GB to 16GB can turn “won’t run” into “usable,” with a clear payoff;
    • Going from 16GB to 24GB mainly improves long-context and high-concurrency ability, with still-considerable returns;
    • Once you enter the professional compute-card range, prices may grow several-fold, but inference speed won’t scale proportionally, because the decode phase is often bottlenecked by bandwidth—excess compute doesn’t necessarily translate into speed.

    So the “sweet spot” isn’t a specific model but the lowest tier that meets your business needs. Price judgments should also account for real-time quotes, since vendors, regions, channels, newness, and promotions all affect cost.

    Recommended selection order:

    1. Clarify the business goal: Q&A, document analysis, or code generation; single-stream or multi-stream concurrency;
    2. Determine the model size and quantization format, and estimate weight VRAM;
    3. Estimate the KV cache based on context length and concurrency;
    4. Estimate decode speed with bandwidth and concurrent throughput with compute;
    5. Check that the inference framework and quantization scheme support the target GPU architecture;
    6. Weigh hardware price, power, cooling, and operating costs, and compare the total cost of self-hosting versus cloud.

    When you’re still inexperienced, you can start with a rough estimate of “parameters ÷ 2 to get INT4 weight GB,” then refine with the script.

    7. Can You Buy Modified or Memory-Upgraded Cards from Non-OEM Vendors?

    The market has cards that are non-OEM modified, memory-upgraded, or heavily altered. Their appeal is straightforward: getting more VRAM at relatively low cost, letting a single card run larger-parameter models.

    But such products can’t be fully evaluated using public specs the way OEM products can. At a minimum, watch for these risks:

    • Compatibility: the memory bus width may not fully match the stated capacity, and access speed across high address ranges may be inconsistent; drivers, firmware, and inference frameworks also need to be verified item by item;
    • Stability: sustained full VRAM load is the most stressful condition, so temperature, cooling, and stability must be thoroughly tested;
    • Warranty and compliance: they may not carry OEM warranty and are hard to fit into a formal procurement process;
    • Deployment and operations: in data-center or edge environments, deployment form, ops tooling, continuous supply, and spare-parts support are all practical constraints.

    The conclusion isn’t “absolutely don’t buy,” but “understand the cost before using.” For personal learning or experimentation in a non-critical environment, pay attention to data security and availability; for production workloads, prefer products that match OEM specs and come with formal support.

    8. Local Deployment vs. On-Demand Cloud: How to Choose?

    Local deployment and cloud usage aren’t either-or; combine them by scenario:

    • Data-sensitive, latency-sensitive, stable-load scenarios: for example, an intranet knowledge-base Q&A system, local deployment is more suitable and long-term cost is more controllable;
    • Bursty traffic, frequently iterating models, proof-of-concept projects: cloud GPU instances are more flexible, with no need to bear hardware purchase or idle costs;
    • Hybrid architecture: run primary inference locally, and use the cloud for elastic scaling, model evaluation, or load-test baselines.

    If you ultimately need to offer a service externally, you can combine it with common cloud capabilities such as container platforms, object storage, logging, and monitoring/alerting to build a complete inference pipeline. For a self-hosted setup, plan ahead for whole-system power, cooling conditions, model-weight and data backup strategies, and long-term health monitoring of VRAM and power.

    9. Common Pitfalls and Practical Recommendations

    1. Common pitfalls

    • “Higher compute means faster”: single-stream generation is mainly bandwidth-bound, so higher compute doesn’t equal faster speed;
    • “Bigger VRAM is always better value”: once capacity is sufficient, piling on more VRAM has diminishing marginal returns;
    • “Quantization has no cost”: INT4 is usable in most scenarios, but tasks such as mathematical reasoning and code generation may need an effectiveness evaluation before going live;
    • “Fits means it runs well”: efficient attention implementations and low-precision matrix units have architecture-generation requirements, and the software ecosystem is part of selection too;
    • “If VRAM is short, just offload to CPU”: first check context, concurrency, KV cache precision, and the quantization scheme; CPU offloading usually comes at a high cost.

    2. A practical three-step method

    1. Model: use the script to estimate VRAM and speed for the target model under different contexts and concurrency levels;
    2. Load test: build test inputs matching your real business distribution, and observe prefill latency, decode speed, VRAM peaks, and temperature;
    3. Finalize: with at least 20% VRAM headroom, choose the lowest-tier configuration that meets your throughput and latency requirements.

    Following this process is how you find the true “sweet spot” that fits your budget, performance, and maintainability.