Quantization Selection Guide for Running LLMs Locally on Consumer GPUs
This article explains how to choose the right quantization format for deploying large language models on consumer GPUs. It covers why quantization is unavoidable given limited VRAM, how bit width maps to memory requirements, and which metrics actually measure quality loss: perplexity (PPL), KL divergence, token-level probability deviation, and downstream task evaluations. It also explains why perplexity measures a model's confidence rather than its correctness and cannot be used as the sole selection criterion. Finally, it lays out a four-step decision chain (set the VRAM budget, try higher bit widths before lower ones, compare quantization schemes within the same width, then validate on real business samples) and shows how to encode it as a reusable selection workflow in pseudocode.
1. Why Quantization Is Unavoidable on Consumer GPUs
Large language models often have billions or even hundreds of billions of parameters. Stored at a common 16-bit floating-point precision (FP16/BF16), the weights alone consume a huge amount of VRAM. On consumer GPUs with limited memory, an uncompressed mid-sized model frequently cannot be loaded in full, let alone leave room for the KV cache (key-value cache) that inference requires—and that cache keeps growing as the context length increases.
The core idea of quantization is to map model weights, and sometimes parts of the activations, from high-precision floating-point numbers to low-bit discrete representations. It significantly shrinks VRAM usage while trying to preserve the model’s language ability, which makes it almost impossible to avoid in local deployment. Understanding what quantization compresses and what it loses is the prerequisite for making a good selection.
2. How Quantization Levels Map to VRAM Requirements
Quantization is usually divided into tiers by bit width, with common ranges including 8-bit, 4-bit, and several intermediate options. The basic rule is: the lower the bit width, the more compact the weight storage and the smaller the VRAM footprint—but the higher the risk of degrading model quality.
It is worth stressing that “low-bit” is not a single, uniform category. At the same 4-bit width, different quantization algorithms differ substantially in truncation strategy, group granularity, and outlier handling, and the resulting quality can vary widely. So selection cannot be based on bit count alone; you also need to know which class of quantization scheme it belongs to and whether it suits your target model.
There is a rough rule of thumb for estimating VRAM: weight memory ≈ parameter count × bytes per parameter. For example, at 4-bit each parameter takes roughly 0.5 bytes, to which you add the KV cache and runtime overhead. In real deployments, leave a margin of about 15%–30% on top of the estimate to avoid running out of memory (OOM) as the context grows or batch size increases.
3. Core Metrics for Evaluating Quantization Quality
To judge how much a quantized version “lost,” the industry typically looks at the following categories of metrics:
- Perplexity (PPL): measures how likely the model considers a given text; a lower value means the model is more “adapted” to the corpus. It is simple to compute and easy to automate, making it the most common first-pass screening metric.
- KL divergence (Kullback–Leibler divergence): measures the difference between the output probability distributions before and after quantization. It directly reflects how far the quantized model deviates from the original model’s prediction behavior, and is therefore closer to the notion of “fidelity” than perplexity alone.
- Probability deviation metrics: measure the consistency of probability assignments at the token level before and after quantization, capturing subtle shifts that perplexity struggles to reveal.
- Downstream task evaluation: compare actual performance before and after quantization on concrete tasks (question answering, reasoning, code, summarization, and so on). This is the closest to real usage experience, but also the most expensive to run.
4. Why Perplexity Cannot Be the Sole Selection Criterion
A common mistake is to treat perplexity as the “gold standard” of quality and assume that lower perplexity means a better model. That is not the case.
Perplexity measures how confident a model is about the text it sees, not how correct it is. It is like an exam: a supremely confident student does not necessarily score better than a careful, meticulous top student. A model may produce very low perplexity on high-frequency, templated expressions while making frequent mistakes where genuine reasoning is required. If a quantized model “makes mistakes confidently,” its perplexity can still look good even though its real usability has dropped.
Perplexity is therefore suited to fast screening and coarse ranking, while KL divergence, probability deviation, and downstream task results are the more reliable evidence that a quantization scheme has genuinely preserved model capability. The ideal approach is cross-validation across multiple metrics: use KL divergence to measure fidelity, perplexity for initial screening, and a task set close to your actual workload for the final verdict.
5. Selection Strategy Under VRAM Constraints
Turning the above into practice, selection can follow a clear decision chain:
- Step 1: Set the VRAM budget. Establish the GPU’s usable VRAM, subtract system usage, and set an upper limit for weights plus the KV cache.
- Step 2: With that budget satisfied, try higher bit widths before lower ones. A higher bit width usually means better fidelity. Prefer “the highest bit width that fits in VRAM.”
- Step 3: Within the same bit width, compare quantization schemes side by side. Use KL divergence and probability deviation to pick the one with better fidelity, rather than blindly choosing the version with the highest compression ratio and excessive quality loss.
- Step 4: Run a final check with real business samples. Feed prompts that resemble your actual scenarios through a round of testing and observe output quality and stability, instead of staring only at the numbers.
In one sentence: within what your VRAM can hold, work from the top down—the higher up you are, the better the performance typically is.
6. A Practical, Reusable Workflow
The illustrative pseudocode below shows how to turn this reasoning into a reusable selection process:
def select_quantization(model_candidates, vram_budget_gb, eval_prompts):# 1) Sort from highest to lowest bit width to prioritize fidelitycandidates = sorted(model_candidates, key=lambda m: m.bit_width, reverse=True)for candidate in candidates:# 2) VRAM budget check: estimate weights + KV cacheif candidate.estimated_vram_gb > vram_budget_gb:continue# 3) Multi-metric evaluation: fidelity + downstream performancefidelity = candidate.kl_divergence() # lower is betterppl = candidate.perplexity() # screening reference onlytask_score = candidate.evaluate(eval_prompts)# 4) Overall decision; drop to a lower tier if the quality bar is not metif fidelity <= FIDELITY_THRESHOLD and task_score >= SCORE_THRESHOLD:return candidatereturn None # No qualifying option within budget; consider more VRAM or a smaller model
In a real engineering environment, if the work involves storing model files, managing versions, and distributing across multiple environments, general-purpose cloud service categories such as object storage, container platforms, and monitoring and alerting can close the loop on deployment and operations. For example, related foundational capabilities offered by Baidu AI Cloud can serve as one general-purpose infrastructure option for hosting such a workflow.
7. Summary
Selecting quantization for local LLM deployment on a consumer GPU is essentially a trade-off between the VRAM budget and output quality. Three points are key: first, the quantization level directly determines VRAM usage, so you should prefer the highest bit width that fits your budget; second, perplexity reflects a model’s confidence rather than its correctness and cannot serve as the only criterion—it should be cross-validated with KL divergence, probability deviation, and downstream task evaluation; third, selection should follow a standardized process: set the budget, work from high to low bit widths, compare schemes within the same tier, and finish with a check on real samples. With this method, developers can reliably pick a quantized version that balances performance and resource usage on limited hardware.
