Choosing a Local LLM: VRAM, Quantization, and Architecture Trade-offs

    This article explains how to choose a local large language model based on available VRAM, quantization precision, model architecture, context length, and real throughput. It covers weight-size estimation, KV cache and runtime overhead, dense versus mixture-of-experts speed, measured results on a 12 GB VRAM machine, task-based model selection, and common pitfalls.

    1. Start with VRAM

    The first barrier to local inference is VRAM. Model weights must fit entirely into VRAM or system memory to participate in computation. When the weights do not fit, some layers are moved to system memory, and speed drops noticeably. The first step in model selection is therefore to confirm how large a model the hardware can hold, not to compare leaderboard scores first.

    The decision order can be summarized in three steps: determine available VRAM and total system memory, then choose quantization precision, and finally consider model architecture and context length. These three steps interact; changing the assumption in any one step changes the final usability.

    2. Estimating VRAM Usage

    The size of model weights can be estimated directly from parameter count and bit width:

    1. def estimate_weight_gb(params_b: float, bits: float) -> float:
    2. """Parameters are in billions (B); returns weight size in GB."""
    3. return params_b * bits / 8
    4. for name, params, bits in [
    5. ("9B 6bit", 9, 6),
    6. ("35B 6bit", 35, 6),
    7. ("35B 4bit", 35, 4),
    8. ("27B 4bit", 27, 4),
    9. ]:
    10. print(f"{name}: about {estimate_weight_gb(params, bits):.2f} GB")

    The results above are: a 9B model at 6-bit is about 6.75 GB, a 35B model at 6-bit is about 26.25 GB, a 35B model at 4-bit is about 17.5 GB, and a 27B model at 4-bit is about 13.5 GB. Actual files are usually slightly larger because of mixed-precision layers.

    Beyond weights, two additional areas need headroom. The first is KV cache, which grows linearly with context length; longer contexts use more. The second is runtime framework overhead and temporary buffers. As a rule of thumb, reserve at least 1 to 2 GB of headroom, and more for long-context scenarios.

    3. Quantization: Trading Precision for Size

    Quantization compresses weights from higher to lower bit width. Lower bit width means smaller size and less data movement during inference, but greater precision loss. Common trade-offs include:

    • 6-bit quantization: smaller precision loss and larger size; suitable when VRAM is plentiful or output quality is important.
    • 4-bit quantization: about two-thirds the size of 6-bit and the most common choice on consumer hardware; quality loss is acceptable for most general chat and code tasks.
    • Lower bit widths: even smaller, but degradation is more noticeable in tasks such as mathematical reasoning and long-text understanding.

    When choosing quantization, first ensure the model can fit entirely into VRAM or a VRAM-plus-system-memory combination. Then choose the highest precision that fits in the available space.

    4. Dense vs. Mixture-of-Experts Speed

    Decoding speed is mainly limited by memory bandwidth. Each generated token requires reading the weights involved in computation once. A rough upper bound can therefore be estimated as:

    Theoretical decode speed ceiling ≈ effective memory bandwidth ÷ bytes actually read per token.

    A dense model reads all weights for every token. A mixture-of-experts (MoE) model activates only some experts per token, so the actual amount of data read is much smaller. For example, a model with 35B total parameters and about 3B active parameters needs to read only about 2.25 GB of active parameters per token at 6-bit.

    This explains a common phenomenon: a larger MoE model may run faster than a smaller dense model, provided that non-active experts can be placed in system memory and access to the active portion does not become a bottleneck.

    Dense models are different. A 27B dense model at 4-bit is about 13.5 GB. When it exceeds 12 GB of VRAM, many layers must be computed in system memory. Every token passes through those layers, and speed drops sharply.

    5. Interpreting a Set of Measured Results

    The following data comes from a machine with 12 GB VRAM, a mid-range desktop processor, and 32 GB DDR5 memory (6000 MT/s), using a local inference tool to run a 35B total-parameter MoE model:

    • 6-bit quantization, short context (about 4K tokens): about 42 tok/s; when context increases to about 20K: about 37 to 39 tok/s.
    • 4-bit quantization, short context: about 55 tok/s; under long context: about 50 tok/s.
    • After disabling thinking mode in the chat template, output is more direct and the number of generated tokens decreases, so perceived response time is shorter.

    On the same machine, a 9B dense model runs at about 56 tok/s at 6-bit and about 70 tok/s at 4-bit. Its short-text speed is not much faster than the MoE model above, but it responds faster and more stably in later processing of very long contexts.

    A 27B dense model at 4-bit runs at only about 3.6 tok/s, which is basically unusable for interaction. This shows that on the same hardware, differences in architecture and VRAM footprint have a greater impact than differences in quantization level.

    Three conclusions follow from this data. First, speed cannot be evaluated separately from context length. Second, the model with the largest parameter count is not necessarily the best fit for the current hardware. Third, model capability and speed must be considered separately; if a capable model cannot run smoothly, its practical value is reduced.

    6. Choosing by Task Type

    Different tasks have different requirements, and model selection should be organized around the task:

    1. Interactive Q&A and rewriting: prioritize response speed; a medium-parameter model with 4-bit quantization is usually sufficient.
    2. Very long document summarization and retrieval-based Q&A: prioritize stable context handling; a smaller dense model may be more reliable than a large MoE model.
    3. Complex reasoning and code generation: when VRAM allows, prioritize higher precision and stronger base capability, and accept lower speed.
    4. Batch offline processing: single-request speed is less sensitive; choose a larger model and measure efficiency by total throughput.
    1. List the hardware budget: VRAM, system memory capacity, and memory bandwidth.
    2. Determine the typical context length and estimate KV cache usage.
    3. Within available space, choose the largest parameter model that can fit entirely and the highest feasible precision.
    4. Run small-scale tests on real tasks, recording time to first token, generation speed, and output quality.
    5. If speed is insufficient, first consider reducing quantization bit width or shortening context; only then consider changing the model.

    8. Common Misconceptions

    Misconception 1: Larger parameter counts are always better. In VRAM-constrained environments, the portion that exceeds VRAM significantly slows down execution, and a large model is not necessarily better than a medium model that can run entirely.

    Misconception 2: Looking only at short-text speed. Long contexts increase KV cache usage and increase computation per token, so speed drops.

    Misconception 3: Treating leaderboard scores as the local experience. Leaderboards are usually measured in the cloud or on high-end hardware and are not equivalent to local speed, stability, and context behavior.

    Misconception 4: Ignoring system memory capacity and bandwidth. When part of a model must reside in system memory, memory frequency and capacity directly affect the experience.

    Selecting a local LLM is essentially a trade-off among VRAM, precision, architecture, and task requirements. By mastering size estimation, understanding bandwidth limits, and validating with real tasks, you can make stable and reproducible decisions on limited hardware.