Local LLM Deployment: How to Choose VRAM, Quantization, and Speed
Running large language models on your own machine is a trade-off between model capability, quantization, context length, and speed under a hard VRAM limit. This guide shows how to estimate weight and KV cache memory, how quantization levels affect footprint and quality, how dense and MoE architectures differ in practice, why long contexts slow down generation, how to measure token/s reproducibly, and which model tiers realistically fit 12GB, 16GB, and 24GB GPUs. It closes with a five-step decision framework for matching a local setup to your hardware and workload.
1. The core tension in local deployment: not enough VRAM
Running a large language model locally is a contest between capability and constrained resources. Desktop GPUs typically offer 12GB, 16GB, or 24GB of VRAM, while model capability, output quality, and context length all pull against each other. Most people starting out ask the same three questions: what do I do when VRAM runs out, how big a model should I pick, how should I choose quantization, and how do I measure token/s?
These questions cannot be answered in isolation. Judging capability without considering VRAM leads to models that simply will not run. Judging precision without considering the task means paying for quality you will never use. Judging parameter count without considering speed ignores the actual experience. The breakdown below moves from hard constraints to softer choices.
2. VRAM is the first hard gate: estimate weights and KV cache
To run at all, a model needs VRAM (or VRAM plus system memory) large enough for two categories of data:
- Model weights: determined by parameter count and quantization bit width. This is a fixed cost.
- KV cache: grows roughly with context length. This is a dynamic cost.
A rough formula is enough to build intuition:
def estimate_vram_gb(param_b, bits=4, context_len=4096):bytes_per_param = {16: 2.00, # FP16 baseline8: 1.05, # 8-bit quantization6: 0.78, # 6-bit quantization5: 0.66, # 5-bit quantization4: 0.55, # 4-bit quantization3: 0.42, # 3-bit quantization}.get(bits, 0.55)weights_gb = param_b * bytes_per_paramkv_gb = (context_len / 1024) * param_b * 0.02return round(weights_gb, 2), round(kv_gb, 2), round(weights_gb + kv_gb, 2)w, kv, total = estimate_vram_gb(35, bits=4, context_len=4096)print(f"Weights ~{w} GB, KV cache ~{kv} GB, total ~{total} GB")
With roughly 35B parameters, 4-bit quantization, and a 4K context, weights come to about 19.25GB, the KV cache to about 2.8GB, for a total of about 22.05GB. A 12GB card cannot hold this at all. A 16GB card will be tight. A 24GB card can run it, but only with tighter context control, attention to framework overhead, or a lower-precision KV cache.
This formula is a rough estimate. It covers weights and a simplified KV cache only. It does not include inference framework overhead, CUDA and driver context, transient activation tensors, system display usage, or desktop environment usage. The real KV cache also depends on the actual number of layers, attention heads, whether GQA/MQA is used, head_dim, and whether KV quantization is enabled, so recalculate it against the specific model architecture.
3. Choosing quantization: 4-bit is the sweet spot, 6-bit if you have headroom
Quantization compresses model weights from higher-precision formats such as FP16 down to 3–8 bits, trading a controlled quality loss for a smaller footprint and a lower resource threshold.
| Quantization | Relative size | Quality | Best fit |
|---|---|---|---|
| FP16 | 100% | Best | Plenty of VRAM, chasing maximum quality |
| 8-bit | ~50% | Close to baseline | Quality-sensitive, comfortable resources |
| 6-bit | ~39% | Good | Balance of quality and footprint |
| 5-bit | ~33% | Fairly good | Mid-range VRAM, quality and speed |
| 4-bit | ~27% | Acceptable | Mainstream choice when VRAM is tight |
| 3-bit | ~21% | Noticeably worse | Extreme compression, just get it running |
The table shows theoretical relative sizes. Actual usage also includes group scaling factors, zero points, metadata, quantization calibration data, and additional inference framework overhead, typically 5%–15% above the pure bit-width calculation.
In practice, 4-bit is the most common balance point in local deployment today. If you have VRAM left over, 6-bit usually delivers steadier expression quality and less degradation. 8-bit is closer to the original precision, but VRAM requirements rise noticeably and usually demand a larger GPU or a shorter context.
The lower the bit width, the more likely you are to see repetition and rambling, disjointed logic, and weaker instruction following. This is especially visible in tasks that demand rigor: complex reasoning, long-document summarization, and code review. A practical approach is to validate with 4-bit that the model runs stably, then decide whether to move to 5-bit or 6-bit based on available speed and remaining VRAM.
4. Parameter count and architecture: the real difference between dense models and MoE
Many people assume more parameters always means slower. That holds roughly for dense models, because every token passes through all parameters. MoE architectures break that intuition.
Dense models use all their parameters. More parameters mean more computation, and usually lower speed.
MoE models separate total parameters from active parameters. Consider a MoE model with 35B total and 1–4B active parameters. The full weight set is still large, but each generated token activates only a small subset of experts. That produces two seemingly contradictory outcomes:
- VRAM pressure stays high: all expert weights must be resident or reachable, so total size remains close to the full parameter count.
- Compute speed can still be good: fewer parameters participate per token, so even with some weights off VRAM, speed can beat a large general-purpose dense model.
This is why a 12GB GPU paired with a large amount of system memory can run a 30B-class MoE model with 1–4B active parameters more comfortably than a 27B dense model. If the dense model’s weights do not fit in VRAM and it is forced to offload heavily to memory, its speed can collapse.
MoE speed is not an unconditional advantage, though. It depends on whether the inference framework routes efficiently, whether memory bandwidth is sufficient, whether the offload strategy is sound, and whether the model’s expert structure suits your hardware. Long-context coherence and instruction stability also need to be measured rather than inferred from total and active parameter counts.
5. Context length: why long contexts slow down noticeably
With the same model and the same quantization, context length has a large effect on speed, and the cause is the KV cache.
Every decoding step reads and appends to the KV cache. The longer the context, the larger the cache processed per step, so decode speed drops. Going from 4K to 20K or 32K and seeing a slowdown is expected behavior.
If your workload is mostly short conversations, code completion, or formatted generation, you can quantize more aggressively and prioritize speed. If it is long-document summarization, analysis of large blocks of material, or multi-turn memory, reserve VRAM for the KV cache and expect speed to fall.
A simple measurement template helps:
| Context length | Prefill token/s | Decode token/s | Peak VRAM | Usable |
|---|---|---|---|---|
| 4K | record | record | record | record |
| 8K | record | record | record | record |
| 16K | record | record | record | record |
| 32K | record | record | record | record |
Only by holding model, quantization, framework, and hardware constant can you tell what context length actually does to your deployment.
6. Reading inference speed: how to measure token/s
Local inference is usually measured in token/s, but the two phases must be separated:
- Prefill: processes the input prompt in one pass. Affects time to first token.
- Decode: generates output token by token. Affects streaming speed.
The “does it feel sluggish” question is mostly answered by decode throughput. A rough guide:
- Above 30 token/s: generally smooth, close to real-time conversation.
- 15–30 token/s: usable, but you notice token-by-token generation on long answers.
- Below 10 token/s: poor experience; long outputs are painful to watch.
These thresholds only mean something under a consistent measurement setup. Record the GPU model, quantization format, inference framework and version, context length, whether offload is enabled, batch size, whether concurrency is used, and whether a long system prompt is present. Without that, token/s numbers from different environments are not comparable.
A reproducible baseline is: same model, same quantization, same framework, 4K context, batch size 1, no concurrency, offload status stated explicitly, then compare. Only with a consistent baseline can you tell whether a problem lies in model choice, quantization, or framework tuning.
7. Choosing by hardware tier: 12GB, 16GB, and 24GB
The realistic model range varies a lot by VRAM tier.
12GB VRAM: mainstream entry point
- Prefer MoE models with small active parameter counts, for example 30B-class total with 1–4B active, paired with a large amount of system memory for usable speed.
- For dense models, aim to stay at 9B or below.
- Trying a 27B-class model at 4-bit means accepting low speed, frequent offloading, or a short context limit.
16GB VRAM: intermediate
- 27B-class models at 4-bit are worth trying, but headroom is limited with long contexts, concurrency, or higher-resolution tasks.
- Mid-sized MoE models are usually steadier.
- If you want higher quality, securing 4-bit is a better move than blindly increasing parameter count.
24GB VRAM: comfortable
- Generally well suited to running 27B-class dense models at 4-bit.
- With enough headroom left, 6-bit is worth trying for better quality.
- 8-bit usually still requires more VRAM or stricter context control.
- Unified-memory devices are an alternative path, but real speed depends heavily on memory bandwidth.
The core logic: use VRAM to set an upper bound on model size, then pick precision, parameter count, and context length to fit the task within that range.
8. Matching model capability to the task: there is no universal winner
Models differ in strengths and weaknesses. Selection should return to the task rather than assuming bigger is better.
- Strong reasoning and complex logic: favor smaller models at larger parameter counts or higher-quality quantization, and trade speed for quality when necessary.
- Very long text, long-document summarization, coherent narrative: MoE or extremely small models are not automatically better; measure consistency at long context.
- Instruction-following stability: smaller models are sometimes more obedient than larger MoE models, with more controllable output.
- High-throughput batch generation: once quality is acceptable, choose the configuration with better speed and VRAM efficiency.
It helps to build a task-to-configuration table with at least these fields:
| Task type | Latency tolerance | Minimum quality | VRAM usage | Measured speed |
|---|---|---|---|---|
| Everyday Q&A | Low latency | Medium | record | record |
| Long-document summarization | Willing to wait | Higher | record | record |
| Code completion | Fast response | Higher | record | record |
| Complex reasoning | Willing to sacrifice speed | High | record | record |
Use fast models for frequent, latency-sensitive tasks and switch to heavier models for occasional complex work. That is more practical than forcing one configuration to handle everything.
9. A practical optimization checklist
Once a model is chosen, there is still room to optimize:
- Quantization choice: start with 4-bit, move to 6-bit if VRAM allows. In long-context scenarios, drop a level to free room for the KV cache.
- Sensible offload: keep critical layers in VRAM and push cold layers to memory. A reasonable tuning order is to try placing all layers on the GPU first, reduce GPU layer count when you hit OOM, and if speed drops sharply, move fewer layers back to the GPU or shorten the context. Some frameworks expose GPU layer count, CPU offload layer count, and VRAM-to-memory ratios; test them one at a time.
- KV cache quantization: when the framework supports it, compressing the KV cache eases long-context VRAM pressure. Support varies widely: llama.cpp generally offers flexible KV quantization options; vLLM supports low-precision KV cache on some versions and hardware; TensorRT-LLM requires configuring it in the engine build or quantization pipeline. KV quantization can introduce slight quality loss, so compare output quality in production.
- Attention optimizations such as Flash Attention: reduce VRAM and compute cost on long sequences, which is especially effective for long contexts.
- Thinking mode toggle: some models let you disable built-in thinking or reasoning for faster responses and cleaner output. Disabling it can change model behavior, task performance, and output safety boundaries, so re-evaluate before shipping.
- Batch size and concurrency control: do not raise concurrency blindly in local deployments. Too much concurrency creates contention for VRAM, compute, and KV cache, lowering overall speed. Set a sensible queue length based on hardware capacity.
10. A five-step decision framework
Local LLM deployment decisions can be reduced to five steps:
- Set the ceiling from VRAM: 12GB, 16GB, and 24GB correspond to different maximum model sizes.
- Choose a path from architecture: when VRAM is tight and speed matters, prefer MoE models with small active parameters, but watch framework routing and offload efficiency.
- Balance with quantization: start at 4-bit, raise precision when speed allows, and reserve space for the KV cache with long contexts.
- Decide trade-offs from speed: if a high-precision large model cannot reach usable token/s, drop a quantization level or switch to a smaller model.
- Match to the task: build a mapping from tasks to configurations rather than assuming bigger is better.
There is no single right answer for local LLM deployment, only the optimum at the intersection of your hardware, your task, and the latency you can tolerate. Understanding how VRAM, quantization, architecture, and context length interact is what makes the choice between quality, speed, and resources a rational and reproducible one.
