Deploying Open-Source LLMs Locally: A Practical Guide to VRAM, Quantization, and Model Selection

1. Identifying the Core Trade-offs in Local Deployment

Unlike cloud-based API calls, local deployment means you bear the compute, VRAM, and bandwidth costs entirely. Model selection is fundamentally about balancing three constraints:

• VRAM Capacity: Determines if the model can fit in memory.
• Memory Bandwidth: Determines inference speed. Local inference is almost always bandwidth-bound rather than compute-bound.
• Task Requirements: Dictate how intelligent the model needs to be and how much context length is required.

Beginners often ask, “Which open-source model is the most powerful?” However, in a local scenario, the correct order of questions is: What size model can my hardware run smoothly? → Which models within that size fit my task? → What quantization precision and inference parameters will optimize the experience? Reversing this order inevitably leads to poor choices.

2. Calculating the VRAM Budget: Parameters × Bit Width + KV Cache

Model weight VRAM can be estimated using a simple formula: Weight VRAM ≈ Parameters × Bytes per parameter. 16-bit precision is about 2 bytes/parameter, 8-bit is about 1 byte, and 4-bit is about 0.5 bytes.

def weight_vram_gb(params_b: float, bits: float) -> float:
    """Roughly estimate model weight VRAM usage (excluding KV Cache and runtime overhead)"""
    return params_b * bits / 8
# Example for a 7B model
print(weight_vram_gb(7, 16))  # ≈ 14GB
print(weight_vram_gb(7, 4))   # ≈ 3.5GB

On top of this, you need to reserve VRAM for the KV Cache (the key-value cache for the attention mechanism) and the inference framework’s runtime overhead, generally adding another 1~3GB:

Parameter Scale	16-bit Weights	8-bit Weights	4-bit Weights (plus overhead)
7B	~14GB	~7GB	~3.5GB
14B	~28GB	~14GB	~7GB
~30B	~60GB	~30GB	~15GB

Two crucial points regarding the KV Cache: First, it grows roughly linearly with context length. Expanding context from 4K to 32K will noticeably increase VRAM usage. Second, longer contexts don’t just consume VRAM—they slow down the generation speed of each token. Community benchmarks show that stretching context from 4K to 20K generally drops generation speed by 10% to 20%. Long context is not a free lunch; always leave a speed margin for it.

3. Quantization Precision: 6-bit for Stability, 4-bit for Value, 3-bit Use with Caution

• 6-bit: Minimal quality loss, closely approaching original precision on most tasks. The trade-off is ~50% more VRAM usage than 4-bit and slightly slower speeds.
• 4-bit: The best cost-to-performance ratio. Quality loss is acceptable for the vast majority of daily tasks. However, degradation is more noticeable in long-text detail retention and large-section analysis compared to short-text tasks.
• 3-bit and below: Emergency use only when at absolute VRAM limits. Noticeable capability loss; not recommended as a daily driver.

A frequently verified rule of thumb: When two models have similar capabilities, prioritize the smaller model with higher precision quantization over the larger model with aggressive quantization. However, when there is a significant scale gap, a “larger model + 4-bit” often still outperforms a “smaller model + 6-bit.” There is no standard answer for quantization; the key is assessing how sensitive your task is to precision.

4. Dense vs. MoE: Architecture Determines the Ceiling of Mid-Range GPUs

• Dense Models: Every token activates all parameters. VRAM usage and speed scale synchronously with total parameters. Dense models over 20B, even compressed to 4-bit, often yield only single-digit token/s speeds on mid-range GPUs, making them practically unusable.
• MoE (Mixture of Experts) Models: Total parameters can be very large, but each token only activates a small fraction. VRAM usage depends on total parameters, but speed depends on activated parameters. Combined with an offloading strategy where partial expert weights reside in system RAM, a ~30B total parameter MoE model with single-digit activated parameters can achieve around 50 token/s at 4-bit and ~40 token/s at 6-bit on a consumer-grade setup with 12GB VRAM + 32GB RAM (based on community benchmarks; actual results vary with memory frequency, offload ratio, and context length).

This is currently the most mainstream “bang for the buck” route for mid-range GPU users, and a priority direction worth trying during local deployment model selection.

5. Speed Baselines: How Many token/s is “Usable”

• Below 10 token/s: Noticeable stuttering. The experience feels like staring at a blinking cursor. Downgrading the model or quantization is recommended.
• 20~40 token/s: Meets or exceeds human reading speed. Smooth interaction. This is the comfort zone for daily use.
• Above 50 token/s: Near-instant replies. Ideal for high-frequency scenarios like code completion and batch processing.

If you fall below the baseline, troubleshoot in the following order:

1. Reduce quantization bit width (e.g., 6-bit → 4-bit).
2. Disable thinking mode (see Section 8).
3. Shorten the context length; do not max it out blindly.
4. Ensure all model layers reside on the GPU rather than spilling over to the CPU.
5. In CPU offload scenarios, upgrading to high-frequency dual-channel RAM yields significant returns.

6. Selection Guide by VRAM Tier

• 8GB and below: 4~6-bit for 7B-class dense models, or 1.5B~4B small models. Suitable for lightweight Q&A and text processing.
• 12GB: 6-bit for 7B~14B dense models is the sweet spot. 4~6-bit for ~30B low-activation MoE models with moderate offloading can balance capability and speed. Dense models over 20B are not recommended.
• 16GB: 6-bit for 14B dense models runs smoothly. ~30B MoE models can use 6-bit. 3~4-bit for ~20B dense models is barely usable but speed is concerning.
• 24GB: 4~6-bit for 20B~30B dense models enters the practical range. ~30B MoE models can use higher precision with longer context. This is widely recognized as the “sweet spot” tier.
• Large unified memory devices: Can load larger models, but speed is constrained by memory bandwidth. Better suited for offline tasks where speed is not critical.

7. Selecting Models by Task: The Triangle of Capability, Speed, and Long Context

A more important question than “which model is strongest” is “what does my task value most?”:

• Daily Q&A, standard writing: Mid-range models are more than enough; prioritize speed.
• Long document summarization, large section analysis: This is the easiest scenario to fail. In practice, a common phenomenon is that some larger parameter models perform better on short tasks but are less stable in logical and narrative coherence over ultra-long contexts compared to slightly smaller models. Long-text capability is highly sensitive to model architecture and training organization, and it is not strictly positively correlated with parameter count—it is better to sacrifice some “smartness” to ensure reliable and coherent long-text output.
• Complex reasoning, code generation, Agent orchestration: Prioritize the model’s capability ceiling. If speed is insufficient, compensate with quantization, disabling thinking mode, or adding more memory.

A practical bottom-line rule: If a model cannot even hit 10 token/s on your hardware, abandon it regardless of how powerful it is. The pain of waiting will negate any capability advantage.

8. Inference Parameters and Security Practices

# Illustrative configuration; field names depend on the actual inference tool used
model: ./models/my-model-4bit   # Local quantized model file
ctx_len: 8192                   # Set as needed; avoid maxing out blindly
gpu_layers: all                 # Keep all layers on GPU if VRAM permits
thinking: false                 # Disable thinking mode for fast response scenarios

A few additional notes:

• Thinking Mode: Many next-generation open-source models have a built-in “thinking mode” that outputs a reasoning process before answering. This improves quality but exponentially increases token consumption. It is recommended to disable it for summarization, rewriting, and simple Q&A, and enable it only for complex reasoning.
• Context Configuration: 4K~8K is sufficient for default tasks. Only increase gradually if long document processing is explicitly required, while simultaneously observing speed degradation.
• Model Source Security: Only download model files from official release channels or trusted mirrors. Verify hashes after downloading to avoid unverified “optimized” or “merged” versions.
• Benchmark Before Settling: Run a few typical samples from your actual tasks against candidate models, recording speed and quality. This is more reliable than looking at any leaderboard.

9. When Local Deployment Fails: Complementing with the Cloud

If you have exhausted your selection options and still cannot run the target task locally, don’t force it. You can treat local and cloud as complementary layers:

• Data-sensitive, high-frequency lightweight tasks belong locally, offering privacy control and zero API call costs.
• Occasional heavy reasoning and long context tasks can be delegated to cloud-based general-purpose LLM services. Major cloud platforms like Baidu AI Cloud offer pay-as-you-go model services, providing maintenance-free, elastic scaling, and on-demand usage.

This hybrid model of “local small model for daily baseline + cloud large model for on-demand enhancement” is currently the most cost-effective practical path for individual developers and small teams.

10. Conclusion: A Decision Checklist

1. Measure VRAM first, then talk models: Estimate if it fits using “Parameters × Bit Width / 8 + 1~3GB overhead”.
2. Mid-range GPUs should prioritize the low-activation MoE route; choose dense models within your means.
3. Start with 4-bit quantization, use 6-bit for stability; below 4-bit is for emergencies only.
4. Downgrade decisively if speed drops below 10 token/s; only discuss user experience above 20 token/s.
5. Do not blindly trust large parameters for long-text tasks; smaller models + stable long context are often more reliable.
6. Thinking mode, context length, and GPU layer residency are the three most effective optimization switches.
7. When hardware hits its limit, bridge the gap with cloud-based general model services instead of forcing an all-in local setup.

The joy of deploying open-source LLMs locally lies in maximizing capability through careful calculation on limited hardware. By following the framework above, you will likely find the model best suited for your GPU within an hour.