Open-Source LLM Local Deployment: Performance Evaluation and Engineering Best Practices

1. Background: Open-Source LLMs Enter the “Near-Frontier” Era

Over the past year, open-source LLMs have iterated at a markedly faster pace. The latest flagship open models can now compete head-to-head with closed-source frontier models on multiple public benchmarks. More importantly, two engineering trends stand out:

1. Native multimodality is becoming standard: Visual understanding is built directly into base models rather than bolted on through external adapters. Image comprehension is shifting from a “bonus feature” to a “baseline requirement”—locally deployed models can also view images, read tables, and parse screenshots.
2. Small-activation MoE architectures are spreading: Mixture-of-Experts (MoE) models with total parameter counts in the hundreds of billions but only single-digit billions of activated parameters per inference are gaining traction. The capability ceiling of such models is set by total parameters, while inference cost is set by activated parameters—turning the idea of running “large models” on consumer hardware from a slogan into reality.

Together, these two trends have pushed local deployment from a geeky toy toward a practical stage. But to use models well and reliably, you need a systematic approach to evaluation and tuning—not just “it runs, so it works.”

2. Performance Evaluation: Look Beyond Benchmarks to Real Tasks

2.1 The Reference Value and Limits of Benchmark Scores

Public benchmarks (knowledge Q&A, code generation, math reasoning, agent tool calling, etc.) remain a handy way to shortlist models, but there are two common pitfalls:

• Benchmark gaming: Some models train extremely long reasoning chains to chase scores. The numbers look good, but the actual experience suffers—thinking-token overhead far exceeds what is necessary.
• Coverage blind spots: Long-tail scenarios such as format constraints in specific linguistic contexts, negative instructions, and multi-turn conversation memory are often not captured by general benchmarks.

2.2 Build Your Own Evaluation Set

A more reliable approach is to build a small private evaluation set around real business scenarios, covering at least four dimensions:

• Instruction following: JSON output, word-count limits, combinations of multiple constraints, negative instructions;
• Agent capabilities: Tool-calling accuracy, multi-step task completion rate, self-recovery after errors;
• Coding capabilities: Completion, refactoring, and bug localization on real codebases—not just standalone algorithm problems;
• Long-context capabilities: Needle-in-a-haystack tests, long-document summarization, cross-paragraph information integration.

A phenomenon repeatedly confirmed in community practice: many models can match top closed-source models on agent-style tasks, yet tend to “overthink” simple brain teasers. This suggests evaluation must cover both hard and easy tasks; otherwise, conclusions will carry systematic bias.

3. Quantization Strategy: The First Technical Decision in Local Deployment

3.1 Choosing Weight Quantization

Quantization Level	Typical Characteristics	Suitable Scenarios
Q8 / FP8	Quality loss almost negligible	Ample VRAM, pursuit of maximum quality
Q4 series	Best quality-to-size balance	The default choice for most consumer deployments
Q3 and below	Quality degradation becomes noticeable	A compromise when VRAM is severely constrained

Two easily overlooked points:

1. Instruction following is more sensitive to quantization versions. Community testing has reported that quantized releases of the same model from different maintainers can differ in instruction-following performance, with some versions showing clear regression. It is advisable to compare multiple quantized versions of the same model and validate with your own evaluation set before finalizing the choice.
2. KV Cache quantization is a lifesaver for long contexts. Dropping KV Cache from FP16 to Q8 nearly doubles context capacity, with usually minimal quality loss. For contexts above 100K, this is almost a must-enable option.

3.2 A Typical Dual-Model Configuration Approach

Take a “dense model + small-activation MoE model” combination as an example (the numbers below are order-of-magnitude references from community testing; specifics vary by hardware, framework version, and quantization scheme):

• Dense model with INT4 weights + Q8 KV Cache + multimodality: on a consumer GPU with modified VRAM, it can sustain generation at around 20 tokens per second while carrying roughly 100K context;
• MoE model (about 3B activated parameters) with INT3 quantization + Q8 KV Cache: generation speed can reach nearly 100 tokens per second, with support for 256K-level context.

# Dual-model routing configuration sketch (pseudocode, not tied to any specific framework syntax)
models:
  quality:                  # Quality tier: dense model
    quant: int4
    kv_cache: q8
    max_context: 100k
    multimodal: true
  fast:                     # Speed tier: small-activation MoE model
    quant: int3
    kv_cache: q8
    max_context: 256k
router:
  simple_query: fast        # Everyday high-speed requests
  complex_task: quality     # Complex reasoning tasks
  sensitive_data: local_only

A GPU in the roughly two-thousand-yuan class is enough to stand up a “quality tier + speed tier” dual-model service—this is precisely the qualitative change brought by MoE architectures.

4. Inference Acceleration: Multi-Token Prediction and Other Techniques

Beyond traditional operator optimization and parallelism strategies, the most noteworthy recent development is Multi-Token Prediction (MTP):

• Traditional autoregressive decoding produces only one token per step; MTP lets the model predict multiple tokens per step, with a verification mechanism ensuring output consistency;
• It is especially friendly to local deployment: no multi-GPU cluster required—a single card can achieve considerable speedup;
• Major open-source inference frameworks have been adding support, so upgrading the framework version often yields “free” performance gains.

Other conventional optimizations that can be layered on include paged KV Cache management, continuous batching, speculative decoding, and operator fusion. It is recommended to run a baseline stress test first to determine whether the bottleneck is memory bandwidth or compute, then target it accordingly—avoid blind parameter tuning.

5. The Cost Ledger: Local Deployment vs. Cloud API Calls

The real cost of local deployment involves three accounts:

1. Hardware account: A single consumer GPU can run near-frontier models, with initial investment in the low thousands; multi-GPU setups, larger contexts, or higher concurrency multiply the cost;
2. Power and depreciation account: Power consumption and GPU depreciation for 24/7 operation are not zero over the long run;
3. Operations account: Environment setup, framework upgrades, quantized-version management, and stability assurance are all hidden labor costs.

The cloud MaaS model bills per token, with zero ops and elastic scaling—suitable for scenarios with fluctuating traffic or the need for ultra-large-scale model capabilities. The pragmatic conclusion is therefore a hybrid strategy:

• High-frequency, low-sensitivity, latency-tolerant everyday tasks → local small-activation models, with marginal cost approaching zero;
• Low-frequency, high-difficulty tasks requiring top-tier capability → call cloud flagship model APIs (for example, via model services on platforms such as Baidu AI Cloud);
• Sensitive-data tasks → local deployment inherently satisfies compliance requirements for data not leaving the premises—this is the hardest value of local solutions.

6. Common Pitfalls: Long Reasoning Chains and Repetition Loops

The current generation of reasoning models generally suffers from “overthinking,” with community feedback concentrated on three points:

• Long reasoning chains: Even simple questions generate thousands of tokens of reasoning, much of it repetitive self-confirmation;
• Infinite loops: On brain-teaser-style questions, the model repeatedly restarts, sometimes continuing to verify a dozen more times even after reaching the correct answer;
• More pronounced in smaller models: Small-parameter models are more prone to repetitive generation after quantization.

Remedies:

1. Increase sampling parameters such as repetition penalty as officially recommended;
2. Disable thinking mode or cap the thinking-token budget for simple tasks;
3. Explicitly request “concise reasoning” in the system prompt;
4. Route high-frequency simple requests to non-reasoning modes or smaller models.

This also echoes the earlier evaluation point: when evaluating reasoning models, thinking-token cost must be included in the total ledger. A model that answers correctly but thinks for 5,000 tokens may be worse in throughput and cost than one that thinks for 500 tokens.

7. Summary: An Actionable Checklist

• Build a private evaluation set first, then choose models and quantization versions—avoid being hijacked by benchmark scores;
• Dense models use Q4 + Q8 KV Cache to preserve quality; small-activation MoE models use low-bit quantization to preserve speed;
• Prioritize upgrading to inference frameworks that support MTP for free acceleration;
• Use “local + cloud” hybrid routing to balance cost, capability, and compliance;
• Enable thinking-budget control for reasoning models and incorporate token cost into the evaluation system.

The moment for local LLMs is indeed arriving, but what truly determines the ceiling of the experience is the evaluation and tuning work after deployment.