Local LLM Performance Evaluation and Engineering Practices

1. Background: Consumer Hardware Is Changing the Inference Cost Curve

For the past two years, large language model inference has been closely associated with cloud GPU clusters. However, with the maturation of quantized inference, attention cache compression, and inference engine support for Mixture-of-Experts (MoE) architectures, running a usable LLM service on a single consumer GPU has moved beyond experimentation and into the realm of production validation.

This shift is not just about cost savings. It fundamentally changes the cost structure of inference services: when a one-time hardware investment can replace ongoing per-token billing, the applicable scope of local deployment expands from privacy-sensitive scenarios to high-frequency invocation scenarios and long-context batch processing scenarios.

This article focuses on the following core questions:

• Under a single-card 22GB VRAM constraint, how should quantization strategies be chosen for models of different parameter scales?
• How should the triangular constraint between quantization precision, KV cache precision, and context length be balanced?
• Is there a quantifiable inflection point between throughput (tokens/s) and output quality?
• What are the technical causes of inference redundancy (overthinking), and what suppression techniques are available?

2. Hardware and Quantization Baseline

2.1 VRAM Budget Breakdown

A single-card 22GB VRAM allocation must accommodate three components: model weights, KV cache, and inference engine runtime overhead. Taking a mid-sized MoE model as an example:

Component	Approximate Usage	Notes
Model weights (4-bit quantized)	Around 14–16GB	The larger the parameter count, the more dominant this remains even after compression
KV cache (8-bit quantized)	Grows linearly with context	Can reach several GB at 100K context
Runtime and activations	1–2GB	Related to batch size and concurrency

The key takeaway is that context length is the second-largest VRAM consumer. If you enable contexts beyond 100K, KV cache quantization precision directly determines whether you can maintain usable throughput on a single card.

2.2 Quantization Strategy Selection

Common quantization bit widths and their applicable scenarios:

• Q4 level: Minimal quality loss and suitable for primary models. This is currently the default choice for local deployment.
• Q3 level: Noticeable quality degradation, but the speed improvement is significant. Suitable for latency-sensitive daily tasks with higher precision tolerance.
• 8-bit KV cache quantization: Nearly mandatory for long-context scenarios. It significantly compresses cache usage while keeping attention quality degradation within an acceptable range.

One engineering detail worth emphasizing: weight precision and KV cache precision can be configured independently. This means you can run your primary model at higher precision while caching historical context at lower precision, preserving both output quality and context length within limited VRAM.

3. Throughput Measurement Dimensions and Influencing Factors

3.1 Speed Differences Across Parameter Scales

On the same GPU, model scale and throughput show a clear inverse relationship:

• Mid-sized models (around 27B level): Approximately 23–25 tokens/s with single concurrency, and around 40 tokens/s with dual concurrency. This speed is fully usable for interactive conversations.
• Lightweight MoE models (around 35B total parameters, fewer activated parameters): Can reach 90–100 tokens/s with single concurrency. The speed advantage is significant, making these models suitable for high-concurrency, short-response daily scenarios.

An important point to note is that MoE inference overhead depends on the number of activated parameters, not the total parameter count. This is why some models with larger total parameter counts actually run faster — each forward pass only activates a small subset of experts.

3.2 Differences Across Inference Backends

Utilization differences across inference backends on the same hardware should not be overlooked. Vulkan-based backends offer better cross-platform compatibility, while backends optimized for specific hardware typically lead in raw speed. In testing, the same model can achieve roughly 10%–20% throughput improvement on an optimized backend, with the difference being more pronounced under concurrent workloads.

Engineering recommendation: During the evaluation phase, do not only look at single-concurrency speed. Test total throughput under dual and even quadruple concurrency, because production services are typically concurrent.

3.3 How Context Length Erodes Speed

Long contexts slow down inference in two ways:

1. Prefill phase: The longer the context, the higher the first-token latency.
2. Decode phase: Each generated token requires attention computation over all historical KV entries. The longer the context, the higher the per-token latency.

Therefore, figures like “97 tokens/s with 256K context” are typically measured with short generation lengths. In real workloads, if you require both long context and high throughput, you need to accept some performance trade-offs and mitigate them through KV cache quantization.

4. Output Quality and Capability Boundaries

4.1 Usability Criteria

When evaluating whether a local model is usable, score it on three dimensions:

• Instruction following: Can it consistently output in the required format and follow constraints?
• Multi-turn consistency: Does it exhibit role drift or factual forgetting in long conversations?
• Task completion rate: Pass rate on specific tasks such as code generation, structured extraction, and summarization.

Testing shows that mid-sized models at 4-bit quantization achieve a production-ready level on all three dimensions, approaching the performance of some mid-sized cloud models. Lightweight MoE models excel in speed but are slightly weaker on complex reasoning tasks, making them better suited as the default choice for high-frequency lightweight tasks.

4.2 The Practical Value of Multimodal Capabilities

Image understanding support means local deployments can cover more scenarios, such as document screenshot parsing, UI element recognition, and mixed image-text Q&A. These capabilities remain basically usable after quantization, but note that vision encoders typically do not undergo quantization, so their VRAM usage must be budgeted separately.

4.3 Differences Across Fine-Tuned Versions

Community fine-tuned versions vary in instruction-following performance. Rules of thumb:

• Prefer quantized artifacts aligned with the base model version.
• Run separate instruction-following regression tests on fine-tuned versions; do not assume they inherit the base model’s performance.
• For scenarios involving safety boundaries, always conduct your own compliance evaluation.

5. Inference Redundancy: Why Models Overthink

5.1 Problem Description

A widespread issue is that models repeatedly self-verify after already reaching the correct answer, producing thousands of tokens of reasoning traces. This is particularly noticeable on brain-teaser type questions — the answer appears early, but the model keeps going in circles.

5.2 Likely Technical Causes

This phenomenon is likely related to the optimization objective during training: to achieve higher scores on evaluation sets requiring long-chain reasoning, the training process rewards longer reasoning chains. When this tendency is over-reinforced, the model applies long-chain patterns even to simple questions, leading to overthinking.

The smaller the model, the more pronounced this phenomenon becomes. Smaller models have weaker control over reasoning chain length and are more prone to falling into repetitive confirmation loops.

5.3 Suppression Techniques

Actionable tuning directions include:

# Pseudo-configuration example, illustrating parameter categories only
repetition_penalty: moderately increase to suppress repetitive generation
max_thinking_tokens: set an upper limit on the reasoning phase, force closure beyond it
stop_sequences: terminate early when an answer pattern is matched
temperature: slightly lower to reduce divergence

It is important to emphasize that these parameters need to be tuned separately by model scale and task type. There is no universal optimal configuration. The recommended approach is to build a small regression set for each task type and grid-search the key parameters.

6. Deployment Recommendations

6.1 Configuration Selection Decision Tree

22GB VRAM
├─ Quality-first
│   └─ Mid-sized model + Q4 weights + 8-bit KV cache + 100K context
├─ Speed-first
│   └─ Lightweight MoE + Q3 weights + 8-bit KV cache + 256K context
└─ Concurrency-first
    └─ Lightweight MoE + medium context + dual-concurrency scheduling

6.2 Pre-Launch Validation Checklist

• Single-concurrency and dual-concurrency throughput both meet targets
• First-token latency under long context is acceptable
• Instruction-following regression tests pass
• Inference redundancy has been suppressed to an acceptable range through parameter tuning
• Multimodal inputs are included in the VRAM budget
• Safety and compliance boundaries have been evaluated separately

6.3 Rethinking the Cost Perspective

Replacing usage-based billing with a one-time hardware investment is clearly attractive for high-frequency invocation scenarios. However, this does not mean cloud solutions will be completely replaced — the capability ceiling of very large models in complex reasoning, multimodal understanding, and long-document processing remains difficult for local deployments to match in the short term.

A more realistic assessment is that local deployment will absorb the “mid-tier capability, high-frequency invocation, data-sensitive” layer of demand, while cloud resources will concentrate on “top-tier capability, low-frequency invocation, elastic scaling.” For developers, this means architecture design should reserve the ability for hybrid local-cloud scheduling, with dynamic routing based on task complexity and cost budget.

7. Summary

Local LLM deployment has moved from “can it run” to “how to run it well.” The core engineering points can be summarized as follows:

1. Quantization is the key lever: Weight precision and KV cache precision are independently adjustable, making them critical for balancing quality, speed, and context length.
2. Architecture determines the speed ceiling: MoE architectures decouple total parameter count from inference overhead, making them an important direction for accelerating local deployment.
3. Parameter tuning cannot be skipped: Inference redundancy is a widespread problem in current models and must be addressed through targeted techniques such as reasoning phase limits and repetition penalties.

Worth exploring next are practical validation of even longer contexts, optimization of multi-concurrency scheduling strategies, and routing framework design for hybrid local-cloud inference.