Open-Source LLM Local Deployment: A Complete Guide to Performance Benchmarking and Tuning
Local deployment of open-source large language models (LLMs) has entered a "single-GPU" era, but judging whether a model is usable, fast, stable, and worth it takes far more than a tokens-per-second number. This guide lays out an engineering methodology for evaluating and tuning an open-source LLM on one GPU for a specific workload. It covers VRAM budgeting (weight footprint and the often-overlooked KV Cache), a four-dimension evaluation framework, performance benchmarking that separates prefill from decode, architecture and quantization tradeoffs (dense vs. MoE, 4-bit vs. 8-bit), the real limits of long context, inference tuning for "thinking" models, building a self-made Agent eval set instead of relying on public leaderboards, reproducible testing, and the hidden costs of local inference versus pay-per-token cloud—including a hybrid deployment pattern.读完本文,你将获得一套可落地的方法论,用来判断一张显卡、一个开源模型和一个具体业务场景到底能不能用、快不快、稳不稳、值不值。
Introduction: Local LLMs Enter the “Single-GPU” Era
A new wave of open-source LLMs has arrived, and the Mixture-of-Experts (MoE) architecture is becoming mainstream: total parameter counts reach the hundreds of billions, yet only a few billion parameters are actually activated per token. This has produced a striking phenomenon: an affordable second-hand consumer GPU with a modified 22GB of VRAM can, after quantization, generate text at more than twenty tokens per second, handle contexts of a hundred thousand or even hundreds of thousands of tokens, and understand images at the same time. Many developers take this as a sign that “the era of local LLM deployment has arrived.”
Four community trends stand out:
- A much lower performance barrier. A single developer building a local inference machine on a limited budget is no longer news.
- Capabilities keep moving up. Mid-sized, small-parameter models now approach the level that only top-tier cloud models could reach on some agentic coding benchmarks.
- Business models are being disrupted. When near-“full-power” capability can be bought once as hardware and used for years, the appeal of “pay-per-token” declines. This also offers another lens on why leading model vendors and cloud providers have reorganized.
- Long context is becoming standard, but “supports 1M context” does not equal “remains reliable at 1M context.”
This article does not argue which model is best, nor does it claim “local is invincible.” It answers a more engineering-focused question: given an open-source LLM, one GPU, and a concrete workload, how do you judge whether it is usable, fast, stable, and worth it? The following seven parts—VRAM budgeting, performance breakdown, architecture and quantization tradeoffs, inference tuning, Agent evaluation, reproducible testing, and deployment architecture—lay out a practical methodology.
Building an Evaluation Framework: Four Dimensions, One Table
The most common mistake when evaluating local deployment is fixating only on “how many tokens per second.” Speed is just a surface metric. What actually determines usability is the balance across four dimensions: whether resources fit, whether latency is acceptable, whether capability matches the task, and whether it stays stable over long runs. If any one dimension becomes a bottleneck, real-world experience degrades noticeably.
| Dimension | Key metrics | Common sources of distortion | Mitigation |
|---|---|---|---|
| Resources | Peak VRAM, system RAM, disk usage | Counting only weights, ignoring KV Cache and runtime overhead; insufficient PSU wattage | Measure peak under max context and target concurrency; leave headroom |
| Efficiency | Time to first token (TTFT), time per output token (TPOT), decode speed, concurrent throughput | Testing only the “top speed” under ideal conditions | Stress-test with realistic input/output length combinations |
| Capability | Instruction following, structured output, long-context retrieval, coding and Agent ability | Relying only on general leaderboards disconnected from your tasks | Build a 30–50 sample eval set from your own workload |
| Stability & cost | Repetition, dead loops, crash rate over long runs, cost per token | Short tests can’t expose tail-end problems | Run long-duration and concurrent stress tests; account cost by whole-machine power draw |
VRAM Budgeting: First “Will It Fit,” Then “Will It Run Fast”
Weight footprint: parameters × bytes per parameter
There is a simple estimate for weight VRAM:
Weight VRAM ≈ parameters (in billions) × bytes per parameter
Bytes per parameter at different precisions:
| Weight precision | Bytes per parameter | Weight footprint for a ~30B model |
|---|---|---|
| FP16/BF16 | 2 | ~60GB |
| 8-bit integer quantization | 1 | ~30GB |
| 4-bit integer quantization | 0.5 | ~15GB |
Take the 27B-class model popular in the community as an example: after 4-bit quantization, the weights need about 13–14GB of VRAM, and adding runtime overhead, it fits exactly onto a 22GB GPU. This is precisely why “modified large-VRAM consumer GPU + low/medium-bit quantization” has become a popular local deployment combination.
KV Cache: the often-ignored “hidden heavyweight”
During decoding, every generated token requires the model to read the Key and Value vectors of all previous tokens. This cache stays resident in VRAM and can be estimated with a formula:
Single-sequence KV Cache ≈ 2 × layers × KV heads × head dimension × cache bytes × token count
(The factor of 2 accounts for Key and Value.)
Take a typical configuration as an example (assumed values, used only to illustrate the algorithm):
| Setting | Value | Single-sequence KV Cache |
|---|---|---|
| 48 layers, 8 KV heads, head dim 128, FP16 cache, 32K context | ~192KB per token | ~6GB |
| Same, but 256K context | Per-token cost unchanged, length 8× | ~48GB |
| Same, 8-bit cache, 32K context | ~96KB per token | ~3GB |
These numbers reveal two key conclusions:
- Under long contexts, the KV Cache can be “heavier” than the weights themselves. For the 256K-context, FP16-cache combination above, the KV Cache alone needs about 48GB of VRAM—more than three times the 4-bit weights. This explains why “supporting very long context” and “actually running very long context on a single GPU” are often separated by a multiple of VRAM.
- 8-bit KV Cache is key to running long contexts on a single GPU. With typically limited precision loss, the cache size is cut in half.
You also need to reserve three kinds of headroom: the inference engine’s own overhead, peak usage when the context nears its limit, and cache stacking from multiple concurrent requests. In practice, reserve 10%–20% of total VRAM as a buffer to avoid frequent out-of-memory errors when processing long documents or high concurrency.
Performance Benchmarking: Prefill and Decode Must Be Measured Separately
Two phases, two bottlenecks
LLM inference divides cleanly into two phases:
- Prefill: processes all input tokens in parallel; it is compute-bound and determines time to first token. The longer the input, the longer the wait.
- Decode: generates output tokens one by one; the volume of weights read far exceeds the actual computation, so it is memory-bandwidth bound.
Because of this, decode speed has an approximate upper bound:
Decode speed (tokens/s) ≈ effective memory bandwidth ÷ bytes read per token
With 1TB/s bandwidth and 4-bit weights: a 30B dense model reads about 15GB of weights per token, giving a theoretical ceiling of about 66 tokens/s. Accounting for scheduling, sampling, and output-layer overhead in the inference engine, measured results typically reach only 50%–70% of the theoretical value. A 16GB entry-level GPU usually has less than 0.5TB/s of bandwidth and is often constrained mainly by VRAM capacity during inference; a GPU with more than 16GB of VRAM is what makes deployment practical.
All four metric classes matter
| Metric | Meaning | Where it affects experience |
|---|---|---|
| TTFT (time to first token) | Time from sending the request to receiving the first token | First impression in chat; determined mainly by input length and prefill speed |
| TPOT (time per output token) | Average time per token in the generation phase | Smoothness of the typewriter effect; related to model size and memory bandwidth |
| Decode speed | Tokens generated per unit time (tokens/s) | Reading experience; determines whether it “keeps up with reading” |
| Concurrent throughput | Overall output speed when serving multiple requests at once | Real productivity when multiple users share one machine or for batch tasks |
A common trap: the “top speed” measured under single-request ideal conditions often does not represent real experience. Once you add system prompts, code context, conversation history, and image inputs, the input easily exceeds tens of thousands of tokens and TTFT can multiply.
A reproducible stress-test method
To make results comparable, test conditions must be fixed and fully documented:
- Fix sampling parameters: temperature, top-p, repetition penalty, etc. The decode phase is stochastic, so directly comparing output quality across different configurations is unreliable.
- Matrix the input/output combinations: e.g., input lengths of 256 / 4K / 32K, output lengths of 512 / 2K.
- Warm up before measuring: the first load has cold-start overhead such as file caching and kernel compilation; discard the first few runs.
- Take the median of multiple runs: reduce the impact of occasional jitter on conclusions.
- Concurrency gradient testing: increase from 1 to 2, 4, 8 concurrent requests, recording the throughput/latency curve to find the best concurrency point for the machine.
# Pseudo-code: local inference stress test (with_latency wraps a streaming interface)def benchmark(model, prompt, max_tokens, repeats=3):ttft_list, tpot_list = [], []for _ in range(repeats):with model.with_latency(prompt) as lat:for token in model.stream_generate(prompt, max_tokens=max_tokens):pass # consume token by token to simulate a real streaming scenariolat.record() # record the end time on the final callbackttft_list.append(lat.ttft_ms)# TPOT = (total time - TTFT) / number of output tokenstpot_list.append((lat.total_ms - lat.ttft_ms) / max(lat.output_tokens, 1))return {"ttft_ms": median(ttft_list),"tpot_ms": median(tpot_list),"tokens_per_s": 1000 / median(tpot_list),}
Architecture, Quantization, and Context: Understanding the Numbers
Dense vs. Mixture-of-Experts: speed depends on “activated parameters”
A dense model activates all parameters for every generated token, so speed is roughly inversely proportional to “parameters × bytes per parameter.” For a 30B dense model, by the bandwidth estimate above, the theoretical ceiling is about 66 tokens/s.
An MoE model consists of multiple “expert” sub-networks, and each token activates only a few of them. The “35B-class models” that the community measures at tens or even over a hundred tokens per second are almost all MoE. With the same 30B total parameters, an MoE model’s activated parameters are often only around 3B, and the bytes read per token drop to about one-tenth, putting the theoretical speed ceiling near hundreds of tokens per second.
But MoE is not free:
- The full model is larger, harder to load entirely into VRAM, and tolerates less precision loss during quantization.
- Expert routing adds scheduling overhead; with small models or imbalanced routing, the speed advantage is noticeably eroded.
- In “high latency tolerance, high concurrent throughput” offline batch scenarios, dense models are often more predictable.
Weight quantization: the cost and benefit of 4-bit
| Scheme | VRAM usage | Quality impact | Suitable scenario |
|---|---|---|---|
| 4-bit weights | ~1/4 of FP16 | Complex reasoning and instruction following may degrade | Running large models on a single GPU; must be strictly evaluated before use as the main scheme |
| 8-bit weights | ~1/2 of FP16 | Usually very little loss | Plenty of VRAM and high quality requirements |
| 4-bit weights + 8-bit KV Cache | Weights halved, cache halved | Limited impact on short contexts | Mainstream compromise for single-GPU long context |
Community experience: quantization has limited impact on the subjective feel of generation fluency, but it can quietly weaken instruction following, multi-step reasoning, and structured output. That is why feedback like “the 4-bit version has instruction-following problems” is not uncommon. When evaluating a quantized version, don’t just judge whether it “reads well”—use objective metrics such as constrained decoding and format parsing to check it.
Context length: support ≠ usable well
The value of very long context is beyond doubt: you can put an entire codebase, a long document, or many rounds of conversation history in at once. But note:
- Compute cost grows with length. Attention cost scales quadratically with sequence length; increasing input from 32K to 256K can multiply TTFT several times over.
- The “lost in the middle” phenomenon is common. Key information placed in the middle of a very long context is more likely to be ignored; placed at the beginning or end, it is more easily retrieved.
- The effective window is usually far smaller than the advertised one. What determines the usable ceiling is often the ability to “precisely locate key information in a long document,” not the max length written in the specs.
A pragmatic approach is a “layered context strategy”: first use retrieval-augmented generation (RAG) to recall the most relevant fragments, summarize oversized codebases by module and expand them on demand, and keep what genuinely needs to be loaded at once within tens of thousands of tokens.
Inference Tuning: Taming the “Verbose Thinker”
The latest models generally adopt a “think first, then answer” reasoning mode. Community practice shows clear gains on coding and Agent tasks, but also three typical problems:
- Overly long chains of thought. For brain teasers like “washing a car” or “fitting a stick through a door,” some models repeatedly self-confirm a dozen-plus times after already giving the correct answer. A simple question becomes a long inner monologue, slowing responses and inflating token consumption.
- Repetition and dead loops. The smaller the model, the more pronounced the problem: repeated phrases and output stuck in loops. Models below 9B are especially affected.
- Diminishing returns. In many cases, answering directly is both faster and correct. Some argue that overly long chains of thought are partly a training result of “optimizing for benchmark scores.”
You can tune at three levels:
Sampling-parameter level:
- Set the repetition penalty using the model publisher’s recommended values; the smaller the model, the more the penalty strength should be raised.
- For structured outputs like JSON or Markdown tables, lower the temperature and tighten top-p to significantly reduce format drift.
# Example local inference service config (values are illustrative; defer to the model's official recommendations)model:quantization: w4a16 # 4-bit weights, FP16 activationskv_cache:quantization: q8_0 # 8-bit KV Cache; significantly cuts long-context VRAMcontext:max_tokens: 131072 # set by VRAM budget, not the model's advertised ceilingreserved_ratio: 0.15 # reserve 15% of VRAM as a buffersampling:temperature: 0.7top_p: 0.9repeat_penalty: 1.05 # small 8B-class models can be raised to 1.1-1.2presence_penalty: 0.2 # suppress repeated high-frequency wordsmax_tokens: 2048reasoning:default_enabled: truemax_thinking_tokens: 1024 # cap the thinking budget to prevent endless self-confirmationdisable_for_simple_task: true
Prompt and system level:
- Adding “give the conclusion directly, and briefly explain your reasoning only if needed” to the general system prompt is often more effective than tuning parameters alone.
- Configure a lightweight “thinking off” channel for simple tasks; switch to thinking mode only for complex reasoning and code generation.
- Use different stop sequences for the thinking and answer phases to avoid reasoning content spilling into the formal response.
Service-orchestration level:
- Enable prefix caching (Prefix Cache) to reuse the KV Cache of system prompts and repeated documents, significantly reducing TTFT under long prompts.
- Moderately lower the context ceiling in high-concurrency scenarios to avoid KV Cache bloat causing out-of-memory or service degradation.
- Set a thinking timeout and a “detect at most N consecutive identical outputs” mechanism; once triggered, force generation to end.
Agent Capability Evaluation: Build Your Own Eval Set Instead of Using Leaderboards
Over the past year, “having an LLM build a small game” has become a new paradigm for testing coding ability, and differences between models—even the same model under different configurations—are large. For example, implementing a small game in a firmware environment where the graphics interface is not yet ready:
- Some models take about two and a half hours, compile successfully on the first try, and run successfully on the first launch.
- Some models struggle for hours, still cannot run properly, and ultimately fail.
- Some models stumble over several pitfalls midway but eventually deliver a good result.
These cases make at least three points:
- Whether a model can complete tasks in “constrained, hard-to-debug” environments reflects real engineering ability better than its score on conventional coding problems.
- Running the same task multiple times shows clear success-rate fluctuation, so a single experience does not represent overall capability.
- For firmware-style work, “success on the first run” is worth far more than “works after many rounds of fixes.”
Capability tiers: first decide what to test
| Tier | Typical task | Metrics of interest |
|---|---|---|
| L1 code generation | Write functions or scripts from requirements | Compile pass rate, unit test pass rate |
| L2 multi-round modification | Locate and fix bugs, refactor existing code | Fix accuracy, regression-introduction rate |
| L3 repo-level understanding | Implement a new feature across multiple files | Task completion rate, number of interaction rounds needed |
| L4 autonomous Agent execution | Read logs, change configs, run tests, inspect results, iterate | End-to-end success rate, number of steps, tool-call accuracy |
The difficulty from L1 to L4 does not grow linearly. When the task shifts from “write some code” to “autonomously close the loop across dozens of tool calls,” the model’s context management, tool-call format compliance, and error recovery are all tested simultaneously.
Build your own eval set: 30–50 samples is enough to start
High scores on general leaderboards don’t necessarily transfer to your own business. A more reliable approach is to extract 30–50 samples from real tasks and build a small, sharp eval set:
- Difficulty distribution: the ratio of easy, medium, and hard samples, mirroring the real-world business distribution.
- Scoring method: prefer automatically-judgeable tasks (compilation, tests, format validation); use blind review or model cross-review for subjective quality.
- Repeatability: fix the random seed, sampling parameters, and runtime environment for each evaluation.
- Cover failure modes: deliberately include error-prone inputs such as very long contexts, mixed-language text, and contradictory requirements.
For Agent tasks, also add process metrics:
- Tool-call accuracy and format-compliance rate
- Average number of steps needed to complete a task
- Proportion of repeated, ineffective operations
- Context-growth rate
Reproducible Testing: From Record Table to Conclusions
The value of an evaluation depends on whether it can be reproduced. Use a standard table to record each test:
| Field | Description |
|---|---|
| Model & version | Name, parameter scale, architecture type (dense/MoE) |
| Quantization scheme | Weight precision, KV Cache precision, quantization framework |
| Hardware environment | GPU model, VRAM, RAM, whole-machine power draw |
| Inference engine | Engine name and version, run mode (graphics API or general compute API) |
| Context settings | Ceiling, actual input length, concurrency |
| Performance metrics | TTFT, TPOT, decode speed, peak VRAM |
| Quality metrics | Eval-set score, structured-output compliance rate |
| Stability | Repetition rate, dead-loop count, long-run behavior |
Test case TC-001: Long-document QAInput: 32K-token technical document + 5 cross-section questionsExpected output: JSON array, each item containing {question, answer, evidence}Scoring rules:1. JSON parse success rate (weight 40%)2. Answer agreement with human labels (weight 40%)3. evidence citation-position accuracy (weight 20%)
With this table, different configurations can be compared side by side. For example, you can clearly see how large the quality gap is between 4-bit and 8-bit weights, how much speed improves after enabling 8-bit KV Cache, and the marginal change in TTFT when context grows from 32K to 128K.
Cost and Architecture: The “Hidden Bill” of Local Inference and Hybrid Deployment
Don’t look only at GPU price
The claim that “a budget consumer card runs a full-power model” easily makes people overlook three hidden bills:
- Whole-machine cost: CPU, RAM (tens of GB of system RAM may also be needed for long contexts), PSU, cooling, and case.
- Electricity: at high load, whole-machine power draw can reach several hundred watts. Running 8 hours a day at typical electricity rates, the annual bill can still add up to a meaningful amount.
- Time and maintenance cost: environment setup, version upgrades, quantization selection, and troubleshooting are all ongoing investments.
So a more reasonable calculation compares “long-term total cost” with “pay-per-token cost”:
Local total cost ≈ hardware amortization + electricity + operations time cost
Cloud token cost ≈ request volume × average tokens × unit price
Local deployment is more advantageous when business call volume is stable and large, data is sensitive, or low latency / offline availability is required; cloud service is more economical when call volume fluctuates, models need rapid iteration, or the task itself is very long and complex.
Hybrid deployment: local handles “high-frequency simple,” cloud takes “low-frequency complex”
Local and cloud are not either/or. A more pragmatic architecture is layered routing:
- Local layer: handles sensitive data, offline scenarios, and high-frequency simple tasks (e.g., summarization, classification, format conversion, everyday chat), served by a small-parameter local model.
- Cloud layer: handles “long-tail complex tasks” such as complex reasoning, long-document analysis, and high-concurrency batch processing. When the local model’s confidence is low or task complexity exceeds limits, it automatically escalates to a cloud model.
- Coordination layer: a unified inference gateway handles routing, rate limiting, caching, and observability, keeping both layers transparent to the business.
For cloud capabilities, the Baidu AI Cloud Qianfan model platform provides API access to a range of mainstream LLMs, along with model fine-tuning and Agent orchestration; cloud products such as object storage, containers, log service, and monitoring/alerting can store eval sets, deploy local inference instances, and observe service quality. That said, “which cloud to call and which model to use” should still be decided by evaluation results and business needs, not by a single experience.
Conclusion
The biggest change brought by this new wave of open-source models is not that some model scored a few points higher on some leaderboard, but that the supply of LLM capability is shifting from “renting by the token” to “buying outright + long-term local use.” For developers, this means a new capability boundary: a budget machine can, offline and without pay-per-call, host a model assistant whose capability approaches the previous “full-power” version.
But expanding the boundary also brings new challenges:
- Budget VRAM carefully; don’t forget the KV Cache.
- Break speed measurement apart; don’t mix prefill with decode.
- Evaluate quantization; don’t judge 4-bit by subjective fluency alone.
- Budget the thinking; don’t let long chains of thought spin idle.
- Build an eval set for Agents; a single success can’t replace stable reliability.
Get these engineering details right, and local LLMs can truly move from “it runs” to “it’s dependable,” rather than staying at a screenshot on social media.
If you’ve already deployed an open-source model locally, share your hardware configuration, quantization scheme, and measured data in the comments, and bring up any confusion you hit while tuning—let’s make this evaluation guide more complete together.
