How Quantization Affects Local Coding Model Performance: A Practical Evaluation
This article examines how quantization affects the real-world performance of local coding models. Drawing on an independent benchmark run on a dual-GPU desktop workstation, it compares three variants from the same model family—using 4-bit integer, FP8, and 4-bit float quantization—on verifiable coding tasks scored automatically by a machine. All three passed every test, showing that basic coding ability is converging while the real differences lie in *how* the work gets done: output length, planning style, tool use, and cache efficiency. The article reconstructs the task design, explains the causes behind the results, and distills the findings into a reusable evaluation methodology and a scenario-based selection framework, along with practical guidance on prompt caching and on bridging local evaluation with cloud deployment. The code samples and the accompanying weighting model make it easy for a team to run its own baseline and choose a configuration that fits its workload.
1. Background: From “Can It Run Locally” to “Can It Complete the Task Reliably”
Over the past two years, the maturing of desktop-class GPUs and quantized inference has made billion-parameter coding models a common sight on local workstations. But there is a wide gap between “it runs” and “it can reliably take over a real task for a developer”—and that gap can only be closed with a proper evaluation methodology.
Early benchmarks mostly measured basic capability: whether a model could answer questions on well-known leaderboards, or complete code on a fixed set of samples. The common weakness of these tests is that they are static, offer no external feedback, and stop after a single step. They say little about how a model performs in real development. Real development is a multi-turn loop: the model inspects the codebase, searches files, runs commands, executes tests, and fixes errors until the goal is met. The coding agent emerged as the mainstream form precisely in this context. It refers to a large-model application that can perceive a codebase and call external tools, typically built around “model + tool calls + loop control.”
Recently, a developer ran an independent benchmark on a dual-GPU desktop workstation: comparing three “Flash”-level variants from the same model family, using different quantization schemes, on local coding tasks. The evaluated versions are mid-sized models designed for low-latency inference, all running locally.
The value of this benchmark is not the ranking it produces, but the answer it gives to an easily overlooked question: when several models can all “pass the tests,” where exactly do the differences show up? This article first reconstructs the task design, then breaks down the causes behind the results, and finally abstracts them into a selection methodology that teams can apply directly.
2. Evaluation Design: Verifiable Tasks + Automatic Machine Scoring
A convincing local evaluation needs to answer at least four questions.
1. Are the tasks close to real work?
This benchmark used two task types:
- Defect fixing: Repair 17 real bugs in an actual project codebase, testing the ability to locate problems, understand context, and make minimal changes.
- Building a complete application: Build a web application from scratch and pass 19 checks, testing the end-to-end ability from design and coding to verification and delivery.
For the defect-fixing tasks, each bug comes with an independent reproduction script and is verified with minimal code to fail before the fix. This is what makes scoring accurately attributable to the bug being fixed, without interference from other factors.
2. Does it exercise real tool use?
The evaluation required the model to act as an agent and complete tasks by calling 21 native tools. These tools typically cover categories such as file reading and writing, command execution, code search, test running, and version-control operations. Crucially, the evaluation does not compare the quality of each platform’s tool implementations, because the underlying model is the same and the toolset is a controlled constant. What really matters is how the model plans, selects, and combines these tools.
3. Is the scoring objective?
Scoring was done entirely by a machine, with no LLM judge introduced. This guarantees the results are reproducible: the same code run against the same test suite yields the same score, which does not drift with a judge’s preferences.
4. Are the results attributable?
The evaluation recorded information such as the quantization scheme, elapsed time, output size, and number of interaction turns, and placed “completeness” and “how it ran” side by side in one table. This is the basis for the analysis that follows.
3. Three Quantization Schemes, Three Working Modes
Quantization means representing model weights at a lower bit width. Common options include FP8, 4-bit float, and various 4-bit integer formats. Each has its own characteristics:
- FP8 (8-bit floating point): Balances precision and efficiency. It preserves the dynamic range of floating point, so precision loss is usually small and hardware decode overhead is low—provided the inference engine and underlying hardware offer native support.
- 4-bit integer quantization: A lower bit width means the smallest memory footprint, helping larger models or longer contexts fit into limited VRAM. The cost is greater precision loss, which must be controlled by relying on efficient quantization formats and matching inference kernels.
- 4-bit float quantization: Balances dynamic range and precision at 4-bit width. Real-world quality depends heavily on how mature the kernel implementation is, and hardware support is the key prerequisite for adoption.
In real engineering decisions, additional techniques such as weight and activation quantization and KV-cache compression are often layered on top. For the sake of discussion, this article uses A, B, and C to represent the three categories above.
Core finding: all three models passed every test. Basic coding ability is becoming homogeneous; “can it get it done” is no longer the main differentiator. The real difference lies in “how it gets it done.”
| Dimension | A: 4-bit integer | B: FP8 | C: 4-bit float |
|---|---|---|---|
| Time to build the app | ~18.4 min (shortest) | Not reported | ~27.6 min (longest) |
| Total output tokens | ~20k (fewest) | Not reported | Not reported |
| Interaction turns | Not reported | 77 (fewest) | Not reported |
| Prompt cache hit rate | Not reported | Up to 98% | Not reported |
| New test cases | Not reported | Not reported | 18 (most) |
| Typical behavior | Direct to the goal, concise output | Plan first, then step-by-step execution | Deep reasoning each turn, repeated polishing |
| Best-fit scenario | Rapid prototyping and quick validation | Unattended background runs, long sessions | Standardized delivery, high test coverage |
3.1 Scheme A: Minimal Output, Faster Through Less
A built the application in the shortest time. The reason is not that its underlying inference engine is faster, but that it generated only about 20k tokens across the whole process. In an agent scenario, the output phase is generated token by token, so elapsed time is roughly proportional to output size. Even without an engine-speed advantage, less output is enough to put total time in the lead.
The flip side of conciseness is that there is less explanation and verification along the way. This is acceptable at the prototype stage, but before delivery it requires added human review and testing.
3.2 Scheme B: Plan First, Execute Later, With Cache Hits Lowering the Cost
B used the fewest interaction turns (77). It tends to lay out a plan first, then execute step by step. This clarity brings two benefits:
- Fewer wasted round-trips: There is markedly less exploring the codebase through trial and error.
- Higher prefix reuse: History context stays more stable and is more likely to hit the prompt cache. Its prompt cache hit rate reached 98%, meaning the vast majority of input tokens can be reused directly during prefill instead of being recomputed. Compute cost is therefore minimized.
This makes B especially well-suited to running unattended in the background for long stretches. It should be noted that cross-request prompt caching usually requires the inference service layer to enable the relevant capability; if the local engine does not enable it, this benefit cannot be realized.
3.3 Scheme C: Deep Reasoning, Quality First
C took the longest (about 27.6 minutes), because it performed a large amount of internal reasoning in every interaction turn. Reasoning and tool calls advanced in an interleaved manner, so token consumption was naturally higher.
This investment bought more standardized code and the most test cases (18). In an engineering context, the number of test cases often reflects how much a model considers edge cases: it does not just “make the feature work”—it proactively validates error paths.
3.4 Key Takeaway: Time Measures Efficiency, Not Quality
| Observation | Direct Cause | Engineering Implication |
|---|---|---|
| A has the shortest total time | Least output, not the fastest engine | Fast iteration should prioritize completion speed |
| B has the fewest turns | Plan first, less trial and error | A stable prefix favors cache hits |
| C has the most test cases | Deep reasoning each turn, proactive edge-case validation | Time and compute buy engineering quality |
These metrics do not live on the same axis: elapsed time measures efficiency, the number of tests measures quality, and turns and cache hit rate measure how resources are used. Conflating them easily leads to wrong conclusions. For example, if you judge purely by elapsed time, C would be undervalued—yet it may be exactly the right fit for teams with strict quality requirements.
4. Beyond Quantization: What Really Shapes Agent Performance
Quantization determines “how large a model can run, and how fast,” while the model’s own instruction-following, planning habits, and reasoning style determine “how the task gets completed.” A, B, and C behave noticeably differently, but they are different quantization versions of the same model family. A reasonable inference is that the reasoning style comes mainly from the model itself, while quantization mainly affects runtime efficiency and fine-grained behavior.
Several other factors are equally important when it comes to engineering adoption.
1. Coordinating context and cache management. Every extra interaction turn accumulates history, tool returns, and reasoning traces. Once the context exceeds the model window, it has to be trimmed or summarized, which may cause the model to “forget” its earlier plan. Cache hit rate, context length, and task stability are interrelated and should be evaluated together as one group of metrics.
2. The size of tool return content. Stuffing hundreds of lines of logs or file content back into context at once quickly inflates token consumption and dilutes key information. A common practice is to return a summary or a structured summary first, and let the model expand and inspect as needed.
3. Sampling and loop-control parameters. Parameters such as temperature, maximum output length, and the turn limit per task all significantly affect results and elapsed time, and should be fixed and recorded during evaluation.
5. A Reusable Evaluation Methodology
Abstracted out of this benchmark, the process boils down to three steps.
Step 1: Build a verifiable task set.
- Defect-fixing tasks: Each bug gets a minimal reproduction script that is guaranteed to fail before the fix, avoiding “fake fixes.”
- Build tasks: Provide an explicit feature list and translate each item into an automatically executable check, covering dimensions such as successful build, working core features, complete error handling, and the presence of necessary tests. Keep the check list stable; otherwise results across models and quantization schemes are not comparable.
Step 2: Record the complete run trace. At a minimum, collect the following metrics:
- End-to-end elapsed time, interaction turns, and output tokens;
- Input tokens and prompt cache hit rate;
- Tool-call success rate and number of failed retries;
- Pass rate on verifiable tasks and number of new tests added.
import timefrom dataclasses import dataclass, field@dataclassclass RunMetrics:label: str # Configuration label, e.g., A, B, Cwall_time_s: float = 0.0 # End-to-end elapsed time (seconds)turns: int = 0 # Number of turns interacting with the modelin_tokens: int = 0 # Cumulative input tokensout_tokens: int = 0 # Cumulative output tokens (reasoning + tool calls)cached_in_tokens: int = 0 # Input tokens that hit the prompt cachetool_calls: int = 0 # Number of tool callstool_failures: int = 0 # Number of failed tool callspassed: int = 0 # Number of tasks passedtotal: int = 0 # Total number of taskstests_added: int = 0 # Number of new test cases addednotes: list = field(default_factory=list)@propertydef pass_rate(self) -> float:return self.passed / self.total if self.total else 0.0@propertydef cache_hit_rate(self) -> float:return self.cached_in_tokens / self.in_tokens if self.in_tokens else 0.0@propertydef tool_success_rate(self) -> float:ok = self.tool_calls - self.tool_failuresreturn ok / self.tool_calls if self.tool_calls else 0.0def run_task(task, client, tools, verifier, cfg: dict) -> RunMetrics:"""Evaluation loop for a single task: run the task -> verify the result -> record metrics"""m = RunMetrics(label=cfg["label"])history = [{"role": "system", "content": cfg["system_prompt"]}]started = time.perf_counter()for _ in range(cfg["max_turns"]):history.append({"role": "user", "content": task.prompt(history)})resp = client.chat(messages=history, tools=tools,temperature=cfg["temperature"], max_tokens=cfg["max_tokens"],)m.turns += 1u = resp.usagem.in_tokens += u["prompt_tokens"]m.out_tokens += u["completion_tokens"]m.cached_in_tokens += u.get("prompt_cached_tokens", 0)if resp.tool_calls:for call in resp.tool_calls:m.tool_calls += 1try:result = tools.execute(call) # Toolset injected by the evaluation frameworkexcept Exception as exc:m.tool_failures += 1result = f"error: {exc}"history.append({"role": "tool", "content": str(result)})elif verifier.done(resp.content): # The model considers the task donebreakm.wall_time_s = time.perf_counter() - started# Automatically judged by the machine: defects fixed, checks passedok, tests_added = verifier.check()m.passed, m.total, m.tests_added = int(ok), 1, tests_addedreturn m
Step 3: Control variables and run multiple times. On the same hardware and with the same context length and sampling parameters, run each configuration at least 3 times, take the median, and observe the variance, so that a chance result is not mistaken for a conclusion. If conditions allow, run a second validation with a separate private task set that has not been published, to reduce the risk of the evaluation set being covered by the model’s training data.
6. From Metrics to Decisions: A Selection Framework
The direction given by the original benchmark can be further structured into a decision framework: check the hard threshold first, then weigh by scenario.
Hard threshold. Any configuration whose pass rate on the target task set falls below the bar is eliminated outright. In this case, all three models passed every test, so they move on to the next layer of comparison.
Scenario weights. Different scenarios are sensitive to different metrics. The following table is a reference for assigning weights:
| Scenario | Speed | Cost | Engineering Quality | Typical Focus Metrics |
|---|---|---|---|---|
| Everyday development assistance | Medium | Medium | Medium | Balanced overall performance |
| Unattended background tasks | Low | High | Medium-high | Cache hit rate, interaction turns, plan stability |
| Rapid prototyping | High | Low | Low | End-to-end elapsed time |
| Standardized delivery | Low | Medium | High | Code conventions, number of new tests |
For each configuration that clears the threshold, compute a composite score under the same weights and rank them:
def rank(configs: list[RunMetrics], weights: dict) -> list[tuple[str, float]]:"""First filter by the pass-rate gate, then sort by scenario weights"""MIN_PASS_RATE = 1.0 # Adjustable based on task riskTIME_REF = 1800.0 # Time normalization baseline: 30 minutes, use the P50 time of the target task setCANDIDATES = [c for c in configs if c.pass_rate >= MIN_PASS_RATE]if not CANDIDATES:return []best_time = min(c.wall_time_s for c in CANDIDATES)def score(c: RunMetrics) -> float:speed = best_time / c.wall_time_s # 0~1, fastest is 1economy = c.cache_hit_rate # 0~1, cache hit ratequality = min(c.tests_added / 10.0, 1.0) # Normalized to 0~1, cap at 10# Tool calls with failures lose one quality tier (3%)quality *= 0.97 if c.tool_success_rate < 0.97 else 1.0return (weights["speed"] * speed+ weights["cost"] * economy+ weights["quality"] * quality)return sorted(((c.label, round(score(c), 3)) for c in CANDIDATES),key=lambda x: x[1], reverse=True)# Scenario example: daily development, equal weights for all threeDAILY = {"speed": 1.0, "cost": 1.0, "quality": 1.0}# Scenario example: standardized delivery, quality weight equals speed + costSTRICT = {"speed": 0.5, "cost": 0.5, "quality": 2.0}
This yields scenario-based recommendations:
- Everyday development tasks: Choose a configuration with balanced overall performance. In the original benchmark, the FP8 scheme (B) balanced efficiency and compute cost and is a typical example.
- Unattended background tasks: Prefer configurations with a clear plan, few turns, a high cache hit rate, and low marginal cost.
- Rapid prototyping and validation: Prefer configurations with concise output and the shortest end-to-end time, and top up testing and review before delivery.
- Standardized delivery: Prefer configurations that proactively write tests and produce more standardized code, trading time for a lower rework rate.
Two additional points. First, for code tasks with high security requirements, no local coding agent should replace human review; it should be paired with processes such as static analysis, dependency auditing, and secret scanning. Second, the weights and thresholds in the scoring framework are only a starting point and should be continuously calibrated against the target task set. Taking the three configurations above as an example, the results computed with the “daily development” weights are:
- B: speed ~0.63, cost 0.98, quality ~0.81, composite score ~0.81;
- C: speed ~0.44, cost 0.50, quality 1.00, composite score ~0.65;
- A: speed 1.00, cost 0.50, quality 0.30, composite score ~0.60.
This ranking matches the empirical judgment that “FP8 is recommended for daily development.” If you switch to the “standardized delivery” weights, the ranking becomes B, C, A, with B still in the lead. It is worth stressing that this is only a comparison of three configurations on a single evaluation set, and the conclusion must not be generalized into “a certain class of quantization scheme is universally better.”
7. Bridging Local Evaluation and Cloud Deployment
Local evaluation answers “which configuration to choose,” while cloud deployment answers “how to run at scale, stably and economically.” The two need a consistent measurement standard to close the loop.
In the cloud (for example, Baidu AI Cloud), building a coding agent service can typically combine the following general capabilities:
- Use a container platform to host the inference service and the agent runtime;
- Use object storage to archive evaluation sets, model files, and build artifacts;
- Use a message queue to distribute long-running tasks asynchronously, matching unattended background scenarios;
- Use a logging service to accumulate per-turn interaction records, making it easy to review failed cases;
- Use monitoring and alerting to continuously track metrics such as elapsed time, cache hit rate, and task success rate.
The baseline established by local evaluation should be written into cloud monitoring, and a regression evaluation should be run after every engine or model version update, to avoid behavior drift caused by upgrades.
Is prompt caching actually taking effect? If the local engine has not enabled the relevant capability, cross-request caching often fails to hit. Use the following method for a quick self-check:
- Send two requests to the inference service;
- Keep the
systemprompt identical across both, at about 800 tokens, with a different user question each time; - Compare the returned input-token statistics: the second request’s cached-token count should be close to 800.
SYSTEM = "You are a coding assistant." + "Here are the project guidelines. " * 200 # ~800 stable tokensdef probe_cache(client):a = client.chat([{"role": "system", "content": SYSTEM},{"role": "user", "content": "Explain in one sentence what a unit test is."}])b = client.chat([{"role": "system", "content": SYSTEM},{"role": "user", "content": "Explain in one sentence what a code review is."}])# The cached-token field name varies by inference service; read it per the actual APIfirst, second = a.usage, b.usageprint("First input tokens:", first["prompt_tokens"])print("Second input tokens:", second["prompt_tokens"],"Cached tokens:", second.get("prompt_cached_tokens", 0))
If the two prompt_tokens values are nearly unchanged and the cached count is close to 800, caching is in effect; if the cached count is always 0, check whether the engine has the relevant feature enabled.
To actually capture the caching benefit, how you organize the prompt matters just as much: put stable content first and changing content last.
# Recommended layout: stable prefix first, changing content lastmessages = [# Part 1: team coding conventions (nearly constant, should go first){"role": "system", "content": project_coding_rules},# Part 2: tool definitions (fixed order and content, no dynamic add/remove){"role": "system", "content": tool_definitions},# Part 3: task description (kept consistent within the same batch of tasks){"role": "user", "content": task_brief},# Part 4: current context and latest instructions (changes every turn, placed last){"role": "user", "content": f"Current file snippet:\n{current_snippet}\nLatest request: {latest_request}"},]
A common mistake is to place volatile fields such as timestamps or random task IDs at the start of the system prompt. Change even a single character, and everything after it becomes a “cache miss” and must be recomputed.
8. Conclusion
This benchmark on a dual-GPU workstation offers three takeaways:
- Completeness is the baseline; how it runs is the difference. The basic capability of local coding agents is becoming homogeneous. Quantization scheme, planning habits, and reasoning depth together determine how well-suited they are to different scenarios.
- Evaluation must be verifiable, reproducible, and attributable. Automatically judged defect fixing and check-based building, fixed controlled variables, and running multiple times to take the median are the prerequisites for a trustworthy conclusion.
- Selection should be based on scenario-based weighted evaluation, not a single ranking. No single configuration dominates across all scenarios. Establishing a baseline with local evaluation and continuously validating it with a cloud observability system is the sustainable engineering path.
For teams preparing to deploy a coding agent locally, the recommendation is to first define 10–20 verifiable tasks that fit your own business, then run a baseline evaluation using the metric system in this article. A configuration chosen on this basis is often more reliable than any ready-made leaderboard.
