How Quantization Affects Local Coding Model Performance: A Practical Evaluation

    This article examines how quantization affects the real-world performance of local coding models. Drawing on an independent benchmark run on a dual-GPU desktop workstation, it compares three variants from the same model family—using 4-bit integer, FP8, and 4-bit float quantization—on verifiable coding tasks scored automatically by a machine. All three passed every test, showing that basic coding ability is converging while the real differences lie in *how* the work gets done: output length, planning style, tool use, and cache efficiency. The article reconstructs the task design, explains the causes behind the results, and distills the findings into a reusable evaluation methodology and a scenario-based selection framework, along with practical guidance on prompt caching and on bridging local evaluation with cloud deployment. The code samples and the accompanying weighting model make it easy for a team to run its own baseline and choose a configuration that fits its workload.

    1. Background: From “Can It Run Locally” to “Can It Complete the Task Reliably”

    Over the past two years, the maturing of desktop-class GPUs and quantized inference has made billion-parameter coding models a common sight on local workstations. But there is a wide gap between “it runs” and “it can reliably take over a real task for a developer”—and that gap can only be closed with a proper evaluation methodology.

    Early benchmarks mostly measured basic capability: whether a model could answer questions on well-known leaderboards, or complete code on a fixed set of samples. The common weakness of these tests is that they are static, offer no external feedback, and stop after a single step. They say little about how a model performs in real development. Real development is a multi-turn loop: the model inspects the codebase, searches files, runs commands, executes tests, and fixes errors until the goal is met. The coding agent emerged as the mainstream form precisely in this context. It refers to a large-model application that can perceive a codebase and call external tools, typically built around “model + tool calls + loop control.”

    Recently, a developer ran an independent benchmark on a dual-GPU desktop workstation: comparing three “Flash”-level variants from the same model family, using different quantization schemes, on local coding tasks. The evaluated versions are mid-sized models designed for low-latency inference, all running locally.

    The value of this benchmark is not the ranking it produces, but the answer it gives to an easily overlooked question: when several models can all “pass the tests,” where exactly do the differences show up? This article first reconstructs the task design, then breaks down the causes behind the results, and finally abstracts them into a selection methodology that teams can apply directly.

    2. Evaluation Design: Verifiable Tasks + Automatic Machine Scoring

    A convincing local evaluation needs to answer at least four questions.

    1. Are the tasks close to real work?

    This benchmark used two task types:

    • Defect fixing: Repair 17 real bugs in an actual project codebase, testing the ability to locate problems, understand context, and make minimal changes.
    • Building a complete application: Build a web application from scratch and pass 19 checks, testing the end-to-end ability from design and coding to verification and delivery.

    For the defect-fixing tasks, each bug comes with an independent reproduction script and is verified with minimal code to fail before the fix. This is what makes scoring accurately attributable to the bug being fixed, without interference from other factors.

    2. Does it exercise real tool use?

    The evaluation required the model to act as an agent and complete tasks by calling 21 native tools. These tools typically cover categories such as file reading and writing, command execution, code search, test running, and version-control operations. Crucially, the evaluation does not compare the quality of each platform’s tool implementations, because the underlying model is the same and the toolset is a controlled constant. What really matters is how the model plans, selects, and combines these tools.

    3. Is the scoring objective?

    Scoring was done entirely by a machine, with no LLM judge introduced. This guarantees the results are reproducible: the same code run against the same test suite yields the same score, which does not drift with a judge’s preferences.

    4. Are the results attributable?

    The evaluation recorded information such as the quantization scheme, elapsed time, output size, and number of interaction turns, and placed “completeness” and “how it ran” side by side in one table. This is the basis for the analysis that follows.

    3. Three Quantization Schemes, Three Working Modes

    Quantization means representing model weights at a lower bit width. Common options include FP8, 4-bit float, and various 4-bit integer formats. Each has its own characteristics:

    • FP8 (8-bit floating point): Balances precision and efficiency. It preserves the dynamic range of floating point, so precision loss is usually small and hardware decode overhead is low—provided the inference engine and underlying hardware offer native support.
    • 4-bit integer quantization: A lower bit width means the smallest memory footprint, helping larger models or longer contexts fit into limited VRAM. The cost is greater precision loss, which must be controlled by relying on efficient quantization formats and matching inference kernels.
    • 4-bit float quantization: Balances dynamic range and precision at 4-bit width. Real-world quality depends heavily on how mature the kernel implementation is, and hardware support is the key prerequisite for adoption.

    In real engineering decisions, additional techniques such as weight and activation quantization and KV-cache compression are often layered on top. For the sake of discussion, this article uses A, B, and C to represent the three categories above.

    Core finding: all three models passed every test. Basic coding ability is becoming homogeneous; “can it get it done” is no longer the main differentiator. The real difference lies in “how it gets it done.”

    Dimension A: 4-bit integer B: FP8 C: 4-bit float
    Time to build the app ~18.4 min (shortest) Not reported ~27.6 min (longest)
    Total output tokens ~20k (fewest) Not reported Not reported
    Interaction turns Not reported 77 (fewest) Not reported
    Prompt cache hit rate Not reported Up to 98% Not reported
    New test cases Not reported Not reported 18 (most)
    Typical behavior Direct to the goal, concise output Plan first, then step-by-step execution Deep reasoning each turn, repeated polishing
    Best-fit scenario Rapid prototyping and quick validation Unattended background runs, long sessions Standardized delivery, high test coverage

    3.1 Scheme A: Minimal Output, Faster Through Less

    A built the application in the shortest time. The reason is not that its underlying inference engine is faster, but that it generated only about 20k tokens across the whole process. In an agent scenario, the output phase is generated token by token, so elapsed time is roughly proportional to output size. Even without an engine-speed advantage, less output is enough to put total time in the lead.

    The flip side of conciseness is that there is less explanation and verification along the way. This is acceptable at the prototype stage, but before delivery it requires added human review and testing.

    3.2 Scheme B: Plan First, Execute Later, With Cache Hits Lowering the Cost

    B used the fewest interaction turns (77). It tends to lay out a plan first, then execute step by step. This clarity brings two benefits:

    • Fewer wasted round-trips: There is markedly less exploring the codebase through trial and error.
    • Higher prefix reuse: History context stays more stable and is more likely to hit the prompt cache. Its prompt cache hit rate reached 98%, meaning the vast majority of input tokens can be reused directly during prefill instead of being recomputed. Compute cost is therefore minimized.

    This makes B especially well-suited to running unattended in the background for long stretches. It should be noted that cross-request prompt caching usually requires the inference service layer to enable the relevant capability; if the local engine does not enable it, this benefit cannot be realized.

    3.3 Scheme C: Deep Reasoning, Quality First

    C took the longest (about 27.6 minutes), because it performed a large amount of internal reasoning in every interaction turn. Reasoning and tool calls advanced in an interleaved manner, so token consumption was naturally higher.

    This investment bought more standardized code and the most test cases (18). In an engineering context, the number of test cases often reflects how much a model considers edge cases: it does not just “make the feature work”—it proactively validates error paths.

    3.4 Key Takeaway: Time Measures Efficiency, Not Quality

    Observation Direct Cause Engineering Implication
    A has the shortest total time Least output, not the fastest engine Fast iteration should prioritize completion speed
    B has the fewest turns Plan first, less trial and error A stable prefix favors cache hits
    C has the most test cases Deep reasoning each turn, proactive edge-case validation Time and compute buy engineering quality

    These metrics do not live on the same axis: elapsed time measures efficiency, the number of tests measures quality, and turns and cache hit rate measure how resources are used. Conflating them easily leads to wrong conclusions. For example, if you judge purely by elapsed time, C would be undervalued—yet it may be exactly the right fit for teams with strict quality requirements.

    4. Beyond Quantization: What Really Shapes Agent Performance

    Quantization determines “how large a model can run, and how fast,” while the model’s own instruction-following, planning habits, and reasoning style determine “how the task gets completed.” A, B, and C behave noticeably differently, but they are different quantization versions of the same model family. A reasonable inference is that the reasoning style comes mainly from the model itself, while quantization mainly affects runtime efficiency and fine-grained behavior.

    Several other factors are equally important when it comes to engineering adoption.

    1. Coordinating context and cache management. Every extra interaction turn accumulates history, tool returns, and reasoning traces. Once the context exceeds the model window, it has to be trimmed or summarized, which may cause the model to “forget” its earlier plan. Cache hit rate, context length, and task stability are interrelated and should be evaluated together as one group of metrics.

    2. The size of tool return content. Stuffing hundreds of lines of logs or file content back into context at once quickly inflates token consumption and dilutes key information. A common practice is to return a summary or a structured summary first, and let the model expand and inspect as needed.

    3. Sampling and loop-control parameters. Parameters such as temperature, maximum output length, and the turn limit per task all significantly affect results and elapsed time, and should be fixed and recorded during evaluation.

    5. A Reusable Evaluation Methodology

    Abstracted out of this benchmark, the process boils down to three steps.

    Step 1: Build a verifiable task set.

    • Defect-fixing tasks: Each bug gets a minimal reproduction script that is guaranteed to fail before the fix, avoiding “fake fixes.”
    • Build tasks: Provide an explicit feature list and translate each item into an automatically executable check, covering dimensions such as successful build, working core features, complete error handling, and the presence of necessary tests. Keep the check list stable; otherwise results across models and quantization schemes are not comparable.

    Step 2: Record the complete run trace. At a minimum, collect the following metrics:

    • End-to-end elapsed time, interaction turns, and output tokens;
    • Input tokens and prompt cache hit rate;
    • Tool-call success rate and number of failed retries;
    • Pass rate on verifiable tasks and number of new tests added.
    1. import time
    2. from dataclasses import dataclass, field
    3. @dataclass
    4. class RunMetrics:
    5. label: str # Configuration label, e.g., A, B, C
    6. wall_time_s: float = 0.0 # End-to-end elapsed time (seconds)
    7. turns: int = 0 # Number of turns interacting with the model
    8. in_tokens: int = 0 # Cumulative input tokens
    9. out_tokens: int = 0 # Cumulative output tokens (reasoning + tool calls)
    10. cached_in_tokens: int = 0 # Input tokens that hit the prompt cache
    11. tool_calls: int = 0 # Number of tool calls
    12. tool_failures: int = 0 # Number of failed tool calls
    13. passed: int = 0 # Number of tasks passed
    14. total: int = 0 # Total number of tasks
    15. tests_added: int = 0 # Number of new test cases added
    16. notes: list = field(default_factory=list)
    17. @property
    18. def pass_rate(self) -> float:
    19. return self.passed / self.total if self.total else 0.0
    20. @property
    21. def cache_hit_rate(self) -> float:
    22. return self.cached_in_tokens / self.in_tokens if self.in_tokens else 0.0
    23. @property
    24. def tool_success_rate(self) -> float:
    25. ok = self.tool_calls - self.tool_failures
    26. return ok / self.tool_calls if self.tool_calls else 0.0
    27. def run_task(task, client, tools, verifier, cfg: dict) -> RunMetrics:
    28. """Evaluation loop for a single task: run the task -> verify the result -> record metrics"""
    29. m = RunMetrics(label=cfg["label"])
    30. history = [{"role": "system", "content": cfg["system_prompt"]}]
    31. started = time.perf_counter()
    32. for _ in range(cfg["max_turns"]):
    33. history.append({"role": "user", "content": task.prompt(history)})
    34. resp = client.chat(
    35. messages=history, tools=tools,
    36. temperature=cfg["temperature"], max_tokens=cfg["max_tokens"],
    37. )
    38. m.turns += 1
    39. u = resp.usage
    40. m.in_tokens += u["prompt_tokens"]
    41. m.out_tokens += u["completion_tokens"]
    42. m.cached_in_tokens += u.get("prompt_cached_tokens", 0)
    43. if resp.tool_calls:
    44. for call in resp.tool_calls:
    45. m.tool_calls += 1
    46. try:
    47. result = tools.execute(call) # Toolset injected by the evaluation framework
    48. except Exception as exc:
    49. m.tool_failures += 1
    50. result = f"error: {exc}"
    51. history.append({"role": "tool", "content": str(result)})
    52. elif verifier.done(resp.content): # The model considers the task done
    53. break
    54. m.wall_time_s = time.perf_counter() - started
    55. # Automatically judged by the machine: defects fixed, checks passed
    56. ok, tests_added = verifier.check()
    57. m.passed, m.total, m.tests_added = int(ok), 1, tests_added
    58. return m

    Step 3: Control variables and run multiple times. On the same hardware and with the same context length and sampling parameters, run each configuration at least 3 times, take the median, and observe the variance, so that a chance result is not mistaken for a conclusion. If conditions allow, run a second validation with a separate private task set that has not been published, to reduce the risk of the evaluation set being covered by the model’s training data.

    6. From Metrics to Decisions: A Selection Framework

    The direction given by the original benchmark can be further structured into a decision framework: check the hard threshold first, then weigh by scenario.

    Hard threshold. Any configuration whose pass rate on the target task set falls below the bar is eliminated outright. In this case, all three models passed every test, so they move on to the next layer of comparison.

    Scenario weights. Different scenarios are sensitive to different metrics. The following table is a reference for assigning weights:

    Scenario Speed Cost Engineering Quality Typical Focus Metrics
    Everyday development assistance Medium Medium Medium Balanced overall performance
    Unattended background tasks Low High Medium-high Cache hit rate, interaction turns, plan stability
    Rapid prototyping High Low Low End-to-end elapsed time
    Standardized delivery Low Medium High Code conventions, number of new tests

    For each configuration that clears the threshold, compute a composite score under the same weights and rank them:

    1. def rank(configs: list[RunMetrics], weights: dict) -> list[tuple[str, float]]:
    2. """First filter by the pass-rate gate, then sort by scenario weights"""
    3. MIN_PASS_RATE = 1.0 # Adjustable based on task risk
    4. TIME_REF = 1800.0 # Time normalization baseline: 30 minutes, use the P50 time of the target task set
    5. CANDIDATES = [c for c in configs if c.pass_rate >= MIN_PASS_RATE]
    6. if not CANDIDATES:
    7. return []
    8. best_time = min(c.wall_time_s for c in CANDIDATES)
    9. def score(c: RunMetrics) -> float:
    10. speed = best_time / c.wall_time_s # 0~1, fastest is 1
    11. economy = c.cache_hit_rate # 0~1, cache hit rate
    12. quality = min(c.tests_added / 10.0, 1.0) # Normalized to 0~1, cap at 10
    13. # Tool calls with failures lose one quality tier (3%)
    14. quality *= 0.97 if c.tool_success_rate < 0.97 else 1.0
    15. return (weights["speed"] * speed
    16. + weights["cost"] * economy
    17. + weights["quality"] * quality)
    18. return sorted(((c.label, round(score(c), 3)) for c in CANDIDATES),
    19. key=lambda x: x[1], reverse=True)
    20. # Scenario example: daily development, equal weights for all three
    21. DAILY = {"speed": 1.0, "cost": 1.0, "quality": 1.0}
    22. # Scenario example: standardized delivery, quality weight equals speed + cost
    23. STRICT = {"speed": 0.5, "cost": 0.5, "quality": 2.0}

    This yields scenario-based recommendations:

    • Everyday development tasks: Choose a configuration with balanced overall performance. In the original benchmark, the FP8 scheme (B) balanced efficiency and compute cost and is a typical example.
    • Unattended background tasks: Prefer configurations with a clear plan, few turns, a high cache hit rate, and low marginal cost.
    • Rapid prototyping and validation: Prefer configurations with concise output and the shortest end-to-end time, and top up testing and review before delivery.
    • Standardized delivery: Prefer configurations that proactively write tests and produce more standardized code, trading time for a lower rework rate.

    Two additional points. First, for code tasks with high security requirements, no local coding agent should replace human review; it should be paired with processes such as static analysis, dependency auditing, and secret scanning. Second, the weights and thresholds in the scoring framework are only a starting point and should be continuously calibrated against the target task set. Taking the three configurations above as an example, the results computed with the “daily development” weights are:

    • B: speed ~0.63, cost 0.98, quality ~0.81, composite score ~0.81;
    • C: speed ~0.44, cost 0.50, quality 1.00, composite score ~0.65;
    • A: speed 1.00, cost 0.50, quality 0.30, composite score ~0.60.

    This ranking matches the empirical judgment that “FP8 is recommended for daily development.” If you switch to the “standardized delivery” weights, the ranking becomes B, C, A, with B still in the lead. It is worth stressing that this is only a comparison of three configurations on a single evaluation set, and the conclusion must not be generalized into “a certain class of quantization scheme is universally better.”

    7. Bridging Local Evaluation and Cloud Deployment

    Local evaluation answers “which configuration to choose,” while cloud deployment answers “how to run at scale, stably and economically.” The two need a consistent measurement standard to close the loop.

    In the cloud (for example, Baidu AI Cloud), building a coding agent service can typically combine the following general capabilities:

    • Use a container platform to host the inference service and the agent runtime;
    • Use object storage to archive evaluation sets, model files, and build artifacts;
    • Use a message queue to distribute long-running tasks asynchronously, matching unattended background scenarios;
    • Use a logging service to accumulate per-turn interaction records, making it easy to review failed cases;
    • Use monitoring and alerting to continuously track metrics such as elapsed time, cache hit rate, and task success rate.

    The baseline established by local evaluation should be written into cloud monitoring, and a regression evaluation should be run after every engine or model version update, to avoid behavior drift caused by upgrades.

    Is prompt caching actually taking effect? If the local engine has not enabled the relevant capability, cross-request caching often fails to hit. Use the following method for a quick self-check:

    1. Send two requests to the inference service;
    2. Keep the system prompt identical across both, at about 800 tokens, with a different user question each time;
    3. Compare the returned input-token statistics: the second request’s cached-token count should be close to 800.
    1. SYSTEM = "You are a coding assistant." + "Here are the project guidelines. " * 200 # ~800 stable tokens
    2. def probe_cache(client):
    3. a = client.chat([{"role": "system", "content": SYSTEM},
    4. {"role": "user", "content": "Explain in one sentence what a unit test is."}])
    5. b = client.chat([{"role": "system", "content": SYSTEM},
    6. {"role": "user", "content": "Explain in one sentence what a code review is."}])
    7. # The cached-token field name varies by inference service; read it per the actual API
    8. first, second = a.usage, b.usage
    9. print("First input tokens:", first["prompt_tokens"])
    10. print("Second input tokens:", second["prompt_tokens"],
    11. "Cached tokens:", second.get("prompt_cached_tokens", 0))

    If the two prompt_tokens values are nearly unchanged and the cached count is close to 800, caching is in effect; if the cached count is always 0, check whether the engine has the relevant feature enabled.

    To actually capture the caching benefit, how you organize the prompt matters just as much: put stable content first and changing content last.

    1. # Recommended layout: stable prefix first, changing content last
    2. messages = [
    3. # Part 1: team coding conventions (nearly constant, should go first)
    4. {"role": "system", "content": project_coding_rules},
    5. # Part 2: tool definitions (fixed order and content, no dynamic add/remove)
    6. {"role": "system", "content": tool_definitions},
    7. # Part 3: task description (kept consistent within the same batch of tasks)
    8. {"role": "user", "content": task_brief},
    9. # Part 4: current context and latest instructions (changes every turn, placed last)
    10. {"role": "user", "content": f"Current file snippet:\n{current_snippet}\nLatest request: {latest_request}"},
    11. ]

    A common mistake is to place volatile fields such as timestamps or random task IDs at the start of the system prompt. Change even a single character, and everything after it becomes a “cache miss” and must be recomputed.

    8. Conclusion

    This benchmark on a dual-GPU workstation offers three takeaways:

    1. Completeness is the baseline; how it runs is the difference. The basic capability of local coding agents is becoming homogeneous. Quantization scheme, planning habits, and reasoning depth together determine how well-suited they are to different scenarios.
    2. Evaluation must be verifiable, reproducible, and attributable. Automatically judged defect fixing and check-based building, fixed controlled variables, and running multiple times to take the median are the prerequisites for a trustworthy conclusion.
    3. Selection should be based on scenario-based weighted evaluation, not a single ranking. No single configuration dominates across all scenarios. Establishing a baseline with local evaluation and continuously validating it with a cloud observability system is the sustainable engineering path.

    For teams preparing to deploy a coding agent locally, the recommendation is to first define 10–20 verifiable tasks that fit your own business, then run a baseline evaluation using the metric system in this article. A configuration chosen on this basis is often more reliable than any ready-made leaderboard.