Evaluating Agentic Coding Capabilities of Locally Quantized LLMs: Benchmarking and Model Selection
When large language models are used as agents rather than one-shot answer generators, evaluating their coding ability becomes far more complex than checking whether a single response passes unit tests. This article explains why agentic coding deserves its own evaluation, outlines the elements of a trustworthy and reproducible benchmark, and breaks down how quantization affects speed, cost, and quality across three orthogonal dimensions. It describes three working modes commonly observed in practice, offers scenario-based model selection guidance, and covers the engineering infrastructure needed to run reproducible evaluations. The core takeaway: the goal is not to pick the strongest model, but to build a methodology that measures speed, cost, and quality independently.
1. Why evaluate agentic coding capability separately?
Traditional code-generation benchmarks usually focus on the accuracy of a single turn: you present a problem and check whether the model can produce code that passes unit tests. But when an LLM is used as an agent, the evaluation dimensions become far more complex. It must autonomously plan tasks, invoke external tools as needed, read and modify code across multiple turns, run tests, and iteratively fix failures based on error output before delivering a working result. This ability to complete an engineering task end to end is fundamentally different from getting a single algorithm right.
At the same time, driven by cost, privacy, and control concerns, more and more teams are deploying open-source models locally or in private environments. These models are typically quantization-compressed to reduce GPU memory usage and improve throughput, but the impact of different quantization schemes on model capability is often opaque. Building an evaluation method that is reproducible, quantifiable, and automatically scored therefore has direct engineering value for local model selection.
2. Elements of a trustworthy evaluation method
A benchmark that produces reliable conclusions should cover at least the following three points.
1) Multi-dimensional task design that covers different capability facets. A single task cannot characterize agentic ability. It is better to combine three types of tasks:
- Tool-use capability: Ask the model, acting as an agent, to invoke several native tools (reading files, executing commands, retrieval, writing, and so on) to assess its planning and tool-orchestration skills;
- Defect-fixing capability: Provide a codebase with known defects and ask the model to locate and fix multiple issues, assessing its context comprehension and reasoning-based localization;
- End-to-end build capability: Build a complete application from a single requirement and validate it against a fixed set of acceptance checks (such as functional points, boundary handling, and a successful build), assessing engineering delivery ability.
2) Machine-based automatic scoring to avoid subjective judge bias. Introducing an LLM-as-judge is convenient but introduces subjectivity and instability. A more robust approach is to use deterministic scoring signals: unit-test pass rate, acceptance-script results, number of static-analysis warnings, whether the build succeeded, number of generated test cases, and so on. These metrics are reproducible and regression-friendly.
3) Recording both efficiency and quality metrics. Beyond accuracy, you should also collect efficiency data such as elapsed time, total output tokens, number of interaction rounds, and cache hit rate. This lets you evaluate “can it do it correctly” separately from “is it fast and cheap.”
An evaluation pipeline can be summarized by the following pseudocode:
for model, quant in evaluation_set: # model x quantization schemeresults = []for task in task_suite: # tool use / bug fixing / app buildingsession = AgentSession(model, tools=native_toolset)while not session.done and session.rounds < MAX_ROUNDS:action = session.step() # model plans and decides the next moveobservation = run_tool(action) # execute the tool and return the resultsession.observe(observation)score = deterministic_scorer(session) # unit/acceptance/static checks, no LLM judgeresults.append(score)report(model, quant, aggregate(results)) # summarize quality + efficiency metrics
3. How quantization affects performance: three orthogonal dimensions
One of the most common misunderstandings in evaluation is treating “model capability” as a single scalar. In reality, the differences introduced by quantization schemes can be decomposed across three orthogonal dimensions:
- Speed ≈ total output tokens ÷ inference-engine throughput. Quantization mainly affects engine throughput, but a model’s verbosity (how much it talks) matters just as much. A model with a slightly slower engine but extremely concise output can actually be faster overall, because the tokens saved offset the per-token speed disadvantage.
- Cost ≈ interaction rounds × input size per round × unit price, strongly influenced by cache hit rate. If a model tends to “plan first, then execute,” it uses fewer interaction rounds, and when the prompt cache hit rate is high, repeatedly passed context does not need to be recomputed, so per-task cost drops significantly.
- Quality is reflected in code cleanliness, maintainability, and test coverage, and correlates positively with a model’s reasoning depth per round (amount of internal thinking), though it usually comes at the cost of longer elapsed time.
Once you understand these three points, you see that “fastest” does not mean “best,” and “cheapest” does not mean “most stable.” The three must be weighed against the specific scenario.
4. Three working modes observed in a typical evaluation
Under the same task set and scoring framework, different quantization schemes for open-source models often exhibit markedly different “working modes.” The following are illustrative observations from one typical evaluation (used to explain the method, not as absolute conclusions):
- Minimal-execution type: Produces very few output tokens over the whole run and completes an end-to-end application build in about 18 minutes, the fastest of the three. Its advantage comes from extreme conciseness: even with average underlying engine throughput, it takes the overall speed lead by “saying less,” making it well suited to rapid prototyping.
- Plan-orchestration type: Has the fewest interaction rounds and a prompt cache hit rate as high as about 98%, giving it the lowest compute cost per unit. It tends to “make a plan first, then execute step by step,” with a clear and auditable process, making it especially suitable for long-running autonomous operation in the background.
- Deep-reasoning type: Performs substantial internal reasoning on nearly every round, so it takes the longest (about 27 minutes), but produces the cleanest code and the most test cases (about 18), delivering the best engineering quality. It suits high-standard, high-coverage production delivery.
It is worth emphasizing that in this evaluation, all three models passed every task. This shows that they have converged at the “can it be done” level; the real differences lie in the efficiency–cost trade-off curve, which is exactly what matters for selection.
5. Scenario-based selection recommendations
Combining the dimensions above, the following selection approach aligns with business goals:
- Everyday development, cost-sensitive: Prefer the plan-orchestration type — high cache hit rate and low round count yield lower per-unit cost, and the execution process is traceable;
- High standards, high test coverage: Prefer the deep-reasoning type — trade longer elapsed time for cleaner code and more thorough tests;
- Rapid validation, quick prototypes: Prefer the minimal-execution type — get a working result fastest with the least output.
One caveat: any conclusion should be validated against your own real task distribution. Differences in codebase size, target language (for example, the toolchain differences between functional languages and mainstream object-oriented languages), the available toolset, and acceptance criteria can all change the ranking. It is therefore advisable to treat the evaluation scripts as a regression asset and re-run them whenever a candidate model or quantization version is updated.
6. Engineering implementation and resource preparation
Building such an evaluation pipeline requires a certain level of compute and engineering infrastructure:
- Compute: A single dual-GPU desktop-grade inference workstation is enough to run a small-scale evaluation; if you want to evaluate multiple “model × quantization scheme” combinations in parallel, you need elastic, scalable heterogeneous compute;
- Pipeline: You can build on general cloud capabilities, such as heterogeneous computing/GPU cloud hosts to carry inference, container platforms to orchestrate evaluation tasks, object storage to archive corpora and artifacts, and logging plus monitoring-and-alerting services to observe elapsed time and resource usage. Taking Baidu AI Cloud as an example, its general capabilities such as heterogeneous computing, container and model services, object storage, and monitoring and alerting can serve as the resource foundation for building such a reproducible evaluation environment;
- Reproducibility: Fixing random seeds, tool versions, and acceptance scripts, and recording complete metrics for each run, are what make evaluation conclusions hold up to regression and comparison.
7. Conclusion
Evaluating the agentic coding capability of locally quantized LLMs is not about selecting the “strongest model”; it is about building a methodology that decouples and measures speed, cost, and quality. When you can replace subjective judges with deterministic scoring, cover real workflows with multi-dimensional tasks, and understand the trade-offs quantization makes across each dimension, model selection shifts from “gut feeling” to “data-driven.” Treating the evaluation scripts as a long-term asset and continuously running them is the robust way to keep up with rapid model iteration.
