Local Coding Agent Evaluation and Quantization Selection on Dual Accelerators
A practical framework for evaluating locally deployed coding agents under fixed tools, time budgets, audit requirements, and limited compute. This article compares three representative execution patterns—minimal-output, plan-and-execute, and deep-reasoning workflows—and explains how to measure quality, cost, latency, stability, caching, and auditability without reducing model selection to a single pass-rate metric.
When deploying coding agents locally, teams often make the mistake of treating a single successful bug fix as a complete measure of model capability. Production environments require a broader answer: under a fixed toolset, time budget, auditable scoring system, and limited compute, can the agent reliably complete multi-step tasks and deliver maintainable code at an explainable cost?
A controlled, anonymized experiment offers several useful observations. Three model configurations using different quantization and inference strategies all completed bug-fixing and application-building tasks. One achieved the shortest completion time by minimizing output. Another reduced interaction rounds and improved cache reuse by planning before execution. A third used deeper internal reasoning to produce more规范 code and more test cases. Rather than presenting a product comparison, this article extracts a reusable evaluation methodology, metric system, and model-selection framework.
1. Abstracting the Experiment into a General Evaluation Problem
Local coding-agent evaluation typically has three layers:
- Model layer: code understanding, completion, reasoning, and generation.
- Tool layer: file operations, command execution, test running, static analysis, and build or release tasks.
- Engineering layer: isolation, auditing, cost, latency, and regression stability.
Comparing only the model layer can produce a system that performs well on individual tasks but is impractical to operate. Comparing only the engineering layer may hide meaningful quality differences caused by different reasoning strategies.
A representative controlled experiment can be designed as follows: the agent uses 21 native tools to complete 17 bug-fixing tasks and an application-delivery task containing 19 checks. Scoring is performed automatically through deterministic tests, static analysis, coverage, and build results, without using an LLM judge.
This design improves reproducibility and auditability. It also reduces the risk that a natural-language explanation will influence the evaluation. In the anonymized records, all three model groups passed the tests. The minimal-output group completed the build in approximately 18.4 minutes and generated about 20,000 tokens. The low-interaction planning group used 77 interaction rounds and achieved a 98% prompt-cache hit rate. The deep-reasoning group took approximately 27.6 minutes and generated 18 test cases.
These figures illustrate relationships between metrics only. They should not be generalized as fixed capabilities of any product.
2. Pass Rate Is Not Enough
Pass rate is a baseline requirement, not a complete selection criterion. At minimum, evaluations should record the following metrics:
- Functional pass rate: Whether bug fixes pass existing tests and whether the application passes all required checks.
- Engineering quality: Static-analysis findings, duplicated code, exception-handling coverage, interface-boundary handling, and the number and effectiveness of generated tests.
- Interaction efficiency: Interaction rounds, tool calls, failed retries, and the percentage of ineffective calls.
- Generation cost: Input tokens, output tokens, cache-hit rate, and equivalent cost per task.
- End-to-end latency: Queue time, prefill time, decoding time, tool-execution time, and test-running time.
- Stability: Success rate across repeated runs, latency variance, output differences, and score fluctuations.
- Auditability: Whether plans, tool calls, patches, and test results are fully recorded and replayable.
A low output-token count does not necessarily mean lower total cost. Prefill, tool waiting, and test execution may account for a large share of runtime. Similarly, fewer interaction rounds do not always mean fewer total tokens, although they generally indicate more concentrated planning and fewer context switches.
A high cache-hit rate shows that repeated context is being reused effectively. This is valuable for long-running background jobs and stable tool protocols, but cold-start performance must be measured separately.
A practical scoring model can be expressed as follows:
task_suite = {"bugfix_cases": 17,"web_checks": 19,"native_tools": 21,"budget": {"wall_minutes": 30, "max_tokens": 120000},"trace": ["plan", "tool_call", "patch", "test", "static_check", "coverage"],"scoring": "deterministic_only"}score = 0.50 * pass_rate+ 0.15 * static_check_clean+ 0.15 * coverage+ 0.10 * time_efficiency+ 0.05 * cost_efficiency+ 0.05 * stability
These weights are not universal. Prototype validation may emphasize time efficiency. Production bug fixing may prioritize pass rate and stability. Platform-scale operation must include cost, auditability, and failure recovery as core metrics.
3. Three Representative Operating Modes
Mode A: Minimal Output for Maximum Speed
Some models minimize explanations and intermediate text, producing only the required patches and commands. This approach often improves end-to-end time because fewer decoding tokens are generated, context grows more slowly, and the tool-execution chain is shorter.
In the anonymized records, this mode completed the build in approximately 18.4 minutes with about 20,000 generated tokens. The result shows that “saying and writing less” can itself be an efficiency strategy.
The risks are equally clear. Comments, change descriptions, and boundary tests may be insufficient, leaving maintainers with less context. Engineering controls should therefore require a minimal change summary, automatically generate a diff summary, and use static analysis and regression tests as safeguards.
Mode B: Plan-and-Execute with Cache Reuse
This mode first creates an execution plan and then invokes tools step by step. It typically reduces interaction rounds and makes the process easier to inspect. When system prompts, tool protocols, and repository context remain stable, prompt-cache hit rates can become very high, reducing repeated prefill cost.
In the anonymized records, this mode used only 77 interaction rounds and reached a 98% cache-hit rate. It is well suited to background autonomy, batch repair, and scheduled repository checks.
However, high cache utilization depends on stable prefixes and reusable context. Cold starts, frequent repository changes, or modified tool protocols can reduce the advantage. Cold-start cost, warm-cache cost, and cache invalidation rate should therefore be tracked separately.
Mode C: Deep Reasoning for Higher Quality
This mode performs more internal reasoning during each interaction. It takes longer, but it is more likely to produce规范 code and more complete tests.
In the anonymized records, it took approximately 27.6 minutes but generated 18 test cases, indicating stronger engineering quality. This approach is suitable for pre-release verification, complex cross-file refactoring, and deliverables with strict standards.
The trade-off is higher latency and compute consumption. It should not be applied to every task. A practical solution is tiered routing: send simple defects through a lightweight strategy and escalate complex tasks to deep reasoning when explicit thresholds are met.
4. Evaluate Quantization by Behavioral Impact, Not Memory Usage Alone
Quantization reduces memory consumption and bandwidth pressure, but different quantization schemes can affect long-context handling, tool-parameter formatting, code-detail preservation, and output stability in different ways.
The key question is not only whether a model can run, but whether it remains reliable across 21 tools, long tool-call chains, and structured parameter generation. Three comparisons are especially useful:
- Cold-start comparison: Run the same task with an empty cache and measure prefill, decoding, and tool-waiting time.
- Warm-cache comparison: Keep the system prompt and tool protocol fixed and measure the cost reduction after cache reuse.
- Long-context comparison: Test cross-file fixes and recovery from repeated test failures, observing the effects of context trimming, summarization, and retry strategies.
Quantization precision should not be treated as a direct synonym for code quality. A tighter context budget may encourage shorter outputs and improve speed, or it may cause the model to miss edge cases. Deeper reasoning may improve test coverage, but it can also lead to overengineering.
The correct approach is to evaluate quantization schemes, prompt strategies, tool protocols, and budget limits together in an experiment matrix rather than comparing isolated single-run results.
5. Selection Matrix: Choose by Scenario, Not by an Absolute “Best” Model
| Scenario | Priority metrics | Recommended strategy | Main risks and controls |
|---|---|---|---|
| Rapid prototypes and one-off scripts | End-to-end time, output tokens | Minimal output, short budgets, automated tests | Insufficient documentation and testing; use manual sampling |
| Daily bug fixing and background batches | Interaction rounds, cache reuse, stability | Plan-and-execute, fixed tool protocols, warm-cache optimization | Cold-start cost and context drift |
| Pre-release checks and complex refactoring | Pass rate, static analysis, coverage | Deep reasoning, more test generation, tiered escalation | Higher latency and compute cost |
| High concurrency and cost-sensitive workloads | Cost per task, retry rate | Lightweight routing, hard budgets, fast failure fallback | Lower success rate on complex tasks |
| Multi-team platform operation | Auditability, replayability, observability | Unified traces, scoring pipelines, staged rollout | Overfitting to a narrow task suite |
The conclusion is usually not to select a single model. Instead, build a routing system: use a low-cost path for simple tasks and escalate complex work to deep reasoning. Require testing and human review for high-risk changes, enable caching for repeated context, and retain a complete trace for every run.
6. Implementation Checklist
- Build a layered task suite: Cover simple defects, cross-file fixes, tool failures, permission denials, long-context tasks, and build-delivery workflows.
- Use deterministic scoring: Prioritize functional tests, static analysis, coverage, and build results. Use natural-language evaluation only as supplementary evidence.
- Record complete traces: Plans, tool calls, patches, test output, retries, and budget consumption should all be replayable.
- Use sandbox isolation: Minimize filesystem, network, command-execution, and credential permissions to prevent unauthorized actions.
- Set hard budgets: Cap time, tokens, tool calls, and retries, and define fallback behavior for timeouts.
- Run repeated trials: Execute each task multiple times and report averages, variance, and failure distributions.
- Separate cold and warm cache results: Do not mistake cache benefits for model capability.
- Create regression gates: Re-run the benchmark suite whenever the model version, prompt, tool protocol, or quantization scheme changes.
- Perform manual quality reviews: Focus on edge cases, exception handling, security issues, and maintainability—not only whether tests pass.
- Roll out gradually: Start with low-risk repositories and background jobs before expanding permissions and task scope.
7. Common Mistakes
First, treating pass rate as the only metric. Passing every test shows that the baseline requirement was met, not that cost, latency, and maintainability are acceptable.
Second, treating cache-hit rate as a model capability. Caching is a systems-engineering outcome that depends on stable prefixes, tool schemas, and context management.
Third, ignoring tool-call failures. In real environments, permission denials, command timeouts, and flaky tests all affect end-to-end success.
Fourth, optimizing too aggressively for short output. Extremely short output can reduce auditability and increase the cost of later communication.
Fifth, drawing conclusions from a single run. Model randomness, quantization, temperature, context, and tool state all introduce variance.
Sixth, using an overly narrow evaluation set. If the suite covers only one language or one type of defect, the resulting selection can easily overfit to the benchmark.
8. Conclusion
The core challenge of running coding agents locally on dual accelerators is not finding a model that dominates every metric. It is building a reproducible evaluation and operations system.
Use deterministic tests to enforce the quality baseline, traces and budgets to control cost, tiered routing to match task complexity, and repeated trials to reduce the impact of randomness. Minimal output is effective for rapid prototyping, plan-and-execute workflows are suitable for background automation, and deep reasoning is valuable for high-quality delivery.
The most mature approach combines these modes into an observable, recoverable, and continuously improving engineering pipeline.
