Evaluating LLM Coding Agents for Production: Real Merge Rate, Cost per Accepted Task, and Long-Horizon Reliability

1. Background: From “Can It Write Code?” to “Is It Worth Using?”

When a new top-tier model is released, developers are usually less concerned with whether it can generate code at all. The more practical questions are: first, can it reliably deliver code in real engineering work that can be merged and accepted; second, is its cost per successful task low enough to support daily development, batch tasks, and multi-agent collaboration?

In the past, model capability was often evaluated with isolated coding problems, algorithm questions, or simple backend API generation. Those evaluations still have value, but they are no longer enough to distinguish top-tier models. The reason is that top-tier models are converging on basic backend engineering tasks: generating a CRUD endpoint, adding middleware logic, or writing a unit test is now within reach for many models. The real differences appear in stages closer to production: whether code can pass maintainer review, whether it reduces regression defects, whether it maintains state consistency in long-horizon tasks, whether it can detect frontend visual issues from screenshots, and how many tokens, retries, and manual acceptance hours are needed to complete a successful task.

Evaluation logic should therefore move from “does the model output look correct?” to “is the task actually accepted?” This is also a prerequisite for cost-effectiveness evaluation. Without a unified acceptance standard, comparing unit prices alone is meaningless.

2. Capability Evaluation: Real Merge Rate Is More Convincing Than Single-Point Pass Rate

For coding agents, priority should be given to “real code merge rate” metrics. A real merge rate does not measure whether generated code passes an isolated test. It measures whether the code can fulfill a requirement in an existing codebase, pass continuous integration, avoid breaking existing behavior, and ultimately be accepted by repository maintainers. This metric is closer to real development because it examines requirement understanding, context localization, style consistency, test supplementation, and maintainability at the same time.

A reusable evaluation framework can include:

• Requirement completeness: Are acceptance criteria covered? Are edge cases missed?
• Integration quality: Does the change follow existing directory structure, naming conventions, and error-handling practices?
• Regression risk: Does it introduce new failing test cases or performance degradation?
• Maintainability: Is the code over-abstracted? Does it generate patches that are difficult to understand?
• Review cost: How many rounds of modification are needed before a human review can merge it?

It is important not to equate a single percentage from a public benchmark with production readiness. A better approach is to build a small internal benchmark set. Select 20 to 50 historical tasks from real repositories, retain the original tests and review records, let candidate models complete the tasks in isolated environments, and then run tests, static checks, and manual spot checks with unified scripts. This produces conclusions that match the team’s technology stack.

3. Cost-Effectiveness: Replace Unit Price Comparison with Cost per Accepted Task

Model pricing is usually based separately on input tokens and output tokens, and output tokens often cost more than input tokens. Therefore, cost-effectiveness should not be judged only by input price or single-call price. Instead, calculate the cost per accepted task:

Cost per accepted task =
  (input token cost + output token cost + retry cost + tool invocation cost + manual acceptance cost)
  / number of tasks that pass acceptance

This formula has three key points.

First, retry cost must be included. Agent tasks often do not succeed on the first attempt. Failed reruns, context replay, and test fixes all increase consumption. A model with a lower single-call price but a high retry rate may be more expensive in practice.

Second, output bloat must be controlled. Many agents generate long explanations, repeated logs, or large blocks of irrelevant code, causing output token usage to grow quickly. Optimization strategies include limiting output format, requiring patch-style output, summarizing long contexts, and setting budget circuit breakers for failed tasks.

Third, manual acceptance cost cannot be ignored. If generated code requires extensive human rework, the overall cost-effectiveness remains poor even if the token price is very low. Evaluation tables should record the human modification time for each task and convert labor cost into total cost.

There is also a Pareto frontier problem between cost and capability. Some models can achieve a high pass rate at a lower reasoning level, while others need the highest level to achieve similar results. For developers, the goal should be to find the lowest level needed to reach the acceptance threshold, rather than blindly pursuing the highest configuration. Tiered strategies can be set by task type: use lower-cost levels for routine changes, and raise the level for complex refactoring or long-horizon tasks.

4. Safety Tiering Defines the Boundary of Usability

From a platform perspective, top-tier models often use safety capability tiering: publicly available versions include guardrails for high-risk domains, while vetted professional organizations may receive controlled versions. This tiering is not simply feature reduction; it is a balance between compliance and usability.

Developers should consider three points when selecting models:

1. Whether the open version covers the target business scenario. If the business involves high-risk professional domains, confirm compliance boundaries first rather than looking only at benchmark scores.
2. Whether safety guardrails affect normal development tasks. Over-blocking reduces agent automation rates and increases manual intervention.
3. Whether logs and audits are traceable. In multi-agent collaboration, it must be possible to determine whether a failure comes from model capability, tool invocation, or a safety policy block.

For most general development scenarios, the open version is sufficient. For regulated industries, safety review, data isolation, and audit capabilities should be included in total cost.

5. Frontend Visual Inspection Becomes a New Capability Divide

Frontend development is one of the most underestimated scenarios in coding agents. Backend APIs can be verified with test assertions, but whether a frontend page is “visually correct, usable, and properly aligned” is difficult to judge from text alone. Models need vision capabilities: reading screenshots, identifying layout misalignment, detecting insufficient color contrast, checking responsive breakpoints, and locating overflow and occlusion issues.

Therefore, chart understanding and screenshot review capabilities deserve separate evaluation. A practical frontend agent acceptance flow can be designed as follows:

• Launch the page and generate screenshots at multiple sizes.
• Have the agent compare the design mockup or historical screenshots and output a list of visual differences.
• Automatically check basic accessibility items such as contrast, focus states, and semantic tags.
• Generate fix patches for the differences.
• Take screenshots again and perform visual regression comparison.

With these capabilities, AI can discover some visual issues on its own and reduce the burden of “generate first, then inspect with human eyes.” However, visual models may misjudge subtle spacing, font rendering, and dynamic interactions, so manual spot checks should still be retained.

6. Evaluation Design for Multi-Agent Long-Horizon Collaboration

Multi-agent long-horizon tasks are a difficult area in current evaluation. Letting multiple agents work in parallel on independent directories and run continuously for hours or even days can reveal model performance in state persistence, task decomposition, conflict avoidance, budget control, and failure recovery. Evaluation should focus on:

• Isolation: Does each agent have an independent working directory, branch, and dependency environment?
• Recoverability: Can the process resume from a checkpoint after interruption?
• Collaboration overhead: Does inter-agent communication produce redundant context?
• Budget control: Can the system automatically downgrade or stop before overspending?
• Acceptance consistency: Do different agents use the same tests and review standards?

It is recommended to define evaluation tasks in a configuration-driven way, for example:

{
  "task_id": "frontend-visual-fix-001",
  "workspace": "isolated",
  "max_budget_tokens": 200000,
  "steps": ["reproduce", "patch", "test", "screenshot", "review"],
  "acceptance": ["unit_test_pass", "visual_diff_below_threshold", "manual_spot_check"]
}

The value of long-horizon tasks is not that they run for a long time, but whether they can maintain stable output without human intervention. If frequent human correction is needed midway, task decomposition or context management still needs optimization.

7. Real-Project Acceptance: Less Intervention, but Strict Acceptance

For real-project evaluation, it is advisable to launch multiple independent sub-agents at the same time, each responsible for an isolated project directory, with as little human intervention as possible throughout the process, followed by unified acceptance. Acceptance should not only check whether errors occurred. It should cover:

• Whether all automated tests pass.
• Whether static checks and style checks pass.
• Whether screenshots of key pages match expectations.
• Whether changes are concentrated within the requirement scope.
• Whether the amount of manual review modification is acceptable.
• Whether total token cost and total time are within budget.

For top-tier models, basic backend engineering should not be the only focus, but it should not be abandoned completely. A more reasonable approach is to reduce the weight of repetitive backend tests and shift evaluation focus to real merge rate, end-to-end acceptance, frontend visual quality, long-horizon agent orchestration, and cost per accepted task. This is not to downplay backend work. Backend fundamentals are increasingly becoming an entry ticket, and comprehensive delivery quality is what truly affects deployment efficiency.

8. Implementation Recommendations

First, build a small but high-fidelity internal benchmark set and continuously record pass rate, retry rate, and manual modification time. Second, set model tiers and budget caps by task type, prioritizing output bloat and failed reruns. Third, include safety guardrails, log auditing, and compliance boundaries in model selection. Fourth, add screenshot review and visual regression for frontend scenarios. Fifth, multi-agent tasks must include isolation, checkpoints, and budget circuit breakers. Sixth, for teams that already use Baidu AI Cloud, general-purpose computing, object storage, and logging and monitoring services can support an evaluation pipeline; the core of the assessment, however, remains task acceptance and cost accounting.

Overall, competition among new top-tier models has moved from single-point capability demonstrations to real engineering delivery and unit cost. Developers should avoid being misled by a single benchmark or short-term price fluctuations. A reproducible acceptance process and cost model are needed to determine whether a version upgrade actually brings efficiency gains.