Evaluating Coding Agents: Pass Rate, Completion, Duration, and Cost Trade-offs
This article explains why coding agent evaluation has shifted from checking individual code snippets to measuring long-horizon engineering delivery. It introduces four key metrics—pass rate, average completion degree, average duration, and per-run cost—and discusses why high completion does not necessarily lead to a higher pass rate. It also explains how agent frameworks affect overall performance and provides practical guidance for building a lightweight, reproducible evaluation pipeline, including task set design, metric definitions, observability, and scenario-based model and framework selection.
1. From “Can It Write Code?” to “Can It Deliver Engineering?”: The Shift in Evaluation Paradigm
Traditional code generation evaluation often stays at the function or snippet level. For example, it checks whether a piece of code passes unit tests, follows syntax rules, or contains obvious defects. This kind of evaluation was useful in the early stage. However, once coding agents are used in real production environments, they face end-to-end tasks such as application creation, feature development, code migration, and application conversion. These tasks often span multiple files, modules, and repositories.
Long-horizon tasks are fundamentally different from snippet-level tasks. Their difficulty comes from several factors:
- Global context is required: The agent must understand project structure, dependencies, and existing conventions across files and modules.
- Multi-step planning and iteration are required: A complete delivery often requires repeated cycles of generation, building, running, debugging, and fixing.
- Feedback cycles are long: A single task may take hours or even longer, covering building, testing, runtime verification, and other stages.
- Failures accumulate: An error in an early step may be amplified along the execution chain, and the cost of fixing it usually increases in later stages.
Therefore, an evaluation benchmark that reflects real engineering work should not focus mainly on “accuracy of a single generation.” Instead, it should measure the agent’s overall performance in a multi-step, long-horizon, and feedback-rich environment. This is why many benchmarks targeting real development scenarios have emerged: they evaluate models and agent systems from an engineering perspective, rather than a toy perspective.
2. Four Core Metrics: Quantifying Engineering Capability
A mature coding agent evaluation system is usually built around four related dimensions. Understanding their definitions and boundaries is essential for interpreting any benchmark report.
1. Pass Rate
Pass rate refers to the proportion of tasks in which the agent fully achieves the expected goal.
In most cases, “passing” is an all-or-nothing judgment. The agent must satisfy multiple conditions at the same time, such as correct functionality, successful build, and passing tests. Partial completion is not enough.
According to publicly reported results, on a set of about 30 long-horizon tasks, pass rates across different configurations generally fall between 17% and 29%. Although this number may seem low, it reflects the difficulty of long-horizon tasks: the proportion of tasks that can be “completely and correctly delivered” is much lower than the proportion of tasks where the agent only “appears to have done something.”
2. Average Completion Degree
Completion degree measures how far the agent progressed on a single task.
Even if a task ultimately fails, completion degree can still show how many subtasks were finished and how far the agent advanced. For example, a completion degree of about 82% may mean that the main framework has been established, but key components are still missing or defective.
Completion degree is a more continuous signal than pass rate. It helps capture the agent’s actual progress on failed tasks and is especially useful for analyzing capability boundaries and guiding future iterations.
3. Average Duration
Average duration is the average time required to complete a task end to end.
In long-horizon tasks, duration may reach tens of hours. This directly affects the iteration speed of research and development feedback. A solution that is more capable but takes tens of hours per run has a very different practical value from one that is slightly weaker but can return results in around ten hours.
4. Per-Run Cost
Per-run cost refers to the resource cost required to complete one evaluation run or one batch of tasks. It includes compute resources, token consumption, tool calls, and other overhead.
Publicly available data shows that per-run costs can vary significantly across configurations, ranging from single-digit dollar amounts to several hundred dollars, a difference of one to two orders of magnitude. Cost is the key metric that turns “technical feasibility” into “engineering economics.”
3. High Completion Does Not Always Mean High Pass Rate
One of the most commonly misread and most instructive phenomena in evaluation is this: a configuration with high completion does not necessarily have the highest pass rate.
Consider two contrasting configurations:
- Configuration A: Average completion degree is about 82%, average duration is about 27 hours, and per-run cost is as high as several hundred dollars.
- Configuration B: Average completion degree is about 71%, average duration is about 12 hours, and per-run cost is only a few dollars.
At first glance, Configuration A seems to do more and do it more thoroughly. However, because pass rate is an all-or-nothing judgment, Configuration A may repeatedly get stuck on a final critical path, such as an edge case or a failed build fix, causing the overall task to fail. Configuration B has a slightly lower completion degree, but its shorter duration and much lower cost make it highly attractive in terms of overall cost-performance.
This reveals a simple engineering fact: process quality and final outcome are not linearly related. Completing 82% of the work does not mean the agent is only 18% away from passing. In many cases, the final 10% to 20% of the critical path is exactly where success or failure is decided. When evaluating an agent, it is not enough to look only at “how much it did.” It is more important to see whether it can reliably finish the task.
4. Four-Way Trade-off: There Is No “Have It All” Solution
When pass rate, completion degree, duration, and cost are considered together, natural tensions appear among them. This forms a typical multi-objective trade-off:
- Capability strength vs. resource overhead: Higher completion usually requires more iterations and a larger reasoning budget, which increases duration and cost.
- Success rate vs. feedback timeliness: Achieving the highest pass rate may require longer thinking time and more verification rounds, which can weaken interactive experience.
- Per-run cost vs. batch scale: When processing a large number of tasks, small differences in per-run cost can be amplified and become a decisive factor.
Therefore, the value of evaluation is not to declare an “absolute first place.” Its real value is to quantify and visualize these trade-offs so that teams can choose the most suitable balance point based on their own constraints, such as budget, latency, and quality requirements. The final output of evaluation should be a clear “capability-cost-latency” trade-off map, not an isolated ranking.
5. Why Agent Frameworks Significantly Affect Results
A frequently overlooked but important conclusion is this: model capability is not equal to system capability.
For the same type of task, different coding agent orchestration frameworks can produce very different results. The reason is that the agent framework acts as the model’s “scaffolding.” Its design directly affects final performance, mainly in the following areas:
- Task decomposition strategy: Whether the framework breaks a long-horizon task into reasonable subtasks, or simply asks the model to generate everything in one shot.
- Context management: How the framework trims, retrieves, and organizes context across files to avoid information overload or missing key details.
- Tool invocation and feedback loops: Whether the framework can reliably call build, test, and runtime tools, and feed their results back to the model to form a closed loop.
- Error recovery capability: When a step fails, whether the framework blindly retries or can locate the root cause and apply targeted fixes.
This means that the evaluation target should be the combination of “model plus framework,” not the model in isolation. Selection should also broaden accordingly: it is not enough to ask whether the model is good enough. It is equally important to ask whether the orchestration framework can reliably unlock the model’s potential.
6. Practical Guide: Building an Evaluation System for Your Team
Using the above metrics, development teams can build a lightweight and reproducible evaluation pipeline.
6.1 Task Set Design
The task set should be realistic, verifiable, multi-type, and reproducible in isolation:
- It should cover task types such as application creation, feature development, code migration, and defect fixing.
- Every task should have a clear and automated acceptance standard, such as all tests passing, successful build, or expected API behavior.
- Task execution environments should be isolated to ensure that results are reproducible and comparable.
6.2 Metric Definitions
The evaluation should remain “four-dimensional” and avoid being driven by a single metric. The report should record and present pass rate, average completion degree, average duration, and per-run cost together. This prevents both “pass-rate-only thinking” and “cost-only thinking.”
6.3 Cost and Observability
Coding agent execution is a long chain and requires fine-grained observability. Teams can use general cloud platform capabilities such as logging, monitoring, alerting, and tracing to collect and attribute the time and cost of each tool call and each iteration.
For example, with cloud platforms such as Baidu AI Cloud, log services and monitoring/alerting capabilities can help build unified observability for the agent execution chain, making it clear where money is spent and where time is consumed.
6.4 Illustrative Evaluation Framework Code
The following is a skeleton for an evaluation harness that collects the four core metrics. It is illustrative code and does not represent any specific product interface.
from dataclasses import dataclass, field@dataclassclass TaskResult:task_id: strpassed: bool # Whether the task fully passedcompletion: float # Completion degree, 0.0 to 1.0duration_sec: float # End-to-end durationcost_usd: float # Per-run cost@dataclassclass EvalReport:results: list[TaskResult] = field(default_factory=list)def summarize(self) -> dict:n = len(self.results)if n == 0:return {}pass_rate = sum(r.passed for r in self.results) / navg_completion = sum(r.completion for r in self.results) / navg_duration = sum(r.duration_sec for r in self.results) / navg_cost = sum(r.cost_usd for r in self.results) / nreturn {"pass_rate": round(pass_rate * 100, 1),"avg_completion": round(avg_completion * 100, 1),"avg_duration_hours": round(avg_duration / 3600, 1),"avg_cost_usd": round(avg_cost, 2),}def run_eval(agent, tasks) -> EvalReport:report = EvalReport()for t in tasks:res = agent.run(t.spec) # Execute a long-horizon taskreport.results.append(TaskResult(task_id=t.id,passed=res.verify_passed(), # Automated acceptance checkcompletion=res.estimate_completion(),duration_sec=res.duration_sec,cost_usd=res.consume_cost(),))return report
The key design points are: acceptance criteria should be automated, the four metrics should be presented together, and duration and cost should be attributable to each subtask.
7. Selection Guidance: Match the Balance Point to the Scenario
The ultimate value of evaluation is to support selection decisions. Different team scenarios require different balance points:
- Budget-sensitive or latency-sensitive teams: Prefer lower-cost, faster configurations. They may accept a controlled completion degree in exchange for significant efficiency and cost advantages. This is suitable for high-frequency, batch-style routine tasks.
- Scenarios requiring the highest success rate and accepting long duration and high cost: Prefer higher-completion configurations for critical deliveries where quality requirements are strict and the task must be completed.
- Interactive coding assistance: Response latency is more important. Average duration should be treated as a core constraint.
- Large-scale offline batch processing: Small differences in per-run cost can be amplified at scale. Cost should be the primary optimization target.
8. Conclusion
Evaluation of coding capabilities for real engineering scenarios has evolved from checking single-point accuracy to assessing comprehensive long-horizon delivery. Pass rate, completion degree, duration, and cost together form a complete picture of the trade-off.
Two insights are especially important: high completion does not always mean high pass rate, and model capability does not equal system capability. Understanding these points helps teams look beyond benchmark scores and focus on engineering reality: there is no absolutely optimal agent, only the balance point that best matches a team’s constraints.
A solid evaluation system and a clear understanding of these metrics enable teams to make rational and sustainable technical decisions across capability, cost, and timeliness.
