Evaluating and Selecting AI Coding Agents for Long-Horizon Engineering Tasks
This article examines how to evaluate AI coding agents for real engineering work, where success depends on completing multi-session, multi-step tasks rather than generating a single correct function. It explains four core metrics—pass rate, average completion level, average elapsed time, and cost per run—how to interpret them together, and how to compare configurations using a public benchmark sample. It also provides practical guidance for task scheduling, observability, code quality gates, security, data compliance, cost governance, and building an internal benchmark. Finally, it offers a selection checklist for teams evaluating whether and how to deploy coding agents.
1. Evaluation Has Shifted from Writing a Correct Code Snippet to Completing an Engineering Task
At this stage of code-generation development, producing a correct single function is no longer the main bottleneck. What developers care about more is whether an AI coding agent can independently plan, iterate, and deliver an acceptable change when given a complete engineering request. Such a request may involve building an application from scratch, adding a new feature, migrating an old implementation to a new framework, or fixing a defect that spans multiple files.
The typical working loop of a coding agent is “perceive—plan—act—observe.” The agent first reads the repository and the requirement, then breaks down the steps and generates modifications. It then executes commands such as builds and tests, observes errors and logs, and decides the next step. In this loop, a task may require dozens of interactions and hours or even longer to complete. This kind of work is commonly called a long-horizon task.
Long-horizon tasks are fundamentally different from single-turn code completion:
- Context spans multiple sessions. The agent needs to remember earlier decisions, avoid conflicting changes, and prevent later modifications from undermining earlier work.
- Results are hard to judge in a binary way. The fact that code compiles or tests pass does not necessarily mean the requirement has been correctly implemented. Whether the architecture is reasonable and edge cases are covered still requires human judgment.
- Cost is not limited to model inference. Agents repeatedly build, run tests, and collect logs inside sandboxes. This time cost grows non-linearly with task complexity.
- Failure is normal, not exceptional. Ambiguous requirements, implicit conventions, and environment differences can all cause a task to drift off course.
As a result, evaluation for real engineering tasks is shifting from “whether a generated snippet is correct” to “whether the quality of the result is balanced against the cost of obtaining it in a long-horizon task.”
The analysis below uses a recent public coding benchmark as an observation sample. The benchmark covers tasks such as application creation, feature development, code migration, and application conversion, and it evaluates a model together with the agent runtime framework that runs it. This choice of evaluation target is itself informative: the same model can perform very differently under different agent frameworks. “Selecting the model” and “selecting the agent framework” must be considered together.
2. Metrics: Four Indicators, Four Perspectives
This type of benchmark usually reports four metrics. Each answers a different question.
| Metric | Definition | Question it answers | Usage notes |
|---|---|---|---|
| Pass rate | The proportion of tasks that meet acceptance criteria end to end, judged as a binary outcome per task | Is the result reliable? | Strongly related to how strict the acceptance script is |
| Average completion level | The average proportion of the task objective that was achieved; measures how far the agent got | How far is the task from success? | Collection methods vary by framework, so cross-source comparison requires caution |
| Average elapsed time | The average wall-clock duration for a single task | Can the task fit into a release window? | Also determines the cost of rework |
| Cost per run | Total cost of running the entire task set | What is the cost per unit of output? | Must standardize whether failed tasks and retries are included |
2.1 Pass Rate: A Direct Measure of Delivery Certainty
Pass rate is the most business-relevant metric because it corresponds to “can this task be handed directly to the team?” However, it is a sparse metric. Most failed samples only provide a “failure” label and cannot tell you how far the agent was from success.
Pass rate is also highly dependent on the acceptance method. Checking only whether compilation succeeds can produce a very different number from running behavioral tests, static analysis, and specification reviews. When comparing pass rates from different sources, the first thing to check is whether the acceptance standards are comparable.
2.2 Average Completion Level: The Remaining Distance Between Input and Return
Average completion level measures the overall degree to which a task was achieved. It compensates for the sparseness of pass rate.
A high completion level means the agent has finished most of the work and is stuck at the last step. A low completion level may mean the agent did not even build the basic skeleton. It is more like a diagnostic metric:
- When average completion is much higher than pass rate, the main failure mode is “missing the last mile.” Human completion or retries may then push the task toward success.
- When the two metrics are close, failures often come from directional errors. Investing more time may not reverse the outcome.
The collection method also affects interpretation. Some implementations score task-defined checkpoints item by item. Others use an evaluation model to score the final artifact. In either case, absolute values should not be directly compared across sources. Instead, focus on trends within the same evaluation system.
2.3 Average Elapsed Time: Deciding Whether a Task Can Be Scheduled
Elapsed time is often underestimated as a metric. A single task may take more than ten hours or even more than a day. This means it is difficult to use in an interactive setting with human monitoring, and it is more suitable for overnight batch jobs, scheduled integration tasks, or asynchronous transformation tasks.
When teams schedule work, they should estimate P50 and P95 elapsed time rather than relying on the average. A few extremely slow tasks can multiply the completion time of an entire batch. This also requires task scheduling to support timeout control, checkpoint recovery, and failure isolation.
2.4 Cost per Run: Only a Consistent Scope Makes Comparisons Meaningful
Cost per run is the cumulative cost of running the entire benchmark task set. It includes model inference, agent framework overhead, and resource consumption for builds and tests inside sandboxes.
When comparing costs, the scope must be consistent:
- Whether failed tasks are included;
- Whether retries are allowed, and if so, what the retry limit is;
- Whether optimizations such as inference caching and parallel execution are enabled.
If the scope is not consistent, the cost numbers are not comparable.
2.5 Metrics Need to Be Converted and Correlated
Raw metrics only describe “how much it costs to run one evaluation round.” For team budgeting, the more relevant question is “how much does it cost on average to successfully complete one task?” The conversion is:
cost per successful task = cost per run ÷ pass rate
Elapsed time and completion level should also be interpreted together:
- High completion and high elapsed time usually mean the agent is making progress, but slowly.
- Low elapsed time and low completion level usually mean the agent stopped early, or was limited by turn count, budget limits, or other constraints.
Only by putting the four metrics together can a team judge what kind of work a configuration is suitable for.
3. Empirical Comparison: Engineering Profiles of Two Typical Configurations
The public benchmark includes a typical comparison. The values are listed below and can be abstracted into two engineering profiles. The data are used only to illustrate the relationships among metrics and do not represent the capabilities of any specific product or model.
| Metric | Configuration A: longer elapsed time, higher completion | Configuration B: lightweight, faster, lower cost |
|---|---|---|
| Task set size | 30 long-horizon tasks | 30 long-horizon tasks |
| Pass rate | 24.0% | 17.3% |
| Average completion level | 82.3% | 71.2% |
| Average elapsed time | 27.4 hours | 12.0 hours |
| Cost per run | US$317.4, approximately CNY 2,100 | US$9.9, approximately CNY 66 |
Four conclusions can be drawn from the data.
First, cost and pass rate are not linearly related. Configuration A’s pass rate is only 6.7 percentage points higher than Configuration B’s, which is a relative improvement of about 39%. However, its cost is about 32 times that of Configuration B. Converting to cost per successful task, Configuration A costs about US$1,322, calculated as 317.4 ÷ 24.0%, while Configuration B costs about US$57.2, calculated as 9.9 ÷ 17.3%. The difference is about 23 times. This shows that on general long-horizon tasks, simply increasing investment yields limited marginal returns.
Second, neither configuration reaches the level of “fully managed” work. Even Configuration A, which performs better, still fails to pass acceptance for about three quarters of the tasks. If an agent is treated as an engineering resource that can be fully delegated to, no current configuration qualifies. A more realistic positioning is a “high-intensity assistant” that handles standardized repetitive work while humans remain responsible for review and decision-making.
Third, the speed difference and the cost difference are not synchronized. Configuration A takes about 2.3 times as long as Configuration B, but costs about 32 times as much. The reason is that in long-horizon tasks, a single large-model inference call is only one part of the total cost. Tool calls, sandbox execution, and the runtime of builds and tests often account for a significant share. Therefore, when evaluating cost, it is necessary to cover the entire agent runtime process. Looking only at inference cost can severely underestimate actual expenses.
Fourth, the two configurations correspond to different scheduling strategies. A 12-hour average elapsed time can fit into an overnight window. A 27.4-hour average spans an entire day and is more suitable for batch tasks tracked on a daily basis. Choosing between them is essentially answering whether the team needs certainty more, or throughput and low cost more.
It is worth emphasizing that these data do not mean “more expensive is always better” or “cheaper is always more cost-effective” for all scenarios. Real engineering decisions should be mapped onto the following task-selection table.
4. Selection: Match the Configuration to the Task Type
| Task type | Core requirement | Recommended strategy |
|---|---|---|
| Interactive completion, single-file refactoring, script writing | Fast response, low trial-and-error cost | Lightweight, high-speed, low-cost configuration with human intervention in real time |
| Prototype validation, one-off feature requests | Speed and cost | Use the fast configuration as the primary option, with quick human correction |
| Version upgrades, framework migrations, cross-platform conversion, legacy code refactoring | Completion level first, time can be asynchronous | Long-running, high-completion configuration, scheduled for overnight or weekend batch runs |
| Batch similar modifications, such as uniformly replacing deprecated APIs | Controllable unit cost, controllable exceptions | Fast configuration for batch execution; escalate exception samples to the high-completion configuration |
| Online defect localization and repair | Completion level first, needs explainability | High-completion configuration, with sufficient logs and human review |
| Tight deadlines and clear cost limits | Controllable certainty | Set hard budgets and timeouts; downgrade or terminate based on failure conditions |
In scenarios where budgets are sufficient and task value is high, enterprises can also adopt a layered strategy of “try cheaply first, then invest selectively”:
- Initial screening: Use a low-cost configuration to try multiple approaches in parallel and quickly eliminate clearly misguided ideas.
- Refinement: Only pass candidates with higher completion levels from the initial screening to the high-completion configuration for further work.
- Acceptance: Run unified acceptance scripts and human reviews on the final artifacts.
This can increase the success probability of high-value tasks without significantly raising total costs.
5. Build Your Own Evaluation: Ground Selection Decisions in Your Own Codebase
Public rankings can provide horizontal references, but they cannot replace evaluation based on your own team. Task distribution, coding style, and domain knowledge can significantly affect the actual performance of an agent. The following six steps can be used to build a lightweight benchmark.
5.1 Task Set: Close to the Real Workflow
Extract 20–30 real tasks from recent tickets, technical debt lists, and iteration plans. Cover the following categories:
- Application creation and scaffolding generation;
- Feature development and API integration;
- Defect fixing and exception handling;
- Code migration and version upgrades;
- Cross-platform adaptation.
Each task should contain four parts: a natural-language requirement description, an initial repository snapshot, an automated acceptance script, such as hidden tests or end-to-end checks, and agreed delivery boundaries, such as which directories may be modified.
5.2 Environment: Reproducible and Isolated
Preinstall build toolchains in containers and cache dependencies so that agents do not spend excessive time downloading dependencies. Each task should use an independent sandbox instance. After a task finishes, retain complete logs and the final code diff for later review.
5.3 Variables: Fix First, Then Compare
In the same evaluation round, keep the following elements unchanged: model version, agent framework version, maximum number of turns, timeout, budget limit, and inference parameters. Also record configuration snapshots for each version so that results are traceable and reproducible.
5.4 Metrics: Combine Binary Judgment with Completion Levels
- Pass/fail determination: Use the exit code of the acceptance script. For critical tasks, add human checks to prevent agents from modifying tests merely to make them pass.
- Completion level buckets for quickly locating bottlenecks:
L0: no effective changeL1: changes can compile, but the requirement is not implementedL2: main requirements are implemented, but some tests failL3: all acceptance tests passL4: passes acceptance and code quality and maintainability meet review requirements
L3 corresponds to “pass” in the automatic acceptance sense. L4 corresponds to code that can be merged directly into the main branch.
5.5 Failure Attribution: Accumulate Lessons by Category
Classify and count failed samples. Common categories include:
- Misunderstanding requirements;
- Missing domain knowledge;
- Build or test environment anomalies;
- Missing changes in related modules;
- Interruption due to timeout or budget limits.
Attribution results can be converted directly into improvement actions. If “environment anomalies” account for a high proportion, improving the sandbox image should be a priority. If “misunderstanding requirements” is common, more context should be added to requirement descriptions and repository convention documents.
5.6 Sample Code: Minimal Evaluation Framework
The following code provides a minimal evaluation framework skeleton. It shows how to collect metrics, record partial completion, and set budget and timeout limits inside the evaluation function.
import timefrom dataclasses import dataclass, field@dataclassclass Task:task_id: strworkspace: str # Path to the initial repository snapshotprompt: str # Requirement descriptionverify_cmd: str # Acceptance scriptbudget_usd: float = 2.0 # Budget limit for a single tasktimeout_sec: int = 7200 # Timeout limit for a single task@dataclassclass Result:task_id: strlevel: str # L0 to L4elapsed_sec: floatcost_usd: floatfailure_tags: list = field(default_factory=list)@propertydef passed(self) -> bool:# L3 means automatic acceptance passes; L4 means passes and is mergeablereturn self.level in ("L3", "L4")def run_task(task: Task, agent) -> Result:start = time.monotonic()try:# Optional: use a lighter model for early screening when the budget is tightagent.execute(workspace=task.workspace,prompt=task.prompt,budget_usd=task.budget_usd,timeout_sec=task.timeout_sec,)cost = agent.total_cost_usd()# Check syntax first, then run the acceptance scriptif not agent.syntax_ok():return Result(task.task_id, "L0", time.monotonic() - start, cost)rc = agent.run(task.verify_cmd)if rc != 0:return Result(task.task_id, "L2", time.monotonic() - start, cost)# Manual review result: pass or revisereview = "pass" if manual_review(task) else "revise"level = "L4" if review == "pass" else "L3"return Result(task.task_id, level, time.monotonic() - start, cost)except (BudgetExceeded, TaskTimeout) as e:elapsed = time.monotonic() - startcost = getattr(e, "cost_usd", agent.total_cost_usd())return Result(task.task_id, "L1", elapsed, cost, [type(e).__name__])def summarize(results):n = len(results)if n == 0:return {}total_cost = sum(r.cost_usd for r in results)passes = sum(r.passed for r in results)times = sorted(r.elapsed_sec for r in results)return {"pass_rate": round(passes / n, 4),"avg_cost_usd": round(total_cost / n, 2),"cost_per_pass_usd": round(total_cost / passes, 2) if passes else None,"p50_sec": round(times[n // 2], 1),"p95_sec": round(times[min(n - 1, int(n * 0.95))], 1),}
5.7 Statistical Analysis: Interpret Small Samples Carefully
Returning to the public benchmark data, one evaluated combination achieved a 28.7% pass rate over 30 tasks, while Configuration A achieved 24.0%. The difference is 4.7 percentage points. However, at a sample size of 30 tasks, the standard error caused by statistical fluctuation alone is approximately:
√[0.24 × 0.76 ÷ 30 + 0.287 × 0.713 ÷ 30] ≈ 0.113
That is, about 11 percentage points, which is much larger than the gap between the two results. In other words, based only on this round of results, one cannot statistically conclude that their true performance levels are different. This gives teams two lessons:
- Focus on intervals rather than point values: When reporting pass rate, provide a confidence interval, for example “24.0% (95% interval approximately 11% to 43%).”
- Be cautious when comparing across rounds: If each evaluation round uses different task samples, differences between two results may mainly come from changes in task difficulty rather than changes in capability.
6. Engineering Practice: Turning Evaluation Findings into the Daily Workflow
6.1 Task Scheduling and Batch Processing
Long-horizon tasks are naturally suited to asynchronous execution. Container platforms can be used to orchestrate sandbox tasks. Message queues can distribute tasks and collect results. Failed tasks can be moved to a retry queue. Object storage can preserve each task’s initial snapshot, code diff, and runtime logs for auditing and retrospectives. For example, Baidu AI Cloud provides general cloud capabilities such as object storage, message queuing, container platforms, logging services, and monitoring and alerting. Teams can use such capabilities to build their own agent evaluation and batch-processing platforms.
At the scheduling level, the following points also require attention:
- Timeout and retry: Set timeout and retry limits for each task to prevent a few abnormal tasks from dragging down the entire batch.
- Checkpoint recovery: Long-running tasks should support recovery from the most recent checkpoint instead of starting over every time.
- Concurrency limiting: Control the number of sandboxes running at the same time to avoid builds and tests competing for resources.
- Avoiding peak hours: Schedule large-scale batch transformations during off-peak periods to reduce impact on shared build resources.
6.2 Observability
Every tool call, command execution, and model interaction of an agent should be recorded in a structured way to form a complete trace from the requirement to the final artifact. Common practices include:
- Full-chain tracing: Connect stages by task ID to locate latency bottlenecks, such as whether time is spent on dependency downloads or test execution.
- Token consumption statistics: Measure input and output tokens by task and by stage to identify where context expansion is severe.
- Monitoring and alerting: Set alerts for task backlogs, abnormal increases in failure rate, and single-task costs exceeding thresholds. This helps detect abnormal behavior such as an agent getting stuck in a loop.
6.3 Code Quality Gates
If average completion level is only about 80%, a considerable proportion of artifacts will require human completion or rewriting. Agent output must enter the same workflow as human-written code. Code review, static analysis, unit tests, integration tests, and security scanning should all be required. Setting quality gates in continuous integration can effectively prevent changes that “look completed but introduce defects” from entering the main branch.
6.4 Security and Permissions
Agents need to read and write files in repositories and execute commands, so permission boundaries must be tightened:
- Least privilege: Grant only the directory and command permissions required for the current task.
- Secret isolation: Sensitive configuration should be injected by an external management system and should not enter the agent’s context, avoiding exposure in model output or logs.
- Artifact review: Check the source and license of third-party dependencies introduced by the agent to prevent supply-chain risks.
6.5 Data Security and Compliance
Before sending code to an external model service, teams should assess whether it involves core algorithms, customer data, or other sensitive information. For sensitive repositories, private deployment or solutions that keep data within a defined boundary can be adopted. Compliance requirements can be met through desensitization, access control, and audit logs.
6.6 Cost Governance
- Set a budget limit for each task. When the limit is exceeded, automatically downgrade to a more economical configuration or terminate the task.
- Allocate costs by team, project, and task type. Regularly review changes in cost per successful task.
- Identify high-cost tasks and promote refactoring or splitting to reduce the agent’s workload at the source.
7. Conclusion and Selection Checklist
Evaluating coding agents for long-horizon engineering tasks is essentially a decision problem under the triangle of quality, time, and cost. Pass rate measures delivery certainty. Average completion level reveals how far the task is from success. Elapsed time determines whether a task can fit into a schedule. Cost determines whether the approach can scale.
Configuration A costs about 32 times as much to run and takes about 2.3 times as long as Configuration B, but only improves pass rate by 6.7 percentage points. This suggests that on general long-horizon tasks, the bottleneck of coding agents is mainly not “how much is invested,” but whether task understanding, environment adaptation, and the acceptance loop are complete.
- For individual developers: Treat the agent as a high-intensity assistant. Let it handle standardized work such as scaffolding generation, batch transformations, and test completion. Spend your energy on requirement clarification, architecture design, and quality control.
- For R&D teams: First build your own evaluation task set, then distribute configurations by task type, and combine them with quality gates, observability, and cost governance. Only then can agent capabilities be transformed into measurable engineering value.
Before introducing coding agents, teams should perform a final check with the following questions:
- Does the task have clear, automatable acceptance criteria?
- Can the sandbox environment reproduce the real build and test process?
- Do failed tasks have timeout and budget limits, and can they be effectively isolated?
- Must artifacts go through code review and automated gates before being merged?
- Are appropriate configurations selected by task type instead of using a one-size-fits-all approach?
- Are the security boundaries of code and data clear, and is there a risk of sensitive information leakage?
- Has a continuously updatable internal benchmark been established for regular retesting and continuous tuning?
By answering these seven questions clearly, teams can avoid blindly following ranking numbers while also avoiding underestimating the value of this technology in real engineering, even as coding tools iterate rapidly.
