Evaluating and Selecting AI Coding Agents for Long-Horizon Engineering Tasks

    This article examines how to evaluate AI coding agents for real engineering work, where success depends on completing multi-session, multi-step tasks rather than generating a single correct function. It explains four core metrics—pass rate, average completion level, average elapsed time, and cost per run—how to interpret them together, and how to compare configurations using a public benchmark sample. It also provides practical guidance for task scheduling, observability, code quality gates, security, data compliance, cost governance, and building an internal benchmark. Finally, it offers a selection checklist for teams evaluating whether and how to deploy coding agents.

    1. Evaluation Has Shifted from Writing a Correct Code Snippet to Completing an Engineering Task

    At this stage of code-generation development, producing a correct single function is no longer the main bottleneck. What developers care about more is whether an AI coding agent can independently plan, iterate, and deliver an acceptable change when given a complete engineering request. Such a request may involve building an application from scratch, adding a new feature, migrating an old implementation to a new framework, or fixing a defect that spans multiple files.

    The typical working loop of a coding agent is “perceive—plan—act—observe.” The agent first reads the repository and the requirement, then breaks down the steps and generates modifications. It then executes commands such as builds and tests, observes errors and logs, and decides the next step. In this loop, a task may require dozens of interactions and hours or even longer to complete. This kind of work is commonly called a long-horizon task.

    Long-horizon tasks are fundamentally different from single-turn code completion:

    • Context spans multiple sessions. The agent needs to remember earlier decisions, avoid conflicting changes, and prevent later modifications from undermining earlier work.
    • Results are hard to judge in a binary way. The fact that code compiles or tests pass does not necessarily mean the requirement has been correctly implemented. Whether the architecture is reasonable and edge cases are covered still requires human judgment.
    • Cost is not limited to model inference. Agents repeatedly build, run tests, and collect logs inside sandboxes. This time cost grows non-linearly with task complexity.
    • Failure is normal, not exceptional. Ambiguous requirements, implicit conventions, and environment differences can all cause a task to drift off course.

    As a result, evaluation for real engineering tasks is shifting from “whether a generated snippet is correct” to “whether the quality of the result is balanced against the cost of obtaining it in a long-horizon task.”

    The analysis below uses a recent public coding benchmark as an observation sample. The benchmark covers tasks such as application creation, feature development, code migration, and application conversion, and it evaluates a model together with the agent runtime framework that runs it. This choice of evaluation target is itself informative: the same model can perform very differently under different agent frameworks. “Selecting the model” and “selecting the agent framework” must be considered together.

    2. Metrics: Four Indicators, Four Perspectives

    This type of benchmark usually reports four metrics. Each answers a different question.

    Metric Definition Question it answers Usage notes
    Pass rate The proportion of tasks that meet acceptance criteria end to end, judged as a binary outcome per task Is the result reliable? Strongly related to how strict the acceptance script is
    Average completion level The average proportion of the task objective that was achieved; measures how far the agent got How far is the task from success? Collection methods vary by framework, so cross-source comparison requires caution
    Average elapsed time The average wall-clock duration for a single task Can the task fit into a release window? Also determines the cost of rework
    Cost per run Total cost of running the entire task set What is the cost per unit of output? Must standardize whether failed tasks and retries are included

    2.1 Pass Rate: A Direct Measure of Delivery Certainty

    Pass rate is the most business-relevant metric because it corresponds to “can this task be handed directly to the team?” However, it is a sparse metric. Most failed samples only provide a “failure” label and cannot tell you how far the agent was from success.

    Pass rate is also highly dependent on the acceptance method. Checking only whether compilation succeeds can produce a very different number from running behavioral tests, static analysis, and specification reviews. When comparing pass rates from different sources, the first thing to check is whether the acceptance standards are comparable.

    2.2 Average Completion Level: The Remaining Distance Between Input and Return

    Average completion level measures the overall degree to which a task was achieved. It compensates for the sparseness of pass rate.

    A high completion level means the agent has finished most of the work and is stuck at the last step. A low completion level may mean the agent did not even build the basic skeleton. It is more like a diagnostic metric:

    • When average completion is much higher than pass rate, the main failure mode is “missing the last mile.” Human completion or retries may then push the task toward success.
    • When the two metrics are close, failures often come from directional errors. Investing more time may not reverse the outcome.

    The collection method also affects interpretation. Some implementations score task-defined checkpoints item by item. Others use an evaluation model to score the final artifact. In either case, absolute values should not be directly compared across sources. Instead, focus on trends within the same evaluation system.

    2.3 Average Elapsed Time: Deciding Whether a Task Can Be Scheduled

    Elapsed time is often underestimated as a metric. A single task may take more than ten hours or even more than a day. This means it is difficult to use in an interactive setting with human monitoring, and it is more suitable for overnight batch jobs, scheduled integration tasks, or asynchronous transformation tasks.

    When teams schedule work, they should estimate P50 and P95 elapsed time rather than relying on the average. A few extremely slow tasks can multiply the completion time of an entire batch. This also requires task scheduling to support timeout control, checkpoint recovery, and failure isolation.

    2.4 Cost per Run: Only a Consistent Scope Makes Comparisons Meaningful

    Cost per run is the cumulative cost of running the entire benchmark task set. It includes model inference, agent framework overhead, and resource consumption for builds and tests inside sandboxes.

    When comparing costs, the scope must be consistent:

    • Whether failed tasks are included;
    • Whether retries are allowed, and if so, what the retry limit is;
    • Whether optimizations such as inference caching and parallel execution are enabled.

    If the scope is not consistent, the cost numbers are not comparable.

    2.5 Metrics Need to Be Converted and Correlated

    Raw metrics only describe “how much it costs to run one evaluation round.” For team budgeting, the more relevant question is “how much does it cost on average to successfully complete one task?” The conversion is:

    1. cost per successful task = cost per run ÷ pass rate

    Elapsed time and completion level should also be interpreted together:

    • High completion and high elapsed time usually mean the agent is making progress, but slowly.
    • Low elapsed time and low completion level usually mean the agent stopped early, or was limited by turn count, budget limits, or other constraints.

    Only by putting the four metrics together can a team judge what kind of work a configuration is suitable for.

    3. Empirical Comparison: Engineering Profiles of Two Typical Configurations

    The public benchmark includes a typical comparison. The values are listed below and can be abstracted into two engineering profiles. The data are used only to illustrate the relationships among metrics and do not represent the capabilities of any specific product or model.

    Metric Configuration A: longer elapsed time, higher completion Configuration B: lightweight, faster, lower cost
    Task set size 30 long-horizon tasks 30 long-horizon tasks
    Pass rate 24.0% 17.3%
    Average completion level 82.3% 71.2%
    Average elapsed time 27.4 hours 12.0 hours
    Cost per run US$317.4, approximately CNY 2,100 US$9.9, approximately CNY 66

    Four conclusions can be drawn from the data.

    First, cost and pass rate are not linearly related. Configuration A’s pass rate is only 6.7 percentage points higher than Configuration B’s, which is a relative improvement of about 39%. However, its cost is about 32 times that of Configuration B. Converting to cost per successful task, Configuration A costs about US$1,322, calculated as 317.4 ÷ 24.0%, while Configuration B costs about US$57.2, calculated as 9.9 ÷ 17.3%. The difference is about 23 times. This shows that on general long-horizon tasks, simply increasing investment yields limited marginal returns.

    Second, neither configuration reaches the level of “fully managed” work. Even Configuration A, which performs better, still fails to pass acceptance for about three quarters of the tasks. If an agent is treated as an engineering resource that can be fully delegated to, no current configuration qualifies. A more realistic positioning is a “high-intensity assistant” that handles standardized repetitive work while humans remain responsible for review and decision-making.

    Third, the speed difference and the cost difference are not synchronized. Configuration A takes about 2.3 times as long as Configuration B, but costs about 32 times as much. The reason is that in long-horizon tasks, a single large-model inference call is only one part of the total cost. Tool calls, sandbox execution, and the runtime of builds and tests often account for a significant share. Therefore, when evaluating cost, it is necessary to cover the entire agent runtime process. Looking only at inference cost can severely underestimate actual expenses.

    Fourth, the two configurations correspond to different scheduling strategies. A 12-hour average elapsed time can fit into an overnight window. A 27.4-hour average spans an entire day and is more suitable for batch tasks tracked on a daily basis. Choosing between them is essentially answering whether the team needs certainty more, or throughput and low cost more.

    It is worth emphasizing that these data do not mean “more expensive is always better” or “cheaper is always more cost-effective” for all scenarios. Real engineering decisions should be mapped onto the following task-selection table.

    4. Selection: Match the Configuration to the Task Type

    Task type Core requirement Recommended strategy
    Interactive completion, single-file refactoring, script writing Fast response, low trial-and-error cost Lightweight, high-speed, low-cost configuration with human intervention in real time
    Prototype validation, one-off feature requests Speed and cost Use the fast configuration as the primary option, with quick human correction
    Version upgrades, framework migrations, cross-platform conversion, legacy code refactoring Completion level first, time can be asynchronous Long-running, high-completion configuration, scheduled for overnight or weekend batch runs
    Batch similar modifications, such as uniformly replacing deprecated APIs Controllable unit cost, controllable exceptions Fast configuration for batch execution; escalate exception samples to the high-completion configuration
    Online defect localization and repair Completion level first, needs explainability High-completion configuration, with sufficient logs and human review
    Tight deadlines and clear cost limits Controllable certainty Set hard budgets and timeouts; downgrade or terminate based on failure conditions

    In scenarios where budgets are sufficient and task value is high, enterprises can also adopt a layered strategy of “try cheaply first, then invest selectively”:

    1. Initial screening: Use a low-cost configuration to try multiple approaches in parallel and quickly eliminate clearly misguided ideas.
    2. Refinement: Only pass candidates with higher completion levels from the initial screening to the high-completion configuration for further work.
    3. Acceptance: Run unified acceptance scripts and human reviews on the final artifacts.

    This can increase the success probability of high-value tasks without significantly raising total costs.

    5. Build Your Own Evaluation: Ground Selection Decisions in Your Own Codebase

    Public rankings can provide horizontal references, but they cannot replace evaluation based on your own team. Task distribution, coding style, and domain knowledge can significantly affect the actual performance of an agent. The following six steps can be used to build a lightweight benchmark.

    5.1 Task Set: Close to the Real Workflow

    Extract 20–30 real tasks from recent tickets, technical debt lists, and iteration plans. Cover the following categories:

    • Application creation and scaffolding generation;
    • Feature development and API integration;
    • Defect fixing and exception handling;
    • Code migration and version upgrades;
    • Cross-platform adaptation.

    Each task should contain four parts: a natural-language requirement description, an initial repository snapshot, an automated acceptance script, such as hidden tests or end-to-end checks, and agreed delivery boundaries, such as which directories may be modified.

    5.2 Environment: Reproducible and Isolated

    Preinstall build toolchains in containers and cache dependencies so that agents do not spend excessive time downloading dependencies. Each task should use an independent sandbox instance. After a task finishes, retain complete logs and the final code diff for later review.

    5.3 Variables: Fix First, Then Compare

    In the same evaluation round, keep the following elements unchanged: model version, agent framework version, maximum number of turns, timeout, budget limit, and inference parameters. Also record configuration snapshots for each version so that results are traceable and reproducible.

    5.4 Metrics: Combine Binary Judgment with Completion Levels

    • Pass/fail determination: Use the exit code of the acceptance script. For critical tasks, add human checks to prevent agents from modifying tests merely to make them pass.
    • Completion level buckets for quickly locating bottlenecks:
    1. L0: no effective change
    2. L1: changes can compile, but the requirement is not implemented
    3. L2: main requirements are implemented, but some tests fail
    4. L3: all acceptance tests pass
    5. L4: passes acceptance and code quality and maintainability meet review requirements

    L3 corresponds to “pass” in the automatic acceptance sense. L4 corresponds to code that can be merged directly into the main branch.

    5.5 Failure Attribution: Accumulate Lessons by Category

    Classify and count failed samples. Common categories include:

    • Misunderstanding requirements;
    • Missing domain knowledge;
    • Build or test environment anomalies;
    • Missing changes in related modules;
    • Interruption due to timeout or budget limits.

    Attribution results can be converted directly into improvement actions. If “environment anomalies” account for a high proportion, improving the sandbox image should be a priority. If “misunderstanding requirements” is common, more context should be added to requirement descriptions and repository convention documents.

    5.6 Sample Code: Minimal Evaluation Framework

    The following code provides a minimal evaluation framework skeleton. It shows how to collect metrics, record partial completion, and set budget and timeout limits inside the evaluation function.

    1. import time
    2. from dataclasses import dataclass, field
    3. @dataclass
    4. class Task:
    5. task_id: str
    6. workspace: str # Path to the initial repository snapshot
    7. prompt: str # Requirement description
    8. verify_cmd: str # Acceptance script
    9. budget_usd: float = 2.0 # Budget limit for a single task
    10. timeout_sec: int = 7200 # Timeout limit for a single task
    11. @dataclass
    12. class Result:
    13. task_id: str
    14. level: str # L0 to L4
    15. elapsed_sec: float
    16. cost_usd: float
    17. failure_tags: list = field(default_factory=list)
    18. @property
    19. def passed(self) -> bool:
    20. # L3 means automatic acceptance passes; L4 means passes and is mergeable
    21. return self.level in ("L3", "L4")
    22. def run_task(task: Task, agent) -> Result:
    23. start = time.monotonic()
    24. try:
    25. # Optional: use a lighter model for early screening when the budget is tight
    26. agent.execute(
    27. workspace=task.workspace,
    28. prompt=task.prompt,
    29. budget_usd=task.budget_usd,
    30. timeout_sec=task.timeout_sec,
    31. )
    32. cost = agent.total_cost_usd()
    33. # Check syntax first, then run the acceptance script
    34. if not agent.syntax_ok():
    35. return Result(task.task_id, "L0", time.monotonic() - start, cost)
    36. rc = agent.run(task.verify_cmd)
    37. if rc != 0:
    38. return Result(task.task_id, "L2", time.monotonic() - start, cost)
    39. # Manual review result: pass or revise
    40. review = "pass" if manual_review(task) else "revise"
    41. level = "L4" if review == "pass" else "L3"
    42. return Result(task.task_id, level, time.monotonic() - start, cost)
    43. except (BudgetExceeded, TaskTimeout) as e:
    44. elapsed = time.monotonic() - start
    45. cost = getattr(e, "cost_usd", agent.total_cost_usd())
    46. return Result(task.task_id, "L1", elapsed, cost, [type(e).__name__])
    47. def summarize(results):
    48. n = len(results)
    49. if n == 0:
    50. return {}
    51. total_cost = sum(r.cost_usd for r in results)
    52. passes = sum(r.passed for r in results)
    53. times = sorted(r.elapsed_sec for r in results)
    54. return {
    55. "pass_rate": round(passes / n, 4),
    56. "avg_cost_usd": round(total_cost / n, 2),
    57. "cost_per_pass_usd": round(total_cost / passes, 2) if passes else None,
    58. "p50_sec": round(times[n // 2], 1),
    59. "p95_sec": round(times[min(n - 1, int(n * 0.95))], 1),
    60. }

    5.7 Statistical Analysis: Interpret Small Samples Carefully

    Returning to the public benchmark data, one evaluated combination achieved a 28.7% pass rate over 30 tasks, while Configuration A achieved 24.0%. The difference is 4.7 percentage points. However, at a sample size of 30 tasks, the standard error caused by statistical fluctuation alone is approximately:

    1. √[0.24 × 0.76 ÷ 30 + 0.287 × 0.713 ÷ 30] ≈ 0.113

    That is, about 11 percentage points, which is much larger than the gap between the two results. In other words, based only on this round of results, one cannot statistically conclude that their true performance levels are different. This gives teams two lessons:

    • Focus on intervals rather than point values: When reporting pass rate, provide a confidence interval, for example “24.0% (95% interval approximately 11% to 43%).”
    • Be cautious when comparing across rounds: If each evaluation round uses different task samples, differences between two results may mainly come from changes in task difficulty rather than changes in capability.

    6. Engineering Practice: Turning Evaluation Findings into the Daily Workflow

    6.1 Task Scheduling and Batch Processing

    Long-horizon tasks are naturally suited to asynchronous execution. Container platforms can be used to orchestrate sandbox tasks. Message queues can distribute tasks and collect results. Failed tasks can be moved to a retry queue. Object storage can preserve each task’s initial snapshot, code diff, and runtime logs for auditing and retrospectives. For example, Baidu AI Cloud provides general cloud capabilities such as object storage, message queuing, container platforms, logging services, and monitoring and alerting. Teams can use such capabilities to build their own agent evaluation and batch-processing platforms.

    At the scheduling level, the following points also require attention:

    • Timeout and retry: Set timeout and retry limits for each task to prevent a few abnormal tasks from dragging down the entire batch.
    • Checkpoint recovery: Long-running tasks should support recovery from the most recent checkpoint instead of starting over every time.
    • Concurrency limiting: Control the number of sandboxes running at the same time to avoid builds and tests competing for resources.
    • Avoiding peak hours: Schedule large-scale batch transformations during off-peak periods to reduce impact on shared build resources.

    6.2 Observability

    Every tool call, command execution, and model interaction of an agent should be recorded in a structured way to form a complete trace from the requirement to the final artifact. Common practices include:

    • Full-chain tracing: Connect stages by task ID to locate latency bottlenecks, such as whether time is spent on dependency downloads or test execution.
    • Token consumption statistics: Measure input and output tokens by task and by stage to identify where context expansion is severe.
    • Monitoring and alerting: Set alerts for task backlogs, abnormal increases in failure rate, and single-task costs exceeding thresholds. This helps detect abnormal behavior such as an agent getting stuck in a loop.

    6.3 Code Quality Gates

    If average completion level is only about 80%, a considerable proportion of artifacts will require human completion or rewriting. Agent output must enter the same workflow as human-written code. Code review, static analysis, unit tests, integration tests, and security scanning should all be required. Setting quality gates in continuous integration can effectively prevent changes that “look completed but introduce defects” from entering the main branch.

    6.4 Security and Permissions

    Agents need to read and write files in repositories and execute commands, so permission boundaries must be tightened:

    • Least privilege: Grant only the directory and command permissions required for the current task.
    • Secret isolation: Sensitive configuration should be injected by an external management system and should not enter the agent’s context, avoiding exposure in model output or logs.
    • Artifact review: Check the source and license of third-party dependencies introduced by the agent to prevent supply-chain risks.

    6.5 Data Security and Compliance

    Before sending code to an external model service, teams should assess whether it involves core algorithms, customer data, or other sensitive information. For sensitive repositories, private deployment or solutions that keep data within a defined boundary can be adopted. Compliance requirements can be met through desensitization, access control, and audit logs.

    6.6 Cost Governance

    • Set a budget limit for each task. When the limit is exceeded, automatically downgrade to a more economical configuration or terminate the task.
    • Allocate costs by team, project, and task type. Regularly review changes in cost per successful task.
    • Identify high-cost tasks and promote refactoring or splitting to reduce the agent’s workload at the source.

    7. Conclusion and Selection Checklist

    Evaluating coding agents for long-horizon engineering tasks is essentially a decision problem under the triangle of quality, time, and cost. Pass rate measures delivery certainty. Average completion level reveals how far the task is from success. Elapsed time determines whether a task can fit into a schedule. Cost determines whether the approach can scale.

    Configuration A costs about 32 times as much to run and takes about 2.3 times as long as Configuration B, but only improves pass rate by 6.7 percentage points. This suggests that on general long-horizon tasks, the bottleneck of coding agents is mainly not “how much is invested,” but whether task understanding, environment adaptation, and the acceptance loop are complete.

    • For individual developers: Treat the agent as a high-intensity assistant. Let it handle standardized work such as scaffolding generation, batch transformations, and test completion. Spend your energy on requirement clarification, architecture design, and quality control.
    • For R&D teams: First build your own evaluation task set, then distribute configurations by task type, and combine them with quality gates, observability, and cost governance. Only then can agent capabilities be transformed into measurable engineering value.

    Before introducing coding agents, teams should perform a final check with the following questions:

    1. Does the task have clear, automatable acceptance criteria?
    2. Can the sandbox environment reproduce the real build and test process?
    3. Do failed tasks have timeout and budget limits, and can they be effectively isolated?
    4. Must artifacts go through code review and automated gates before being merged?
    5. Are appropriate configurations selected by task type instead of using a one-size-fits-all approach?
    6. Are the security boundaries of code and data clear, and is there a risk of sensitive information leakage?
    7. Has a continuously updatable internal benchmark been established for regular retesting and continuous tuning?

    By answering these seven questions clearly, teams can avoid blindly following ranking numbers while also avoiding underestimating the value of this technology in real engineering, even as coding tools iterate rapidly.