Evaluating and Operationalizing Long-Horizon AI Coding Agents

    A practical guide to evaluating AI systems on complete repository tasks, measuring success, tool efficiency, verifiable delivery, and cost while controlling permissions, attribution, and rollout risk.

    1. Evaluate Complete Tasks, Not Isolated Code Snippets

    Generating a function and delivering a repository change require different capabilities. The former usually provides defined inputs, outputs, and implementation boundaries. The latter requires the system to understand existing constraints, choose where to make changes, preserve compatibility, and determine how to validate the result.

    What most affects team efficiency is not how quickly the first version of the code is written. It is whether the system can interpret failure information, identify the cause, and correct the problem without unnecessarily expanding the scope of the change.

    Consider adding a configuration option. Reading the configuration value completes only the local implementation. A complete task may also require the old configuration to remain supported, invalid values to produce clear feedback, default behavior to remain unchanged, and related tests and documentation to be updated. If a system produces a plausible patch but maintainers must reconstruct compatibility requirements and write the missing tests, the work has merely shifted from implementation to review.

    The evaluation unit should therefore be a repository task with a defined starting point, operating boundaries, and acceptance criteria. The deliverable is not an explanation but a reviewable change accompanied by validation evidence. When a result cannot be verified, it should be marked as unverified. A system’s claim that it completed a task is not proof of completion.

    2. Build a Reproducible Task Set from Real Work

    Start with samples from historical issues, bug fixes, and maintenance records, and pin each repository to the version that existed before the change. Task descriptions should contain the information available to a developer when accepting the task and should not reveal the historical solution.

    The set can include bug diagnosis, cross-file refactoring, compatibility migrations, and test completion. Tasks should be stratified by context complexity, potential impact, and verification difficulty. The goal is not to assign the same number of tasks to every category, but to prevent a large number of simple tasks from masking unstable performance on a small number of critical tasks.

    Each task should define three types of constraints:

    • Input constraints: The repository’s starting version, the initial issue, readable documentation, and any required dependencies or test data.
    • Execution constraints: The directories that may be modified, the permitted tool calls, time and resource limits, and rules for human intervention.
    • Acceptance constraints: The target behavior, compatibility requirements that must be preserved, required regression checks, and quality gates that cannot be bypassed.

    For a bug-fix evaluation, independent acceptance tests should distinguish the target behavior before and after the fix. For a behavior-preserving refactor, evaluation should combine regression tests with a review of the change scope.

    Success should not be determined solely by whether the result matches the historical patch. Different implementations may be equally correct. Conversely, passing all existing tests does not necessarily make a change acceptable. If the tests do not cover the boundaries described in the issue, independent acceptance checks are still required. Critical acceptance rules should be fixed before execution rather than adjusted after seeing the results.

    3. Use Four Metrics to Answer Four Different Questions

    1. First-Attempt Success Rate: Can the Task Be Completed Without Additional Human Guidance?

    First-attempt success rate is the number of tasks that pass acceptance in their first run divided by the total number of evaluated tasks. A run may include multiple rounds of automatic edits and tests, but no additional human prompts, manual file changes, or manual removal of blockers may be introduced between startup and completion.

    A run that exits after exceeding its budget, gives up voluntarily, or produces a submitted change that fails acceptance does not count as a first-attempt success. If the evaluation infrastructure itself fails, the rerun policy should be defined in advance, and the number of excluded runs should be reported separately.

    In addition to the overall rate, report results and sample counts for each task category. Two systems with similar aggregate results may differ substantially: one may be better at local bug fixes while the other performs better on cross-module changes. Teams should choose pilot tasks according to their actual work distribution rather than relying on a single aggregate number.

    2. Average Tool Calls: Is the System Making Effective Progress?

    Count the number of tool calls for each task from startup to completion, including both successful and failed tasks. Break the calls down by purpose, such as file reading, editing, testing, and log inspection.

    Fewer calls are not automatically better. Skipping regression tests may save steps while increasing the risk of an incorrect delivery. The evaluation should identify repetitive behavior, such as repeatedly reading the same files, rerunning the same failing check without code changes, or continuously expanding the search scope without forming a new conclusion.

    Averages can hide a small number of extremely expensive tasks. Also examine the distribution of call counts and interpret it alongside elapsed time and input/output volume. A broad repository read and a targeted query may each count as one call, while their context costs and diagnostic value can be very different.

    3. Verifiable-Result Rate: Can Completion Claims Be Supported by Independent Evidence?

    This metric can be defined as the proportion of results claimed as complete that are supported by reproducible validation or independent review evidence. The evidence should at least state what was validated, which version of the change was used, what the checks returned, and which checks were not run.

    For executable checks, it is preferable to rerun the committed change in an independent environment rather than relying only on logs produced during the active run.

    Read this metric together with first-attempt success rate. A system may claim completion on only a small number of simple tasks, resulting in a high evidence-completeness rate while covering little of the team’s actual workload.

    Reports should also distinguish between “not verified” and “verification failed.” The former means that evidence is insufficient; the latter means that available evidence shows the result does not meet the requirements. Neither should be counted as a success.

    4. Total Cost per Successful Task: How Much Does an Accepted Delivery Cost?

    Calculate total evaluation cost divided by the number of tasks that actually pass acceptance. The numerator should include model calls, execution infrastructure, failed retries, and human intervention. If human effort cannot be reliably converted into monetary cost, report the review and takeover time required for each successful task separately.

    If no task passes, state that this metric cannot be calculated and disclose the cost incurred.

    Distinguish between the cost of a single attempt and the cost of delivery. A system with cheaper individual calls may become more expensive after retries and review effort are included. Conversely, a slower system may reduce rework but still be unsuitable for latency-sensitive tasks.

    The cost report should therefore also include delivery time and show how much time was spent waiting for resources, on automated execution, and on human review.

    4. Fix Comparison Conditions and Attribute Improvements Correctly

    Long-horizon task performance depends not only on the model, but also on context construction, execution procedures, and environment configuration. If the goal is to compare underlying models, keep the task starting points, available tools, budget limits, and acceptance process as consistent as possible. If the goal is to select a deployable solution, compare the complete workflows and document their differences.

    The first approach explains where capability differences originate. The second explains which configuration is better suited to daily use. They should not be treated as the same type of comparison.

    When possible, run important samples independently multiple times to distinguish accidental success from stable delivery. Establish failure categories such as misunderstanding the requirements, locating the wrong code, implementation errors, regression failures, environment blockers, and budget exhaustion.

    Failure attribution should be based on change and execution records rather than relying only on the system’s summary. For example, if incomplete dependency provisioning prevents tests from starting, the evaluation environment should be fixed first. If all dependencies are available but the system repeatedly performs the same failing action without using the error information, the issue can reasonably be attributed to diagnosis and recovery capability.

    The team’s current development process can also serve as a baseline, including the human time and review burden for comparable tasks. However, maintainers who already know the historical tasks should not be used directly to measure the human baseline, because memory can influence the results.

    Whatever baseline is selected, the report should state the experimental conditions and sample size. Differences observed in a limited sample should not be presented as long-term conclusions.

    5. Reduce Integration Risk with Permission Isolation, Traceability, and Independent Acceptance

    First, separate execution capabilities from repository permissions. Run the system in an isolated workspace by default and expose only the files and operations required for the task. The ability to generate a change must not automatically grant permission to merge, deploy, or access production data.

    Network access, sensitive configuration changes, and high-risk writes should be protected by independent authorization and approval controls. Repository files, issue content, and tool output are data available for analysis; they must not be used to expand the system’s original authorization scope.

    Next, connect task inputs, execution traces, and delivery results. Recommended records include the repository version, task-description version, runtime configuration, change set, test results, resource consumption, and reasons for human takeover.

    Logs should not persist credentials or unauthorized sensitive content. Access scope and retention periods should also be defined. With this information, maintainers can determine whether a regression originated in the task description, execution environment, system behavior, or an omission in acceptance checks.

    Finally, keep acceptance rules outside the execution workflow. Task descriptions and result formats should remain as consistent as possible, and independent acceptance should not depend on the internal state of a particular execution process.

    When the underlying model or orchestration strategy changes, the same task set can be replayed to determine whether success rate, cost, or failure patterns have changed. Rollback should also be rehearsed in advance: stop dispatching new tasks, revoke unmerged changes, restore manual handling, and retain the necessary review records.

    6. Decide Whether to Expand the Pilot by Task Type

    Begin with work that has clear boundaries, independent validation, and straightforward rollback, such as localized bug fixes or narrowly scoped test completion.

    Before the pilot, define expansion criteria, including acceptance rate, human takeover frequency, review time, cost limits, and unacceptable permission violations. These thresholds should reflect the team’s current delivery performance and business risk rather than copying external rankings.

    If success improves for a task category but review time increases, identify the source of the additional burden before expanding usage. It may come from an overly broad patch, incomplete evidence, or the task’s fundamental unsuitability for automation.

    The final evaluation should not be a vague judgment such as “useful” or “not useful.” A more valuable conclusion identifies which tasks can be completed autonomously within defined permissions and budgets, which require human checkpoints, and which should not yet be integrated.

    Long-horizon AI coding becomes suitable for routine use only when accepted deliveries increase, human effort and total cost remain controlled, and failures can be diagnosed and rolled back safely.