Evaluating AI Coding Agents on Long-Horizon Tasks: Balancing Pass Rates and Cost

    This article explains how to evaluate AI coding agents on long-horizon engineering tasks—building app skeletons from an empty repo, adding end-to-end feature flows, migrating between technology stacks, and porting apps across platforms—where results are far harder to quantify than code completion. It defines and clarifies four core metrics (pass rate, average completion, average duration, and per-run cost), warns against common misreadings such as equating a low pass rate with weak capability, and derives decision-oriented metrics like cost per effective output. It then shows how to read a sample results table, how to build a reproducible evaluation harness with fixed images, declarative acceptance specs, and failure attribution, and how to translate metrics into tiered scheduling and graded review across low-, medium-, and high-risk tasks. It closes with observability and cost-governance practices and four principles for treating evaluation as a decision tool rather than a leaderboard.

    1. Why long-horizon tasks are the real test for coding agents

    Code completion has already been thoroughly validated; what is genuinely hard to quantify is long-horizon engineering work: creating a runnable application skeleton from an empty repo, adding a complete feature path to an existing codebase, migrating an implementation from an old stack to a new version, or even porting the logic of an app from one platform to another. These tasks share three common traits:

    1. Many steps, long context: A single task may involve dozens of file reads and writes, dependency resolution, and build-and-test iterations; any wrong assumption at an intermediate step gets amplified by the steps that follow.
    2. Objective but complex criteria: The final result can usually be judged by “does it build, do the tests pass, is static analysis clean,” but “completeness” is a continuous quantity that needs to be weighted across dimensions.
    3. Non-trivial resource consumption: The time and call cost of long tasks is an order of magnitude higher than a single code completion, and it goes straight into the R&D budget.

    For this reason, evaluating agent capability in mobile app development typically uses a fixed task set run under the same sandbox and timeout constraints, recording results across four dimensions. Understanding what each of these four dimensions means and the tension between them is the prerequisite for making the right choice.

    2. The four core metrics: definitions and common misreadings

    2.1 Pass rate: strict binary judgment, reflecting “deliverability”

    Pass rate is the proportion of tasks that fully satisfy all acceptance criteria. Its value lies in being strict: a task only counts as passed if the build succeeds, all tests are green, static checks pass, and manual review finds no blocking defects. It therefore measures “can this go straight into mainline.”

    Misreading 1: equating a low pass rate with weak capability. With a small sample of 30 tasks, 1 percentage point is 0.3 of a task, the confidence interval is wide, and differences between similarly ranked models often lack statistical significance. Misreading 2: ignoring the composition of the task set. If the set contains many hard items such as “cross-major-version migration,” the pass rate will naturally be depressed, and completion becomes more informative in that case.

    2.2 Average completion: a continuous quantity that reveals “progress capability”

    Completion breaks each task into checkable subgoals (a buildable project, reachable target pages, implemented interface contracts, unit tests covering key paths, complete logging and telemetry, and so on) and weights them into a 0–1 score. It answers the question: has the model already gotten 80% of the work right and is only stuck on the last few steps?

    High completion combined with a low pass rate is an engineering-meaningful signal: it means the output is close to usable and the marginal cost of human intervention is low, making it suitable for a “human-agent collaboration to close out” model. Conversely, low completion combined with a high pass rate essentially never happens together; if it does, suspect first that the evaluation script has a loophole or that the task is too easy.

    2.3 Average duration: the underestimated key variable

    Duration is usually treated as a secondary metric, but it directly determines three things:

    • Feedback loop length: A single task running for a dozen-plus hours means an engineer can barely complete one “state requirement — verify result — revise requirement” loop within a day, pushing review and rework to the next day.
    • Resource occupation window: Long tasks occupy the build sandbox and quota for a long time, and evaluation tasks end up competing with real CI tasks.
    • Cost of failure: The longer a task runs, the higher the time cost of re-running after a failure, which pushes teams toward trying less and more cautiously.

    2.4 Per-run cost: amortizing cost onto “effective output”

    Per-run cost is not just the model call fee; a rigorous cost definition includes at least four parts:

    1. total cost = model input/output call fees
    2. + sandbox and build machine occupancy time fees
    3. + repeated build fees from dependency fetching and cache misses
    4. + extra call fees from tool calls, retrieval, and retry amplification

    From this we can derive two metrics that are more meaningful for decisions than “total cost”:

    • Cost per effective output = per-run cost ÷ number of passed tasks. It answers “how much does it cost to deliver one usable result.”
    • Time cost conversion = average duration × hourly rate of the humans waiting and reviewing × human occupancy rate. In many teams this item significantly exceeds the model and compute costs themselves.

    3. Illustrative comparison: how to read an evaluation results table

    The following table uses hypothetical sample data, only to demonstrate how to interpret the metrics, and does not represent the performance of any real product:

    Solution Pass rate Avg completion Avg duration Per-run cost Cost per effective output
    Solution A 30.0% 85.0% 26 h ¥300 ¥1000/task
    Solution B 18.0% 72.0% 12 h ¥68 ¥378/task

    Key takeaways:

    • Solution A has a higher capability ceiling, but its effective cost per delivered item is about 2.6× that of Solution B, and its feedback loop is more than twice as slow.
    • Solution B’s pass rate is lower, but a 72% completion means most outputs are “a few steps short of the finish line,” making it very suitable as the workhorse for daily iteration, paired with a human or a stronger model to close out.
    • The conclusion is not “A is worse than B” or “B is worse than A,” but that the two sit at different cost-and-latency working points and should map to different task tiers.

    4. Engineering the evaluation set and judgment logic

    Whether the metrics are trustworthy depends 80% on the evaluation infrastructure. A reproducible evaluation needs at minimum: fixed images and dependency lockfiles, controlled network policies, unified timeout and budget caps, replayable check scripts, and protection against training data contamination (task descriptions and acceptance scripts do not enter public repositories).

    Task acceptance criteria are best managed with declarative configuration:

    1. # task_spec.yaml — sample acceptance criteria for a task
    2. id: feature-offline-cache
    3. category: feature_development # app creation / feature dev / code migration / app conversion
    4. budget:
    5. max_wall_clock_hours: 8
    6. max_call_cost: 12.0 # currency unit unified by the team
    7. acceptance:
    8. build: ["assembleDebug", "testDebugUnitTest"]
    9. tests:
    10. - name: CacheRepositoryTest
    11. must_pass: true
    12. static_checks: ["lint", "dependency-scan", "secret-scan"]
    13. weighted_items: # used to compute completion
    14. - {name: build_ok, weight: 0.25}
    15. - {name: feature_reachable, weight: 0.30}
    16. - {name: tests_green, weight: 0.25}
    17. - {name: logging_and_fallback, weight: 0.20}
    18. human_review_points:
    19. - data migration and backward compatibility
    20. - graceful degradation path on first launch with no network

    The evaluation executor can be wrapped in a unified sandbox that outputs structured records for cross-run comparison:

    1. def evaluate(task, agent, cfg) -> Record:
    2. box = Sandbox(image=cfg.image, cpu=4, mem="8g", net="restricted")
    3. t0 = time.monotonic()
    4. try:
    5. res = agent.run(
    6. task.prompt,
    7. workspace=box.workspace,
    8. budget_tokens=cfg.token_budget,
    9. deadline=cfg.deadline,
    10. )
    11. report = box.run_checks(task.acceptance) # build + tests + static checks
    12. finally:
    13. box.destroy()
    14. hours = (time.monotonic() - t0) / 3600
    15. cost = (res.input_tokens * p_in
    16. + res.output_tokens * p_out
    17. + box.active_seconds * p_infra
    18. + res.retry_calls * p_tool)
    19. return Record(
    20. passed=(report.all_green and res.stop_reason == "completed"),
    21. completion=report.weighted_score(), # 0.0 ~ 1.0
    22. hours=hours,
    23. cost=cost,
    24. failure_stage=report.first_failed_stage, # aids root-cause analysis
    25. )

    On top of this, it is recommended to bucket failures by root-cause attribution (insufficient context, dependency resolution errors, build config errors, mistaken test assertions, timeout exhaustion, budget exhaustion). The attribution distribution is more useful than an aggregate score for guiding improvements: failures caused by budget exhaustion can be solved by relaxing constraints or running in stages, while contract-understanding errors require a better supply of repository context.

    5. A selection and tiered scheduling framework for teams

    Metrics ultimately have to translate into scheduling strategy. In practice, a fairly robust approach is tiered scheduling plus graded review:

    • Daily low-risk tasks (adding small features, filling in tests, fixing copy and telemetry, minor dependency version upgrades): hand them to a low-cost, low-latency configuration, optimizing for the lowest cost per effective output; the cost of a failed retry is small, so you can re-run them automatically.
    • Medium-complexity tasks (new page flows, local refactors, single-module migration): have a fast configuration produce a draft first, and once it reaches “buildable plus unit tests pass,” it enters human review, so people spend their time on design and contract judgment rather than typing boilerplate.
    • High-risk hard tasks (cross-major-version migration, architecture changes, security-related modifications): use a stronger configuration, set longer timeouts and higher budgets, enforce two-person review and canary release, and keep a complete execution trace for retrospective.

    The corresponding strategy can be expressed with simple routing logic:

    1. def route(task):
    2. if task.risk == "high" or task.estimated_steps > 60:
    3. return Tier(deep=True, deadline_h=24, reviewers=2, canary=True)
    4. if task.risk == "medium":
    5. return Tier(deep=False, deadline_h=6, reviewers=1, retry=2)
    6. return Tier(deep=False, deadline_h=2, reviewers=0, retry=3) # low-risk boilerplate tasks

    Supporting engineering constraints are equally important: set clear stop conditions and budget caps for the agent to avoid infinite iteration; agent output must go through exactly the same CI and gates as human submissions (build, unit tests, static checks, dependency and secret scanning) without lowering the bar just because it is “AI generated”; and all code changes keep records traceable to a specific task and execution trace.

    6. Observability and cost governance

    Both evaluations and live agent runs produce large volumes of execution traces; without an observability system, continuous optimization is impossible. A workable approach is to build it as an ordinary cloud pipeline:

    • Use a container platform to host isolated sandboxes, dynamically created per task and automatically reclaimed on timeout, avoiding long-term resource occupation by long tasks;
    • Use object storage to store code artifacts, build logs, test reports, and trace archives, organized by task ID for easy lookup;
    • Use a message queue for task orchestration and peak shaving, staggering evaluation batches away from daily CI peaks;
    • Use logging plus monitoring and alerting to collect duration distributions, token consumption, the share of failure stages, and the number of budget overruns, with threshold alerts;
    • Code hosting and R&D productivity tools handle the review workflow and gate integration.

    On Baidu AI Cloud, these capabilities fall under standard infrastructure categories and can be flexibly configured by team size; specific quotas and billing should follow the official documentation and actual usage. The three key actions of cost governance are: caching and reusing dependencies and build results (significantly reducing repeated build overhead), retrieving context on demand rather than stuffing it all in (improving both duration and accuracy), and setting budget caps by risk tier with enforced circuit-breaking.

    7. Conclusion: treat evaluation as a decision tool, not a leaderboard

    A valuable agent evaluation does not tell you “who is first” but “who to choose under which constraints.” When interpreting results, keep four principles in mind:

    1. Look at completion, not just pass rate: it measures salvageability and determines the cost of human-agent collaboration.
    2. Treat duration as a first-class metric: it determines the team’s feedback speed and R&D rhythm.
    3. Replace per-run cost with cost per effective output: fold failure re-runs and human review cost into total cost.
    4. Pay attention to the task set and judgment criteria: small samples and task sets biased toward high difficulty make rankings meaningless; cross-run comparison requires fixed images, timeouts, and acceptance scripts.

    Turning evaluation conclusions into a tiered scheduling strategy and unified gates is what truly lets AI coding agents enter the daily R&D workflow in a stable way—gradually expanding the share of long-horizon tasks they take on, under a premise of controllable budget and predictable response.