Evaluating AI Agents on Long-Horizon Engineering Tasks: Completion, Runtime, and Cost Trade-offs

    This article examines how AI coding agents perform on long-horizon engineering tasks such as mobile app delivery, and why a single leaderboard number is misleading. It breaks scoring into four metrics—task pass rate, average completion, average wall-clock time, and cost per run—and explains the trade-off behind "high completion with long runtime." It then shows how to build a reproducible evaluation pipeline of your own, with realistic task sets, isolated sandboxes, tiered gating scorecards, and unified cost accounting, and closes with a tiered-routing strategy and a list of common pitfalls to avoid.

    1. Background: Why Engineering Benchmarks Are Getting “Heavier”

    Early evaluations of coding ability mostly focused on short-horizon tasks: single-function completion, single-file algorithm problems, and explaining isolated code snippets. These tasks share common traits—a limited context window, a single validation method (usually one batch of unit tests), and failures that can be rolled back immediately. Real-world mobile application engineering delivery looks completely different:

    • Cross-file coupling: Adding a single screen often requires changes to the view layer, the data layer, routing configuration, and resource files at once. Missing any one of them breaks the build.
    • Cross-stage validation: “Compiles successfully” is not enough; the app must install and run, key interactions must be reachable, and existing features must not regress.
    • Cross-round iteration: An agent must first read the build scripts and understand directory conventions, then write code, then run the build, parse errors, and fix them, repeating this loop for several rounds before it can wrap up.

    Because of these traits, the industry has begun to build benchmarks around real engineering tasks: given an initial repository and a requirements description, an agent is allowed to freely read code, modify code, and run build commands in a controlled environment, after which a scorer decides whether the deliverable is usable. Mobile app development is especially well-suited as a vehicle for these benchmarks: its build pipeline is long, its dependencies are complex, and its validation methods are explicit (whether it builds, whether it runs, whether tests pass), which makes scoring relatively objective.

    2. Four Metrics: Unpacking the Score

    A public leaderboard usually reports four dimensions at once, and each measures something completely different:

    Metric Meaning Capability reflected Common misreading
    Task pass rate Share of tasks that fully meet the acceptance criteria on the given task set Delivery reliability A single number that carries no information about “how far off” a failure was
    Average completion Weighted completion ratio after tasks are broken into subitems Progress-making ability Completion is not the same as deliverable
    Average wall-clock time Time from start to termination for a single task Iteration speed, feasibility of parallelism Strongly correlated with human intervention cost
    Cost per run Compute and API-call cost to finish one round of tasks Unit economics and scalability Scales with task size and number of retries

    Take one round of a public mobile app development benchmark as an example, where three typical data shapes can be observed (to keep things neutral, these are referred to as Scheme A/B/C):

    • Scheme A: pass rate 28.7%, serving as the reference upper bound for that round.
    • Scheme B: pass rate 24.0%, average completion 82.3%, average wall-clock time 27.4 hours, cost per run around $317.4.
    • Scheme C: pass rate 17.3%, average completion 71.2%, average wall-clock time 12.0 hours, cost per run around $9.9.

    Three points must be emphasized:

    1. Pass rate and completion have to be read together. The huge gap between Scheme B’s 82.3% completion and its 24.0% pass rate shows that there is a high-friction zone between “most subitems completed” and “the whole is acceptable”—usually the final build convergence, resource references, and platform compatibility.
    2. The order-of-magnitude gap in cost and time matters more. Scheme B and Scheme C differ by about 30x in cost and about 2.3x in time, yet only 6.7 percentage points in pass rate. This means the marginal cost per unit of pass rate differs enormously between the two schemes.
    3. Data must be compared under the same constraints. A pass rate obtained under “no time limit + no budget limit” cannot be directly compared with one obtained under “2 hours per task + fixed token quota.” The looser the constraints, the higher the score, but the further it drifts from real engineering feasibility.

    3. The Engineering Trade-off Behind “High Completion + Long Runtime”

    This is the most worthwhile point to dig into. If a model has high average completion but takes tens of hours to wrap up, it is making a clear trade-off: trading time and compute for the depth of task completion.

    3.1 Three Technical Causes

    First, the scale of context exploration. Short-horizon tasks require reading only a few files. Long-horizon tasks require building a “map of the repository” first: identifying the build entry point, dependency lists, directory conventions, and existing similar implementations. This step’s cost is roughly linear in repository size and is one of the main sources of wall-clock time.

    Second, the number of validation loops. One build failure may expose three new problems (a missing resource reference, a signature mismatch, an overly strict test expectation), each needing multiple rounds of fixes. Every round includes a build, a wait, and log parsing, which multiplies wall-clock time by several times.

    Third, the completeness of self-checks and backtracking. Some implementations proactively run regression validation, cross-module consistency checks, and documentation/config synchronization before submitting. This “extra work” raises completion but also directly lengthens wall-clock time.

    3.2 The Cost: Parallelism and Feedback Cadence

    Wall-clock time is not merely a matter of waiting; it changes the team’s engineering rhythm:

    • High concurrency cost: If a single task occupies an isolated environment for tens of hours, the number of tasks that can be advanced at the same moment is hard-capped. Assuming 8 isolated environments run simultaneously, Scheme B can advance at most 8 tasks a day, while Scheme C can advance 16 or more.
    • Longer feedback latency: The developer’s usual habit is “submit requirement—wait for result—fix.” When the feedback cycle is measured in hours or even days, human-machine collaboration degrades into asynchronous batch processing, and the interactive value drops.
    • Hard to move into the pipeline early: Most teams’ pre-commit checks require a response in minutes. A step taking ten-plus or tens of hours can only sit in a nightly batch or a separate review channel.

    3.3 When “Slow” Pays Off

    • Framework/architecture migration, legacy code conversion: These tasks inherently require a large one-time investment with long-term payoff, and manual completion is extremely costly, so trading tens of hours for 80%+ completion is very economical.
    • Unattended batch jobs: Nightly runs and clearing the weekend backlog—wall-clock time is not the bottleneck, and compute utilization is actually steadier.
    • Low risk but high volume: Filling in comments, standardizing directory structure, batch-replacing configuration items, generating boilerplate code.

    3.4 When You Must Be “Fast”

    • Interactive development: The developer is waiting at the terminal, and feedback must come in seconds to minutes.
    • Pre-commit guards: Checks must complete before a merge, and latency directly equals the team’s waiting cost.
    • High-concurrency automation: One merge may need validation across multiple modules simultaneously, so it must be able to scale out horizontally.

    The conclusion is not “which one is better,” but rather: pass rate sets the ceiling, while wall-clock time and cost determine whether that ceiling can be reproduced at scale. The real engineering question is “under my constraints, which combination has the lowest cost per deliverable task.”

    4. Build Your Own Evaluation: Make the Conclusion Hold for Your Own Codebase

    The value of public leaderboards is to provide a reference frame, but every team’s tech stack, directory conventions, and acceptance standards differ, so copying the scores directly is meaningless. It is worth building a reproducible evaluation pipeline of your own.

    4.1 Constructing the Task Set

    • Sources must be real: Select from historical defect tickets, unmerged feature requests, and records of completed refactors, keeping the original triple of “requirements description + final merged code + acceptance evidence.”
    • Layered design: Divide into four categories—app creation (scaffolding a runnable skeleton from an empty repo), feature development (adding capabilities on existing code), code migration (rewriting across languages or frameworks), and config/dependency conversion. The wall-clock time and difficulty distributions across these four differ greatly and must be tracked separately.
    • Control the size: 10–30 tasks per category is enough to support trend judgment; going larger lets a single evaluation run’s cost spiral out of control.
    • Keep the holdout set uncontaminated: Reserve some tasks that are never published and never enter examples or prompts, to detect whether the model is merely “memorizing answers.”

    4.2 Environment and Isolation

    1. # Basic requirements for the evaluation environment
    2. - Dependency versions locked; build cache cleared at the start of each task to guarantee reproducible cold starts
    3. - Independent sandbox per task: separate filesystem, separate build directory, separate network egress policy
    4. - Fixed resource quotas: CPU, memory, disk, max wall-clock time, max build attempts
    5. - Secrets and credentials injected read-only, revoked immediately after the task ends
    6. - Full trace recorded: command sequence, build logs, timing, call metering, for post-hoc attribution and cost accounting

    4.3 Scorer Design (Tiered Gating)

    The recommended approach is a combination of “step-by-step gating + weighted completion.” The gates decide pass or fail; weighted completion is used to diagnose which layer things are stuck at.

    1. # Scorecard example (illustrative, not the config format of any specific product)
    2. task: feature-add-settings-page
    3. gates: # all must be satisfied to pass
    4. build_release: true # release build succeeds, no errors
    5. unit_tests: pass # new unit tests pass + existing unit tests do not regress
    6. static_checks: no_new_error
    7. runtime_smoke: true # target screen reachable after install, key interactions don't crash
    8. no_secret_leak: true # deliverable contains no plaintext credentials
    9. subitems: # used to compute average completion, weights adjustable to team goals
    10. - name: data_layer # data layer implementation weight: 0.25
    11. - name: ui_layer # UI and interactions weight: 0.30
    12. - name: routing_config # routing and config weight: 0.15
    13. - name: resource_i18n # resources and localization weight: 0.10
    14. - name: test_coverage # test coverage for new logic weight: 0.20
    15. budget:
    16. max_wall_clock_minutes: 240
    17. max_build_attempts: 12
    18. max_token_quota: 4000000
    1. # Core loop of evaluation orchestration (illustrative)
    2. def run_task(agent, task, rubric, budget):
    3. env = Sandbox(image=task.image, limits=budget.as_limits())
    4. with env:
    5. agent.init(repo=task.initial_repo, prompt=task.spec, tools=env.tools)
    6. while not agent.done and env.consumed_within(budget):
    7. action = agent.next_action()
    8. result = env.execute(action) # read files / search / edit code / build / run tests
    9. agent.observe(result)
    10. if result.kind == "build_failed":
    11. # feed the log back after trimming, to avoid drowning the context in noise
    12. agent.observe(truncate(result.log, keep_last_lines=200))
    13. diff = env.diff()
    14. gates = evaluate_gates(diff, rubric.gates, env)
    15. score = weighted_completion(diff, rubric.subitems)
    16. return Result(
    17. passed=all(gates.values()),
    18. completion=score,
    19. wall_clock_minutes=env.wall_clock_minutes(),
    20. cost=env.cost_usd(),
    21. trace=env.trace,
    22. )

    4.4 Statistical Methodology

    • Run multiple times and report the distribution: Long-horizon tasks have high variance, so a single run is not enough to draw conclusions. It is recommended to run each task 3–5 times, report the median and interquartile range, and also provide pass@k.
    • Distinguish “timeout” from “failure”: A timeout indicates the budget is set too tight or the strategy is too conservative; counting it as a failure underestimates capability.
    • Unify the cost accounting: Fold API-call cost, sandbox compute cost, and maintenance and human-review labor hours into a “cost per deliverable task.” That is the number usable for decisions.

    5. A Reusable Engineering Strategy: Tiered Routing + Mandatory Verification

    The conclusion of the evaluation ultimately lands on “how to use it.” A relatively robust approach is to route by task risk and complexity, rather than using a single strategy for everything.

    1. # Tiered routing strategy (illustrative)
    2. def route(task):
    3. if task.risk == "high" or task.scope == "cross_module_migration":
    4. return Plan(model_tier="deep",
    5. wall_clock="8-24h",
    6. parallel=1,
    7. review="human_required")
    8. if task.scope in ("single_module_feature", "test_generation"):
    9. return Plan(model_tier="balanced",
    10. wall_clock="30-90min",
    11. parallel=4,
    12. review="ci_gate")
    13. if task.scope in ("format", "doc", "boilerplate", "local_edit"):
    14. return Plan(model_tier="fast",
    15. wall_clock="<5min",
    16. parallel=16,
    17. review="ci_gate")

    Combined with three hard constraints:

    1. Admission must rely on verification, not self-assessment. No matter which tier, every task must pass the same build and test gates before merging. An agent’s own claim of “done” cannot replace executable evidence.
    2. Budgets and circuit breakers. Set a wall-clock cap, a build-attempt cap, and a token-quota cap for each task; when a limit is exceeded, terminate and archive the trace, which helps later prompt and retrieval-strategy optimization.
    3. Auditable traces. Produce every change as a separate branch or patch set, and keep the full command and log trace, both for human review and for security auditing.

    When it comes to hosting this pipeline, general-purpose cloud capabilities are enough: use a container platform to provide isolated build environments and elastic scaling, use object storage to archive repository snapshots, build artifacts, and full traces, use a logging service to collect build and runtime logs centrally, and use monitoring and alerting to track trends in task success rate, P95 latency, and unit cost. If a team uses Baidu AI Cloud, these capabilities can be assembled in a conventional way, without introducing extra proprietary dependencies for the evaluation system.

    6. Common Pitfalls

    • Looking only at pass rate: Ignoring wall-clock time and cost may lead you to pick a scheme with “pretty numbers but no scalability.”
    • Treating completion as deliverability: The gap between 80% completion and “mergeable” is often the last 10% of convergence work, and that 10% is usually the main source of manual rework.
    • Comparing under constraints that are too loose: Scores obtained with no time limit and no budget limit cannot predict performance in a real engineering environment.
    • Overlapping the eval set with the prompts: Once evaluation tasks enter examples or the retrieval index, scores inflate noticeably.
    • Skipping security review: Dependency changes, permission declarations, and network configurations produced by agents need separate review and cannot be treated the same as a human submission.
    • Ignoring regressions: A new feature passing while an existing feature breaks is the most common failure mode, so the scorer must include a regression gate.

    7. Conclusion

    The four numbers a public benchmark reports—pass rate, completion, wall-clock time, and cost—are essentially consumption profiles of four different resources. The high-runtime, high-completion route trades time and compute for task depth, suiting migration, conversion, and batch processing. The low-cost, fast route trades controllable rework for throughput and responsiveness, suiting local development, test generation, and pipeline guards. A genuinely engineering-valuable judgment is not memorizing some score, but building an evaluation pipeline for your own codebase: layered tasks, isolated environments, explicit gates, multiple sampling runs, and unified cost accounting. Once you can compute “cost per deliverable task” under your own constraints, technology selection stops depending on leaderboard rankings and starts depending on reproducible data.