Long-Horizon Agent Evaluation and Cost Trade-offs in Mobile App Development Benchmarks

    This article examines how mobile app development benchmarks are shifting from snippet-level code generation to long-horizon, end-to-end delivery tasks. It breaks down four evaluation dimensions—pass rate, completion, wall-clock time, and cost—and analyzes the engineering trade-offs between high-completion, long-running agents and low-cost, short-running agents. It also proposes a reproducible internal evaluation framework and a risk-tiered hybrid pipeline for agent selection and deployment.

    1. Benchmark Shift: From Snippet Generation to End-to-End Delivery

    Agent evaluation for mobile app development is changing. Early benchmarks focused more on single-point code generation, function completion, or local fixes, with an evaluation style close to “given the context, judge whether the output is correct.” Such benchmarks are valuable, but they struggle to reflect the complexity of real engineering delivery. Real projects often include project initialization, dependency configuration, cross-file modifications, UI and business-logic integration, data migration, build repair, test verification, and final acceptance. A failure in any one of these stages can make the overall task undeliverable.

    As a result, benchmarks for mobile app development are beginning to emphasize long-horizon tasks: agents need to call tools across multiple turns, read project state, modify multiple files, run builds or checks, and continue correcting based on feedback. Common task types in public evaluations include app creation, feature development, code migration, and app conversion. What these tasks share is a long chain, many states, strict acceptance criteria, and a large number of hidden dependencies. In short-horizon tasks, a single correct generation can earn a score; in long-horizon tasks, the agent must continuously keep project state consistent, and contamination at any intermediate step can be amplified in later stages.

    This also explains why long-horizon evaluation cannot rely on a single metric. One solution may complete many local feature points and achieve relatively high average completion, but fail the final check due to integration errors, environment differences, or hidden acceptance failures. Another solution may have a slightly lower pass rate but take less time and cost less, making it suitable for rapid exploration and low-risk tasks. Understanding the engineering meaning behind these metrics is more important than consuming leaderboard conclusions.

    2. Four-Dimensional Metrics: Pass Rate, Completion, Wall Time, and Cost

    Long-horizon agent evaluation usually needs to record four types of metrics at the same time. They answer different questions.

    The first is pass rate. Pass rate is the end-to-end binary acceptance result, reflecting whether a task meets the deliverable standard. It is closest to the business outcome, but it is also easily pulled down by long-tail failures. In long-horizon tasks, a low pass rate does not necessarily mean the main functionality is wrong; it may simply mean that a build configuration, boundary condition, or hidden check was not satisfied. Therefore, pass rate must be analyzed together with failure reasons rather than treated only as a ranking number.

    The second is completion. Completion is usually calculated as a weighted score over task checkpoints or feature points, reflecting how deeply the agent advances the task. High completion indicates strong local problem-solving ability, but high completion does not equal deliverability. If completion is above 80% while pass rate is only around 20%, it often means that “most feature points have been touched, but systematic gaps remain in end-to-end integration and strict acceptance.” Such gaps may come from cross-file dependencies, incomplete state rollback, insufficient test coverage, or non-reproducible environments.

    The third is wall time. Wall time for long-horizon tasks is not single-inference latency; it includes model inference, tool execution, build and test time, retry waits, and queueing. Wall time directly affects the feedback cycle and concurrent resource occupancy. When a single task takes tens of hours, teams must consider asynchronous execution, checkpoint recovery, parallel scheduling, and human takeover mechanisms. Otherwise, long-running solutions will occupy concurrency slots and reduce overall throughput.

    The fourth is cost. The cost of a single benchmark run should be split into inference cost, tool-call cost, environment runtime cost, retry cost, and human review cost. A low single-run cost does not mean a low total cost. The real focus should be unit success cost:

    1. unit_success_cost = total_cost / successful_tasks
    2. effective_throughput = successful_tasks / total_wall_time
    3. human_intervention_rate = tasks_requiring_manual_fix / total_tasks

    If the pass rate is only 20%, the single-run cost needs to be multiplied by about five to estimate unit success cost. If human review and rework are added on top, the apparent advantage of a low-cost solution may be eroded. Conversely, a high-completion, long-running solution may be better on total cost of ownership if it significantly reduces manual rework.

    3. Engineering Trade-offs Behind High-Completion, Long-Duration Solutions

    High completion and long duration usually correspond to a “quality-first” strategy: more validation rounds, more frequent self-repair, more thorough cross-file checks, and more conservative rollback and retry. The benefit of this strategy is fewer omissions and better local correctness, making it suitable for high-risk complex tasks, one-time migrations, and refactoring scenarios with high compliance requirements. Its costs are also obvious: long feedback cycles, high concurrent resource occupancy, poor cost predictability, and greater risk of environment drift and state contamination in long chains.

    From an engineering perspective, teams cannot simply pursue “higher completion.” Long-horizon tasks should set budget circuit breakers, maximum steps, checkpoint commits, and human takeover thresholds. For example, when multiple consecutive validation rounds make no progress, when unit task cost exceeds budget, or when a key checkpoint fails, automatic execution should stop and move to human diagnosis. High completion has business value only when it translates into acceptable delivery, so hidden acceptance, regression testing, and manual spot checks cannot be omitted.

    4. Applicability Boundaries of Low-Cost, Short-Duration Solutions

    Low-cost, short-duration solutions may have a slightly lower pass rate, but their speed and cost advantages are clear, making them suitable for budget-sensitive and response-time-sensitive development teams. They are better suited to low-risk modifications, rapid exploration, candidate solution generation, issue localization, and batch screening. A typical usage pattern is “explore first, fall back later”: use a low-cost solution to quickly narrow the problem scope, generate candidate patches, or identify affected files, and then use a high-completion solution or human review to complete final acceptance.

    However, low-cost solutions cannot replace the final quality gate. If the agent stops too early or validates insufficiently, hidden defects may lead to higher rework costs. Teams should establish a tiered strategy: low-risk tasks can be completed automatically by low-cost solutions with spot checks; medium- and high-risk tasks should use a hybrid pipeline where low-cost solutions explore and high-quality solutions converge; critical migration tasks must retain human review.

    5. Reproducible Internal Evaluation Framework

    To make evaluation results truly guide selection, teams should build internal benchmarks rather than relying only on public leaderboards. The task set should be layered: short-horizon fixes, medium-horizon feature development, and long-horizon migration and conversion. Environments should be snapshotted. Acceptance should include hidden checks. Execution traces should record cost, wall time, retries, and human interventions.

    1. record TaskResult {
    2. task_id
    3. passed
    4. progress
    5. wall_time
    6. cost
    7. retries
    8. human_interventions
    9. }
    10. function evaluate(agent, suite, budget):
    11. for task in suite:
    12. env = snapshot(task.environment)
    13. trace = agent.execute(task, env, limits)
    14. passed = verifier.accept(trace, task.hidden_checks)
    15. progress = verifier.progress(trace, task.checkpoints)
    16. cost = sum(trace.resource_usage)
    17. time = trace.wall_time
    18. emit(TaskResult(...))
    19. return aggregate(
    20. pass_rate,
    21. avg_progress,
    22. unit_success_cost = total_cost / max(success_count, 1),
    23. effective_throughput = success_count / total_wall_time,
    24. human_intervention_rate
    25. )

    During evaluation, teams should also monitor regression sets to prevent solutions from overfitting to public samples; record failure categories to distinguish build failures, integration failures, acceptance failures, and environment failures; and use asynchronous queues, environment snapshots, and log traces for long-running tasks to ensure failures are reproducible. In cloud environments, general logging services, monitoring and alerting, and containerized environments can be used to record runtime processes, but the core remains evaluation design and data governance.

    6. Implementation Recommendations

    Selection should not stop at “who has a higher pass rate.” A more robust approach is to tier tasks by risk, run small-scale controlled experiments first, record pass rate, completion, wall time, cost, human intervention rate, and rework rate, and then calculate unit success cost. High-completion, long-duration solutions are suitable for high-value, high-complexity tasks that can run asynchronously; low-cost, short-duration solutions are suitable for exploration, low-risk modifications, and batch screening. Ultimately, teams should form a hybrid pipeline: low-cost solutions explore quickly, high-quality solutions converge on delivery, and humans handle critical acceptance. Leaderboards are inputs; engineering decisions must return to business risk and total cost of ownership.