Engineering Trade-offs in Benchmarking AI Coding Agents on Long-Horizon Android Tasks

    This article explains why long-horizon task benchmarks are replacing isolated code-completion evaluations for AI coding agents. It covers the four core metrics—pass rate, average completion rate, average wall-clock time, and cost per run—and shows how higher completion often comes at the price of longer runtime and greater expense. It also provides practical guidance for teams choosing between lightweight and high-capability models, along with infrastructure recommendations for running long-running agent tasks reliably.

    1. Why long-horizon task benchmarks are replacing single-point completion benchmarks

    In the past, the ability of a code generation model was often evaluated through short-range tasks such as completing a single function or explaining a small code fragment. These tasks have short contexts and simple validation, but they can be far removed from real engineering work.

    In real mobile application development, especially for Android projects, engineers often work across a complete workflow that spans multiple files, modules, and stages: creating a project skeleton, integrating networking and persistence, implementing business features, completing code migrations, and performing platform conversion and adaptation. Such tasks are rarely completed in one pass. Instead, the agent must repeatedly plan, execute, validate, roll back, and retry within a long context.

    This is why coding benchmarks designed for realistic development scenarios have started to emerge. Instead of asking, “Can you write a function?”, they ask, “Can you independently complete a deliverable feature?” The evaluation method has also changed. Rather than scoring a one-time answer, a coding agent is allowed to operate autonomously in a sandbox environment for hours, sometimes tens of hours. Whether the goal has been achieved is judged through compilation, execution, and automated tests. This evaluation paradigm is closer to the nature of software engineering and is harder to “game.”

    2. Four core metrics: how to read a long-horizon task scorecard

    In this type of benchmark, a model’s performance is usually described by four dimensions. Understanding the relationships among them is essential for making sound decisions:

    • Pass rate: the percentage of tasks in a given set that are ultimately judged as fully successful. It reflects “whether the work can be completed” and is the metric closest to delivery quality.
    • Average completion rate: for tasks that are not fully completed, this metric measures the percentage of progress achieved. It is smoother than pass rate and helps show “how far the task got,” avoiding a purely binary evaluation.
    • Average wall-clock time: the time required to complete tasks. Long-horizon tasks can easily take more than ten hours, so runtime directly affects feedback speed and iteration cadence.
    • Cost per run: the cost of completing one full evaluation round. It is usually strongly related to computing resources consumed and the number of API calls made.

    For example, to illustrate the scale rather than any specific product, one high-capability model may achieve a pass rate of about one quarter on roughly thirty long-horizon tasks, with an average completion rate above 80%. However, its average runtime may approach 30 hours, and the cost per run may reach several hundred RMB. Another lightweight model may achieve a pass rate below one fifth and a completion rate of about 70%, but reduce average runtime to a little over ten hours and keep the cost per run at a single-digit dollar level. The former clearly leads comparable models in pass rate, while the latter holds a strong advantage in speed and cost.

    3. The engineering trade-offs behind high completion and long runtime

    The data above reveals a simple but often overlooked rule: an agent’s capability is often bought with time and cost. A model that pursues a high completion rate usually has longer runtime and higher cost for three reasons:

    1. More thorough exploration and validation. High-capability agents tend to understand the codebase before making changes and run more tests and checks after submission. If a failure is found, they roll back and retry. Each retry consumes time and compute, but these extra steps are also what push the completion rate higher.
    2. Stronger long-context handling. Long-horizon tasks require the model to maintain consistency across large and complex contexts. This significantly increases the load of each inference step, which raises both runtime and unit cost.
    3. Different trade-offs between risk and benefit. High-capability models are more inclined to proceed cautiously, sacrificing speed for correctness. Lightweight models are more inclined to deliver quickly, accepting a lower pass rate in exchange for faster response and lower cost.

    Therefore, it is not accurate to simply say one model is “better” than another. Pass rate, completion rate, runtime, and cost form a set of mutually constrained objectives. Choosing a model is essentially about finding the Pareto optimum for a specific engineering scenario.

    4. How to choose based on team profile

    Rational decision-making depends on aligning model capability with a team’s actual constraints. The following recommendations apply to several typical scenarios:

    • Budget- and response-sensitive teams
      Examples include teams building rapid prototypes, internal tools, or startups with frequent iterations. These teams should prioritize lightweight, low-cost, low-latency solutions. Even if the pass rate is slightly lower, human intervention can be used to correct outputs, resulting in a better overall return on investment.

    • Quality-first teams
      Examples include core business systems, external deliverables, and scenarios with high compliance requirements. When budget allows, these teams should choose higher-completion solutions. Longer runtime and higher cost can be traded for higher delivery quality and less rework later.

    • Mixed-strategy teams
      These teams can route tasks by level. Exploratory, low-risk tasks can be assigned to lightweight solutions for faster progress, while high-difficulty, high-risk tasks can be escalated to higher-capability solutions. This achieves a dynamic balance between quality and efficiency.

    Whatever approach is chosen, a practical method is to run a small-scale pilot with a few representative tasks. Record pass rate, runtime, and cost quantitatively, then estimate the manageable task throughput based on the real budget. This helps avoid intuition-driven decisions.

    5. Turning benchmark conclusions into engineering practice

    Regardless of which agent is used, automating long-horizon tasks places requirements on engineering infrastructure. The following general capabilities are worth preparing in advance:

    • Isolated and reproducible execution environments.
      Use container platforms to provide clean, isolated, and disposable sandboxes for each run. This prevents tasks from contaminating one another and makes problems reproducible.

    • Structured logs and observability.
      Use log services to record the agent’s planning, command execution, and error traces in detail. Combine these with monitoring and alerting so that teams can intervene when an agent stalls, avoiding long periods of ineffective runs that waste resources.

    • Artifact and intermediate state management.
      Use object storage to persist code artifacts, build caches, and test reports. This makes cross-run comparison and retrospection easier.

    • Message-driven task orchestration.
      Use message queues to schedule, distribute, and rate-limit long-horizon tasks. This can smooth resource consumption during concurrent execution while balancing cost and throughput.

    When these capabilities are integrated into a unified cloud-based development workflow, as is common on Baidu AI Cloud, agents can be incorporated into an automated pipeline from submission to validation. This supports a collaboration model in which humans focus on decision-making and acceptance while agents handle execution.

    6. Conclusion

    The real value of long-horizon task benchmarking is not to produce an isolated ranking. Its value lies in putting pass rate, completion rate, runtime, and cost on the same table, forcing teams to confront the trade-off between capability and efficiency.

    Once developers understand the engineering rule that higher completion often comes with longer runtime and higher cost, they can make more rational model choices based on budget, delivery cadence, and risk tolerance. Benchmarking is the means; implementation is the goal. By understanding the metrics thoroughly and preparing the necessary infrastructure, AI coding agents can become reliable productivity tools for engineering teams.