Android Benchmark Leaderboard: Choosing AI Agents for Long-Horizon Tasks

    This article explains how to interpret an Android benchmark leaderboard for AI agents on long-horizon tasks. It reviews four key metrics and the questions to confirm before comparing results, analyzes the trade-offs between a heavy-reasoning agent and a lightweight efficient agent, and provides task-based selection guidance plus fault-tolerance and pre-selection checklists.

    1. How Are the Four Leaderboard Metrics Calculated?

    Comparing different agents requires consistent metric definitions. Before selecting an agent, confirm how the following four metrics are calculated.

    Metric Meaning Questions to confirm
    Task pass rate The proportion of tasks that the agent completes independently and that pass all automated tests Are compilation, UI assertions, and logic checks all counted?
    Average completion The average progress reached on tasks that are not fully passed Is progress measured by files, test cases, or manual judgment?
    Average duration The total time from receiving a task to outputting a result Does it include queueing, API waiting, and human intervention time?
    Cost per benchmark run The cost calculated from token consumption and model billing rules Which date’s listed pricing is used, and are retries included?

    There is one further point to verify in the numbers provided by the original article. The heavy-reasoning approach has a pass rate of 24.0%. If the total number of tasks is 30, the number of passed tasks must be an integer, but 7 tasks correspond to 23.3% and 8 tasks correspond to 26.7%; neither equals 24.0%. The lightweight approach’s 17.3% has the same issue. This suggests that the numbers may be averages over multiple runs, or that the total number of tasks is not 30. Before citing any number, check the official leaderboard page for the version, release date, task set, and test environment.

    2. Why Does the Heavy-Reasoning Approach Have Higher Pass Rates and Higher Costs?

    The original data shows that this approach has a 24.0% pass rate on 30 long-horizon tasks, an average completion of 82.3%, an average duration of 27.4 hours, and a cost per benchmark run of $317.4. In the original comparison, its pass rate is below the top-ranked model’s 28.7% and above the other compared models.

    The duration and cost of this type of approach mainly come from its working method. Long-horizon tasks involve dependencies across multiple files. The agent needs to read the codebase first, generate multiple candidate plans, and perform internal evaluation. The following code only illustrates this flow; it is not a runnable implementation.

    1. # Illustrative only; not a runnable implementation
    2. while not task.is_complete():
    3. state = agent.observe_codebase()
    4. plans = agent.generate_plans(state, k=3)
    5. best_plan = agent.self_debate(plans)
    6. result = executor.run_tests(best_plan)
    7. if result.failed:
    8. agent.deep_analyze_error(result.log)

    Each loop increases the number of code reads, reasoning steps, and tests. The 27.4-hour duration indicates that many modifications, compilations, and error backtracking occur during the task cycle. A cost of $317.4 per run is high for individual developers; teams need to evaluate it against the output.

    The approach reaches an average completion of 82.3%. Even if a task is not fully passed, the generated code framework may still have high reuse value. Whether the cost is worth paying depends on the value of the task itself and the subsequent manual repair workload.

    3. Where Does the Lightweight Approach Save, and What Is the Trade-off?

    The original data shows that this approach has a 17.3% pass rate, an average completion of 71.2%, an average duration of 12.0 hours, and a cost per run of $9.9. Based on the original data, its duration is about 44% of the heavy-reasoning approach, and its cost is about 3%.

    The gap comes from different strategies. The lightweight approach performs less multi-plan debate and usually uses a loop of fast generation, fast compilation, and fast repair based on errors. This reduces token consumption per reasoning round, but it also provides less global planning for complex dependencies, so the pass rate is lower.

    For tasks that need a quick first draft, 71.2% completion can allow developers to finish the remaining logic in a shorter time. Budget-limited teams or business lines that need large-scale concurrent runs can prioritize cost control. However, if a task requires passing all tests in one go, a 17.3% pass rate means most tasks still require manual takeover.

    4. What Are the Differences in a Side-by-Side Comparison?

    Dimension Heavy-reasoning Lightweight efficient
    Task pass rate (original data) 24.0% 17.3%
    Average completion (original data) 82.3% 71.2%
    Average duration (original data) 27.4 hours 12.0 hours
    Cost per run (original data) $317.4 $9.9
    Better-suited tasks High-value tasks such as framework refactoring and low-level protocol implementation High-frequency tasks such as routine pages and standard APIs
    Main risks Long duration, high cost, and high overhead for large-scale concurrency Low pass rate under complex dependencies, requiring more manual repair

    All data in the table comes from the original article and has not yet been checked item by item against the official leaderboard. Use the officially published data before making decisions.

    5. Which Approach Should Be Chosen for Different Tasks?

    Selection can start by layering tasks by type, then deciding which type of agent to invest in.

    1. High-value, high-complexity tasks, such as core framework refactoring and low-level protocol implementation. Failure is costly for these tasks, so it is reasonable to prioritize an approach with a higher pass rate and accept longer duration and higher cost.
    2. High-frequency, low-difficulty tasks, such as routine UI page generation and standard CRUD API implementation. These tasks are repeated many times and are cost-sensitive, so the lightweight approach is usually more suitable.
    3. Exploratory tasks, such as researching an unfamiliar third-party library or validating a prototype approach. You can use the lightweight approach for rapid trial and error first, then switch key parts to an approach with a higher pass rate.

    6. What Fault Tolerance and Operational Safeguards Are Needed in Deployment?

    AI output is uncertain, so any approach needs supporting fault-tolerance mechanisms.

    • Sandbox isolation: Run each compilation and test in an independent environment so they do not interfere with each other, and restrict network and file access scope.
    • Log auditing: Save the agent’s complete operation records and compilation output to support issue reproduction and later tuning.
    • Timeouts and circuit breaking: Set task duration and token limits, and stop automatically when they are exceeded to prevent the agent from falling into an infinite loop and wasting resources.
    • Human review: Before merging code, have a developer check key changes, especially code related to permissions, data storage, and network requests.
    • Metric monitoring: Record the pass rate, duration, and cost of each run, and compare them regularly with official leaderboard data to detect deviations in time.

    7. What Should Be Checked Before Selection?

    • Has the leaderboard version, release date, and task set been confirmed on the official page?
    • Are the measurement definitions for pass rate, completion, duration, and cost clear?
    • Is the team’s task complexity close to the benchmark task set?
    • Are the per-task budget cap and monthly total budget clear?
    • Is the cost still acceptable when calculated at the expected concurrency level?
    • Do code and log storage and transmission meet data compliance requirements?
    • Is human review arranged, along with a fallback process for failures?
    • Is there a plan to review the latest leaderboard periodically?

    After a leaderboard is updated, rankings and data may change. Rechecking the latest data before selection is more reliable than relying on old conclusions.