Evaluating Android AI Coding Agents: Balancing Pass Rate, Time, and Cost
Recently, a benchmark leaderboard for Android development AI agents was updated with two sets of evaluation results. This article explores how to interpret these metrics across complex, long-horizon development tasks. It breaks down the meaning of pass rate, completion rate, execution time, and cost per run, analyzes the suitable scenarios for different agent solutions, and provides a framework for enterprise-level internal testing, risk management, and weighted scoring.
Recently, a benchmark leaderboard for Android development AI agents was updated with two new sets of evaluation results. The evaluated agents are intelligent coding assistants capable of handling requirement comprehension, code writing, project building, issue localization, test fixing, and continuous iteration. When reading these results, the key is to understand what each metric represents and what insights it supports.
1. What Do Long-Horizon Android Tasks Assess?
Long-horizon Android tasks differ significantly from simple function completions. The agent must first understand requirements, analyze the project structure, locate relevant modules, modify code, execute builds and tests, and iterate based on error messages.
These tasks typically have four characteristics:
- Large context span: Tasks may simultaneously involve the UI layer, business logic layer, data access layer, and configuration files. Modifying only local parts easily leads to interface mismatches or state synchronization anomalies.
- Continuous execution: Early misinterpretations of requirements propagate through every subsequent step. Even if the code syntax is correct, it may fail acceptance criteria.
- Verification required: A successful compilation only proves basic implementation integrity. Page behavior, data state, exception handling, and compatibility still need checking.
- Environmental impact: Build times, dependency downloads, test device status, and resource limits all affect time consumption. Results across different environments should not be directly compared.
2. Verify the Metrics Before Comparing
The raw data shows: the first solution has a pass rate of 24.0%, an average completion rate of 82.3%, an average duration of 27.4 hours, and a single benchmark run cost of approximately $317.4. The second solution has a pass rate of 17.3%, an average completion rate of 71.2%, an average duration of 12.0 hours, and a cost of about $9.9 per run.
One issue requires attention: out of 30 tasks, 24.0% equals 7.2 tasks, and 17.3% equals 5.19 tasks—neither are integers. Therefore, these percentages are likely averages of multiple runs or use a different calculation method. Before citing them, verify the original leaderboard’s version, release date, number of repeated runs, sample size, and runtime environment. Until the methodology is confirmed, treat these numbers as references rather than the sole basis for procurement or architectural decisions.
3. What Do the Four Metrics Indicate?
1. Pass Rate
Pass rate indicates the proportion of tasks completed and accepted, closest to the final deliverable. However, it does not reflect the degree of failure. Some tasks fail by a single edge case, while others have severe architectural flaws. It is recommended to categorize failure reasons, such as:
- Requirement misunderstanding
- Key features unimplemented
- Compilation or dependency configuration failures
- Insufficient test coverage
- Incomplete exception handling
- Regression in existing features due to excessive modification scope
Knowing the failure type helps determine whether the issue lies in model reasoning, tool invocation, task description, or the project environment.
2. Average Completion Rate
Average completion rate measures how far the output is from being usable. If the leaderboard does not explicitly define it, readers can use this formula:
Completion Rate = Passed Acceptance Items ÷ Total Acceptance Items × 100%
Assuming a task has 10 acceptance items and the agent passes 8, the completion rate is 80%. The task still fails because the remaining 2 items are unmet. A high completion rate with a low pass rate usually indicates the agent can write the main code but struggles with edge cases, state recovery, multi-module integration, error prompts, and build compatibility.
3. Average Duration
Time data only reflects the agent’s runtime, not the actual delivery time. A more useful metric is end-to-end delivery time, which includes agent execution, human review, failed retries, test environment preparation, and pre-merge fixes. If an agent quickly generates a large volume of drafts but developers spend hours debugging, apparent speed does not equal real efficiency.
4. Cost Per Run
Single-run cost is the total expense of one complete benchmark run and cannot be used directly to compare cost-effectiveness. A more reasonable metric is the cost per effective delivery:
Cost Per Effective Delivery = Total Run Cost ÷ Number of Passed Tasks
For example, if a run costs $200 and passes 6 tasks, the cost per effective delivery is $200 ÷ 6 ≈ $33.3. This example illustrates the algorithm and is not from the leaderboard. If a high-cost solution has a significantly higher pass rate, its cost per effective delivery might not be worse. Conversely, if a low-cost solution requires frequent retries or extensive manual fixes, its overall cost might be higher.
4. Suitable Scenarios for Two Types of Solutions
| Dimension | Solution 1 (High Pass Rate, High Cost) | Solution 2 (Fast, Low Cost) |
|---|---|---|
| Pass Rate | 24.0% | 17.3% |
| Avg. Completion Rate | 82.3% | 71.2% |
| Avg. Duration | 27.4 hours | 12.0 hours |
| Cost Per Run | ~$317.4 | ~$9.9 |
| Suitable Tasks | Complex business modules, multi-file refactoring, core features | Code drafts, batch simple modifications, routine test additions |
| Human Review Requirement | Relatively low, but acceptance needed | Relatively high, requires checkpoints |
Solution 1 is suitable for teams with limited manual review resources looking to reduce trial and error. Solution 2 fits scenarios with limited budgets requiring rapid candidate implementations. Low-cost solutions can also handle complex work if tasks are broken down, automated tests are added, and manual checkpoints are set. However, splitting tasks increases scheduling and management overhead, requiring teams to balance the trade-offs.
5. Enterprise Self-Assessment: Tiered Evaluation Sets
Leaderboard results cannot replace internal testing. Differences in codebase scale, tech stack, testing standards, and delivery workflows will affect the agent’s actual performance. It is recommended to build an evaluation set across four tiers:
- Basic Code Tasks: Interface completion, data structure adjustments, page field modifications, and simple logic fixes to assess basic code generation capabilities.
- Module-Level Tasks: Adding pages, modifying state management, integrating local data, and adding tests to assess multi-file collaboration and project understanding.
- Long-Horizon Delivery Tasks: Full feature development, historical bug fixes, cross-module refactoring, and regression testing to observe continuous execution stability.
- Real-World Business Tasks: Extracted from desensitized historical tickets, retaining real constraints, edge cases, and acceptance criteria. This tier is closest to production value but requires strict code security, data isolation, and permissions.
Maintain unified records for each evaluation for comparison. Recommended fields:
| Task ID | Total Acceptance Items | Passed Items | Failure Type | Manual Intervention (mins) | Cost Per Run ($) | Retry Count | Regression Defects |
|---|---|---|---|---|---|---|---|
| Sample T-03 | 12 | 10 | 1 Compile/Dependency Failure | 35 | 4.2 | 2 | 0 |
The sample row is fictional data, used only to illustrate the field format.
6. Risk Control via Three Lines of Defense
The first line is task constraints. Requirement descriptions should specify inputs, outputs, allowed modification scopes, prohibited modules, and acceptance criteria to prevent the agent from expanding the modification scope when the goal is unclear.
The second line is automated verification. Execute static checks, compilation, unit tests, and critical path tests after each modification. UI tasks should also verify interaction paths and state changes.
The third line is human review. Focus on business logic, permission boundaries, exception handling, performance impact, and maintainability, rather than just whether the build passes.
A trackable task workflow can be designed as: requirement parsing, impact analysis, modification plan generation, code execution, build and test, log-based fixing, and human review and merge. The agent plays a repeatable, trackable, and verifiable collaborative role, but the final decision remains with the developer.
7. Example Weights for Comprehensive Scoring
Beyond the four basic metrics, you can also track first-time success rate, retry count, regression defects, manual intervention duration, code change scale, test completeness, explainability, and security. Weights should be adjusted based on business scenarios. The following are examples:
- High-risk core scenarios: Accuracy 40%, Security 20%, Manual Intervention 20%, Cost 10%, Speed 10%.
- Low-risk batch scenarios: Speed 30%, Unit Cost 30%, Accuracy 20%, Manual Intervention 10%, Security 10%.
8. Conclusion
The basic principle for selection is: prioritize pass rate, regression defects, and security for high-risk core tasks; prioritize speed and cost per effective delivery for low-risk batch tasks.
This leaderboard shows that AI coding systems can now handle continuous development tasks, but different solutions have distinct advantages. High pass rates usually accompany longer durations and higher costs, while low-cost solutions often require more manual review and task splitting. Before citing specific numbers, teams should verify the methodology and then test using their own task sets and record templates. Only by combining task boundaries, automated verification, tiered permissions, and human review can model capabilities be reliably translated into R&D efficiency.
