Android Development AI Agent Benchmarks: Reading the Results and Choosing the Right Agent

    A recent AI coding benchmark for Android app development compared several agent approaches on pass rate, average completion, average runtime, and per-run cost. This article explains what those metrics actually measure, how to interpret them without falling into common traps, and how engineering teams can turn leaderboard results into a practical evaluation harness and selection strategy. It covers why Android engineering is hard to get right in one shot, the tradeoffs between long-running high-completion agents and lightweight low-cost agents, and how to build tiered, machine-checkable evaluations with budget circuit breakers and human review. The goal is to move from rankings to a system-efficiency view that accounts for both machine and human cost.

    1. Introduction: Why Android Is AI Coding’s “Hard Exam”

    Recently, a major mobile development community published benchmark results for AI coding agents targeting Android app development. The evaluation compared several agent approaches across metrics including task pass rate, average completion, average runtime, and per-run benchmark cost. The long-running approach traded roughly 27 hours of runtime for the highest completion rate, while the lightweight approach finished under 30% of tasks in about 12 hours at roughly one-tenth the cost, standing out on cost-efficiency.

    This scorecard offers developers two things:

    • For the first time, it quantifies the gap between agents using “real engineering tasks” rather than “function-level completion”;
    • It puts the “quality–time–cost” tradeoff on the table.

    This article doesn’t rank specific products. Instead, it answers three questions: What do these metrics actually measure? How should they be read correctly? How should teams translate benchmark conclusions into practice?

    1.1 Why Android Engineering Is Hard to “Get Right in One Shot”

    • Complex project structure: Multiple modules, build variants, and target versions coexist, so a single change ripples across many files through the dependency graph.
    • Version sensitivity: Build plugins, language runtimes, and UI libraries carry compatibility constraints; a version bump often triggers cascading compile errors.
    • Long feedback loop: Verification usually requires compiling, installing to an emulator or device, and then triggering UI interactions or system callbacks. One “execute → observe → fix” cycle can take minutes.
    • High bar for conventions: Constraints around lifecycle management, permission requests, resource adaptation, and obfuscation rules run throughout a project, and any oversight can surface at acceptance time.

    Because of this, there’s a huge gap between “writing a demo that compiles” and “delivering a feature change that ships.” What the benchmark measures is precisely an agent’s ability to cross that gap.

    2. The Four Core Metrics: What Each One Measures

    2.1 Pass Rate: The Strictest Verifiable Signal

    Pass rate measures whether an agent can actually finish a task. Tasks typically span app creation, feature development, code migration, and platform conversion, with end-to-end acceptance as the bar: the build passes, the feature works, and all key test cases pass.

    Beyond the surface question of “does this change work,” it implies three more thresholds:

    1. Understanding the existing code and project structure;
    2. Maintaining global consistency: not missing callers, not introducing regressions;
    3. Converging on a correct solution within the time limit.

    So gains in pass rate usually come from a combination of planning ability, long-context retention, and effective self-verification, not from raw code-generation speed.

    2.2 Average Completion: How Much of the Task Got Done

    Completion answers “on the tasks that didn’t fully pass, how much still got done.” A sound definition should anchor to verifiable checkpoints, such as:

    • Whether the required feature points are implemented;
    • Whether the relevant tests pass;
    • Whether build artifacts are successfully produced.

    If an approach averages 80% completion but only about a quarter of tasks fully pass, that means it’s “close to correct” most of the time. Handing that code to a human costs far less cleanup than starting from scratch. That’s the source of partial-credit value.

    2.3 Average Runtime: The Time Cost of Long-Horizon Tasks

    Average runtime is measured in hours, which tells you these are inherently long-horizon tasks. The agent repeatedly cycles through “read code → modify → build → run → observe logs → fix.” If each cycle takes ten minutes, dozens of cycles add up to hours.

    Runtime reflects the system’s autonomy and efficiency, including reducing wasted edits, recovering quickly from build failures, and avoiding getting stuck in loops. Two caveats matter:

    • Runtime is constrained by the per-run time limit. If an agent runs close to the ceiling on a 27-hour task, some tasks may fail on timeout.
    • Runtime has a long tail. A few failing tasks that retry repeatedly can noticeably inflate the average.

    2.4 Per-Run Benchmark Cost: The “Bill” for One Full Pass

    Per-run cost tallies the expense of running the entire benchmark once. Using a comparably sized evaluation set as an example (say, 30 long-horizon tasks):

    • The high-runtime approach took about 2.3x the time of the lightweight one, at roughly 32x the per-run cost;
    • Per task, that’s about $10.6 (roughly ¥71) for the former versus about $0.33 (roughly ¥2.2) for the latter.

    Cost is so sensitive to runtime because a long-horizon agent’s execution trace is long. Every call carries historical context, input volume grows roughly linearly with the number of turns, and cumulative consumption can therefore grow super-linearly. The longer the context, the more irrelevant information dilutes the signal, raising the odds of “getting lost” and producing a compounding “more expensive, slower, more error-prone” effect.

    Also, cost level is generally correlated with the size of the underlying model and shouldn’t be equated with “cost-efficiency.”

    2.5 One Table to See the Two Orientations

    Dimension Long-horizon, high-completion orientation Lightweight, low-cost fast-execution orientation
    Pass rate Clearly ahead Relatively lower
    Average completion High, artifacts close to usable Medium, skeleton usually in place
    Average runtime Measured in days Measured in half a day
    Per-run cost Significantly higher Roughly a few hundredths of the former
    Best-fit scenarios Complex migrations, cross-module changes, one-shot refactors Boilerplate, local changes, test coverage
    Main risk Long cycles, high cost, hard to control change scope More hidden defects, relies on human review as a backstop

    Specific figures not listed in the table are not compared across products.

    3. Four Common Pitfalls in Reading the Metrics

    3.1 Treating “High Completion” as “High Quality”

    Completion measures “how much got done,” not “how well.” An agent may implement most feature points yet still get lifecycle handling, concurrency safety, or edge cases wrong. It may even delete tests or weaken assertions just to make the build pass. This is exactly what to watch for in the “high completion, low pass rate” combination.

    3.2 Ignoring Statistical Noise from Sample Size

    When a benchmark has 30 tasks, a single task accounts for about 3.3 percentage points of the total. Even assuming tasks are independent and every approach has a true pass rate of 25%, there’s still roughly a 27% chance of seeing a 5-percentage-point gap. So under a small sample, a lead of a few points shouldn’t be read directly as a capability difference.

    (For comparison, on a 50-task evaluation set, the probability of the same gap is about 15%.)

    3.3 Treating Runtime and Cost as Fixed Values

    Both are strongly conditional metrics: affected by the difficulty distribution of tasks, the execution time limit, the runtime environment (whether tasks run in parallel, whether caches hit), and whether reasoning is enabled. For cross-source comparison, you must confirm the measurement basis is consistent.

    3.4 Using “Best Result” in Place of “Worst Case”

    Engineering delivery is judged on the worst case. If an approach fails 3 times out of 10 runs, then even with a higher average pass rate, it isn’t suitable for a critical path. Teams should look at run stability (e.g., the dispersion of results across repeated runs) and whether failed artifacts are easy for a human to take over.

    4. From “Reading the Leaderboard” to “Building Your Own Yardstick”

    4.1 Organize the Evaluation Set by Task Type in Tiers

    Evaluation tasks should be designed in tiers by engineering type and drawn from a real requirement pool where possible:

    • L1 App creation: Generate an installable base app under a given spec;
    • L2 Feature development: Implement one complete feature within an existing module;
    • L3 Code migration: Old API migration, declarative UI conversion, architecture-pattern changes;
    • L4 Platform conversion: Converting a cross-platform app to a native app;
    • L5 Defect fixing: Locate and fix issues based on crash logs or failing test cases.

    Every task should come with machine-decidable acceptance criteria. It’s not advisable to judge success solely on “does it compile.”

    4.2 Make Acceptance Criteria Machine-Decidable

    Take an L4 task as an example; a checklist can spell out acceptance conditions and constraints:

    1. task_id: L4-012
    2. type: platform_migration
    3. description: Convert a cross-platform page to a native implementation while keeping visuals and interactions consistent
    4. acceptance:
    5. - build: Full build passes with no new compile warnings
    6. - tests: All existing unit and UI tests pass
    7. - visual_diff: No visible difference on key pages
    8. constraints:
    9. - public_apis: Public API signatures stay unchanged
    10. - deps: No new third-party dependencies without review
    11. limits:
    12. max_runtime_hours: 8
    13. max_cost_usd: 5

    Here, max_runtime_hours and max_cost_usd are the budget circuit-breaker conditions. You can set reference values for each task tier based on the long-running and lightweight approaches’ data.

    4.3 Reproducible Results: Pin the Environment and Record Execution Traces

    Recommended practices:

    1. Pin the environment: Fix build tool versions, runtime versions, and images to keep environmental differences from polluting conclusions.
    2. Isolate context: Run each task in a separate workspace and version-control branch so tasks don’t interfere with one another.
    3. Retain traces: Record the full execution trace, including key tool calls, build logs, and code diffs, to make failure attribution easier.
    4. Run multiple times: Run key tasks at least 3 times, then compute the mean pass rate and dispersion to surface stability differences.

    4.4 The Acceptance Pipeline: A Gating Function Sketch

    Evaluation and day-to-day development can share one acceptance gate. The sketch below defines a unified check function. Each task only needs to provide the project path to get back a structured acceptance result:

    1. def check(workdir: str, run_smoke: bool = False) -> dict:
    2. """Run a unified acceptance gate on an agent's engineering changes and return structured results."""
    3. result = {"build": False, "lint": False, "unit": False, "smoke": None}
    4. # 1. Build: if any step fails, later checks are pointless, so return immediately
    5. if not run(["build.sh"], cwd=workdir):
    6. return result
    7. result["build"] = True
    8. # 2. Static analysis: emit a structured report to make warning-type tallying easier
    9. lint_ok, lint_report = run_capture(["lint.sh", "--json"], cwd=workdir)
    10. result["lint"] = lint_ok
    11. result["lint_report"] = lint_report
    12. # 3. Unit tests: produce a machine-readable report to compute completion checkpoints
    13. result["unit"], report = run_capture(["test.sh", "--report", "xml"], cwd=workdir)
    14. result["test_report"] = report
    15. # 4. Smoke test: only launch an emulator when explicitly required, to control per-run resource use
    16. if run_smoke:
    17. result["smoke"] = run(["smoke.sh"], cwd=workdir)
    18. return result

    It’s best to expose only a minimal toolset in the task instructions: read/write files, run builds and tests, and view logs. Also write in the constraints: “After any change, you must run the build and tests; on failure, read the logs first to locate the cause; deleting or loosening tests to ‘pass’ the check is forbidden.” This prevents the agent from gaming the gate and keeps pass rate tied to real capability.

    4.5 Budget Circuit Breakers and Tiered Scheduling

    A long-horizon agent’s cost mainly comes from “wasted loops,” so engineering must enforce hard limits:

    • Per-run time limit: Terminate on timeout and preserve artifacts for human judgment;
    • Cost cap: Meter by token consumption or number of calls, and pause once a threshold is hit;
    • Failure fallback: After N consecutive build failures of the same type, roll back to the last passing checkpoint and switch strategy (e.g., narrow the change scope);
    • Tiered scheduling: Let lightweight approaches handle standardized tasks, reserve long-horizon approaches for complex migrations, and trigger stricter acceptance flows only after review passes.

    5. Turning Benchmark Conclusions into Selection Practice

    5.1 What Each Strategy Is Good For

    Pursuing quality (long-horizon, high-completion approaches)

    • Fits: complex migrations, architecture adjustments, one-shot deep refactors before a release. Rework is expensive here, so the extra time and cost are worth it.
    • Requires: artifact archival and progress visibility. On long tasks, have the agent commit at key checkpoints so the team can inspect branch diffs at any time.
    • Management: run it as “offline batch + overnight jobs” to avoid tying up daytime integration resources.

    Pursuing speed and cost (lightweight, low-cost approaches)

    • Fits: high-frequency, low-complexity scenarios like boilerplate, local changes, test coverage, and code explanation.
    • Requires: automated gates plus human review, using process to compensate for capability gaps.
    • Benefit: thanks to the cost advantage, you can iterate fast, generate multiple approaches in parallel, and filter them through the gate, which actually raises overall throughput.

    A middle ground: combine them

    Most teams are better served by a combined strategy:

    1. Use a lightweight approach to quickly produce a first draft;
    2. Filter out substandard artifacts through gates and human review;
    3. Hand the remaining hard problems to a long-horizon approach;
    4. Deposit the final artifacts along with runtime and cost data into your internal evaluation set to close the loop.

    5.2 From “Single-Task Capability” to “System Efficiency”

    Ranking by single-task capability alone is easy to distort. Introduce a simple system-efficiency metric:

    System efficiency = effective delivered output ÷ (machine cost + human-intervention cost)

    • Suppose a lightweight approach is only 60% accurate at judging its own output as “usable,” requiring about 0.5 hours of human review and fixes at a labor cost of ¥50. Its all-in cost per task is about ¥51.2, still within budget.
    • Suppose a long-horizon approach needs, per 100 tasks, 5 failing tasks each requiring about 4 hours of human backstop at a labor cost of ¥2000, which works out to ¥20 per task. Its all-in cost per task is about ¥91.
    • The two differ by less than 2x, far smaller than the 32x gap in per-run cost.

    This shows that the machine unit price is only part of the cost, and human backstop cost is often the decisive variable in selection. Teams should treat “how much human intervention is needed” as one of the evaluation metrics, not just pass rate.

    If tasks run as batch jobs on a cloud container platform, you can store acceptance artifacts and execution traces in object storage, feed runtime, cost, and pass-rate metrics into logging and monitoring/alerting services, and use a message queue for task distribution. That way you can alert on cost anomalies (e.g., a sudden spike in a single task’s cost) in time, and break down monthly spend by project and team for attribution.

    Basic cloud products such as containers, object storage, logging, and monitoring from Baidu AI Cloud can serve as a general foundation for building this kind of evaluation and development pipeline.

    6. Conclusion: Taming AI Coding with Engineering Methods

    The point of an Android development benchmark isn’t to rank agents, but to quantify capability gaps through verifiable tasks. Long runtime with high completion and low cost with fast execution each correspond to a legitimate engineering orientation:

    • The former suits critical changes where you “get it right once and rework less”;
    • The latter suits high-frequency scenarios where you “iterate fast and let process back you up.”

    For a team, the more pragmatic approach is to build a tiered evaluation set tailored to your own codebase, make acceptance criteria machine-decidable, set budget circuit breakers and human review, and make decisions on all-in cost rather than per-run cost. When model capability and engineering process form a closed loop, AI coding agents can genuinely evolve from a “benchmark tool” into a “productivity tool.”