Long-Horizon Mobile Development Agent Evaluation and Selection Guide
This article explains why mobile development agent evaluation is shifting from code generation to long-horizon task delivery, defines four core metrics—pass rate, average completion, end-to-end latency, and cost per run—and uses anonymized 30-task results to illustrate quality-efficiency trade-offs. It also outlines a reproducible evaluation pipeline and a routing and quality-gating approach for balancing quality, speed, and budget in production.
1. From Short Tasks to Long-Horizon Tasks: Evaluation Focus Is Shifting
Mobile development agent evaluation is moving from “generate a block of compilable code” to “complete a deliverable long-horizon task.” A long-horizon task is usually not single-file completion. It involves a full chain: requirement understanding, cross-file retrieval, dependency analysis, multi-round modification, test repair, static checks, and final acceptance. This is closer to real R&D workflows and better exposes differences in planning, context management, tool use, error recovery, and cost control.
As a result, a single pass rate is no longer enough to describe agent capability. One solution may have a higher pass rate but significantly higher average latency and cost per run. Another may have a slightly lower pass rate but complete many iterations faster and at lower cost. For development teams, the real question is: under a given task complexity, delivery deadline, and budget constraint, which solution offers the better overall return?
In a recent public evaluation, 30 long-horizon tasks were used as observation samples. To keep the discussion neutral, this article uses Solution A and Solution B to refer to two anonymized agent solutions. The data is used only as a methodology example and does not constitute a recommendation or evaluation conclusion for any specific commercial product.
2. Four Core Metrics: Pass Rate, Completion, Latency, and Cost
1. Pass Rate: Delivered Proportion Under Strict Acceptance
Pass rate usually refers to the proportion of tasks that fully pass automated tests, static checks, build validation, and necessary manual review. It is the most intuitive quality metric, but for long-horizon tasks the pass rate tends to be low because failure in any link can cause the whole task to fail. When reading pass rates, teams should also consider task difficulty distribution, acceptance strictness, and retry strategy; otherwise, different leaderboards are not directly comparable.
2. Average Completion: A Quality Scale for Partial Success
Average completion measures how much of a task is completed, such as requirement coverage ratio, test pass ratio, or modified-file coverage. It is more granular than pass rate and reflects the ability to get “close to success.” For complex migration and cross-module refactoring tasks, completion can help teams judge whether an agent provides staged delivery value instead of looking only at a binary result.
3. Average Latency: End-to-End Delivery Time Cost
Average latency is end-to-end wall-clock time, including planning, tool calls, code generation, test execution, failure retries, and environment waiting. Long-horizon tasks may take tens of hours, so this should not be understood simply as “response speed.” Teams should separate time spent on human intervention waits, sandbox queuing, test execution, and model inference; otherwise, optimization efforts may target the wrong area.
4. Cost per Run: More Than Model API Fees
The cost of a single benchmark run should cover model calls, sandbox resources, storage, network, logs, retries, and manual review. Counting only model token fees underestimates the real cost. For budget-sensitive teams, a low-cost solution allows more exploration under the same budget. For high-value tasks, a high-cost solution may still be reasonable if it significantly improves first-pass success.
3. Interpreting Anonymized Results: Marginal Trade-Offs Between Quality and Efficiency
Across the 30 long-horizon task samples, Solution A achieved a 24.0% pass rate, 82.3% average completion, and 27.4 hours average latency, with a cost per run of about USD 317.4. Solution B achieved a 17.3% pass rate, 71.2% average completion, and 12.0 hours average latency, with a cost per run of about USD 9.9.
In a side-by-side comparison, Solution A is higher in pass rate and completion, but its latency is about 2.3 times that of Solution B, and its cost is about 32 times higher. Its pass rate is 6.7 percentage points higher, and its completion is 11.1 percentage points higher. This result shows that high-quality solutions carry clear marginal costs, and whether they are worth it depends on task value density. For high-value tasks such as cross-module refactoring, complex defect repair, and migration, higher completion may reduce manual rework. For small-scope fixes, batch style adjustments, and fast-feedback tasks, low-cost solutions are better suited to high-frequency trial and error.
It is important to emphasize that leaderboard results are not an absolute basis for selection. Task-suite composition, acceptance criteria, environment stability, and retry strategy all affect results. Teams should build their own hidden task set and layered reporting to avoid being misled by a single average.
4. Reproducible Evaluation Pipeline: From Task Design to Cost Accounting
To make evaluation useful for engineering decisions, consider building the following pipeline:
- Layer the task suite: Separate single-file fixes, cross-file changes, dependency upgrades, test repair, performance optimization, and security constraints, with enough samples in each layer.
- Automate acceptance: Rely mainly on unit tests, integration tests, type checks, static scans, and build results, with manual review used for sampled calibration.
- Isolate environments: Use containerized sandboxes with fixed dependency versions, network allowlists, and resource quotas, and record environment drift.
- Make the process observable: Save complete execution traces, including tool calls, failure categories, retry counts, latency distribution, and cost details.
- Report statistical confidence: When reporting means, also provide percentiles and confidence intervals to avoid amplifying small-sample fluctuations.
- Attribute costs: Account separately for model, sandbox, storage, network, and manual review to form a per-task cost profile.
evaluation_pipeline:task_suite:layers: ["single_file_fix", "cross_file_change", "dependency_update", "test_repair"]hidden_set_ratio: 0.2acceptance:gates: ["unit_test", "integration_test", "type_check", "static_scan"]manual_review_sampling: 0.1runtime:sandbox: "containerized"timeout_hours: 30retry_limit: 2reporting:metrics: ["pass_rate", "completion_rate", "p50_hours", "p95_hours", "cost_per_run"]cost_breakdown: ["model", "sandbox", "storage", "network", "review"]
5. Engineering Implementation: Balancing Quality, Speed, and Budget with Routing and Gates
In production, it is not advisable to send all tasks to a single solution. A more robust approach is layered routing: low-cost solutions handle high-frequency, low-risk, fast-feedback tasks; high-quality solutions handle complex, high-value tasks where failure is expensive; and critical changes introduce manual approval and mandatory quality gates. At the same time, set budget thresholds, timeout circuit breakers, failure fallbacks, and cache reuse to prevent long-horizon tasks from running out of control.
agent_router:policies:- name: fast_feedbackscope: "small-scope fixes and single-file changes"cost_budget_usd: 15timeout_hours: 4quality_gate: ["unit_test", "lint"]fallback: "standard"- name: high_assurancescope: "cross-module refactoring and migration"cost_budget_usd: 350timeout_hours: 30quality_gate: ["unit_test", "type_check", "security_scan", "review"]fallback: "manual_review"
At the infrastructure layer, general-purpose container platforms, object storage, logging, and monitoring/alerting services can support a reproducible evaluation environment, but the core remains task-suite design, acceptance criteria, and cost accounting. Only by connecting evaluation metrics to daily R&D monitoring and periodically regressing against a hidden set can benchmark data become a sustainable basis for engineering decisions.
