Evaluating Code Agents on Long-Horizon Mobile Development Tasks: Pass Rate, Completion, Latency, and Cost

    This article explains how code agents are now evaluated on long-horizon mobile engineering tasks rather than simple code completion. It breaks down pass rate, average completion, average latency, and cost, derives practical metrics for comparing configurations, and outlines how teams can build reproducible evaluations and combine heavy and lightweight agents into layered workflows.

    1. Evaluation shift: from code completion to long-horizon engineering tasks

    In the past, code generation was mainly evaluated with code completion benchmarks and single-turn question answering: the model received a function signature or a comment, generated a few lines of code, and the evaluation simply compared the output with a reference answer or ran unit tests. These evaluations mainly measured “fragment correctness.”

    Today, code agent evaluations have moved up to the level of a complete software engineering activity. In mobile development scenarios, an agent may need to create an app, implement features, migrate code, and convert applications. Such tasks do not have a single correct answer. The agent must read code, modify code, run builds, inspect errors, fix problems, and verify again. This “explore, implement, verify” loop may span dozens or even hundreds of interactions, which is why these tasks are called long-horizon tasks.

    Mobile development adds further difficulty: the build pipeline is long, incremental compilation can still take considerable time, runtime verification often depends on simulators or real devices, and UI automation testing itself can be brittle.

    Recently, the official developer account of a major mobile development platform published a batch of long-horizon agent benchmark results. One code agent achieved a 24.0% pass rate across 30 tasks, with an average completion rate of 82.3%, an average elapsed time of 27.4 hours, and a cost of about 317.4 USD per benchmark run, equivalent to about 2,123 CNY at the exchange rate reported at the time. Another configuration, using a lightweight model and a lightweight agent framework, achieved a 17.3% pass rate, 71.2% average completion, 12.0 hours average elapsed time, and about 9.9 USD per run, equivalent to about 66.2 CNY.

    The two configurations differ by roughly 30x in run cost, but their pass rates differ by only 6.7 percentage points. Teams should consider rerunning similar benchmarks against their own real requirements.

    2. What the four metrics measure

    • Pass rate: The proportion of tasks in a fixed task set that are judged as fully completed. Judgments are usually driven by executable acceptance cases, such as whether the build succeeds or whether functionality works as expected. It is a binary metric: a failure is a failure. It best represents “deliverable.”
    • Average completion: Measures how far incomplete tasks got. In the reported context, it usually refers to the average score across all tasks based on the proportion of functionality implemented. It is a continuous metric.
    • Average elapsed time: The actual time from task start to task end for a single task. It determines how long an evaluation takes and indirectly reflects the depth of the agent’s iteration.
    • Cost per run: The agent’s spend on model calls. It is often recorded together with metadata such as context length and number of tool calls.

    The elapsed time and cost among the first four metrics come from the same run log. The publisher also provided currency conversion, indicating that overseas teams also need localized cost interpretation. This is consistent with how enterprises approach technology selection and annual budget planning.

    In practice, it is useful to add three derived metrics:

    • Overall achievement rate = pass rate × average completion: Answers “how much can be delivered per task on average.” In the example above, 24.0% × 82.3% ≈ 19.8%, while the other configuration is 17.3% × 71.2% ≈ 12.3%.
    • Cost per achievement = cost per run ÷ (number of tasks × overall achievement rate): In the example above, this is about 317.4 ÷ (30 × 0.198) ≈ 53.4 USD, and 9.9 ÷ (30 × 0.123) ≈ 2.7 USD.
    • Effective throughput = parallelism ÷ single-task elapsed time: This determines the pace from idea to usable prototype.

    This means capability, latency, and cost must be compared in the same framework.

    3. Long latency and high completion reflect engineering trade-offs between exploration and verification

    A task taking 27 hours, with 82% completion but only about one quarter fully passing, is not simply “inefficient.” It can be understood in three ways:

    1. Completion measures how far the agent went; pass rate measures whether it truly worked. In offline scenarios, code that compiles and passes static checks can be considered a completed feature. In mobile development, however, features often need to be actually triggered on a simulator or real device. Issues such as UI element locator failures, permission dialogs blocking flows, or asynchronous loading timing problems may only appear at runtime. The gap between “passing in a simulator” and “failing on a real device” is a common reason for the difference between completion and pass rate.
    2. Long elapsed time often comes from deeper exploration and stricter self-verification. The agent converges on issues through more build-test loops, discovers defects earlier, and produces more explainable failure reasons. Its output is usually complete and well structured, but may still not meet the acceptance bar. Common causes include:
      • implicit requirements not being identified, such as state preservation after screen rotation or error messaging under weak network conditions;
      • leftover dependencies on old interfaces during migration;
      • platform differences introduced by conversion not being covered.
    3. High completion itself shows that the agent can “finish the journey.” If the agent gave up after an error in the third round, its completion rate would be low and its elapsed time short. If it can iterate all the way to near the finish line, it shows that long-horizon execution capability is already present. The remaining issues are requirement understanding and verification strictness, which are usually easier to improve through prompts, acceptance cases, and failure-mode analysis than by switching models.

    Where the boundary lies depends on the termination conditions set by the team. Termination conditions can include goal achievement, verification passing, budget exhaustion, or manual stoppage. The law of diminishing returns means that simply “thinking longer” is not necessarily worthwhile:

    Time investment Exploration adequacy Risk
    Stopping after insufficient exploration Low Requirement misunderstanding, migration omissions, output appears complete but requires rework
    Moderate exploration Medium Similar to long-latency strategy, but more cost-controlled
    Excessive exploration High Diminishing returns, context growth, cost overrun, and sometimes “the more changes, the worse it gets”

    4. Lower cost, faster feedback: the lightweight alternative

    The lightweight configuration follows a different approach:

    • A slightly lower pass rate does not mean unusable. When tasks terminate within budget, tasks that do not fully pass may still leave reusable code and modifications.
    • Iteration feedback is faster. An average of 12 hours per task means a shorter feedback cycle. Teams can iterate through “generate, inspect, feedback, regenerate,” quickly try ideas, and adapt to rapidly changing requirements.
    • Cost is only a small fraction of the heavyweight approach. Higher throughput makes batch tasks possible, such as filling in basic functionality for legacy projects or performing preliminary migration.
    • It can be engineered as a fallback. Teams can run multiple attempts within budget and select the best result, or let a stronger agent handle tasks that failed under the lightweight configuration. Combining the two creates a layered approach.
    • Two things require caution. Tasks with similar average elapsed time may generate code with token volumes that differ by several times, because token unit prices vary widely. Comparing only “pass rate ÷ cost” can be misleading; “overall achievement rate ÷ cost” is more appropriate.

    The two approaches can be combined as follows:

    Dimension Heavyweight long-horizon Lightweight fast
    Advantages Broad task coverage, complete output Fast feedback, low cost, batch execution
    Costs Longer elapsed time, higher unit cost Lower pass rate; complex requirements need fallback
    Suited to Production feature development, architecture migration, platform conversion Prototype validation, batch refactoring, initial exploration

    5. Build your own evaluation: turning public scores into decision evidence

    Public benchmarks are useful for creating a horizontal reference. However, task sets, acceptance standards, and run parameters differ, so scores cannot be directly copied into your own project. Each team should reproduce evaluations against its own technology stack.

    5.1 Task set construction

    • Source tasks from real requirements: Extract tasks from the requirement pool and historical defect tickets, covering high-frequency mobile development scenarios.
    • Complete metadata: Each task should record difficulty, type, initial code snapshot, natural language requirement, hidden acceptance cases, and budget limit.
    • Version traceability: Provide an initial project from a fixed snapshot to prevent the agent from changing the task definition and “proving itself correct.”

    Example task definition (field names are illustrative only):

    1. {
    2. "task_id": "feature-017",
    3. "category": "feature development",
    4. "difficulty": "hard",
    5. "snapshot": "feature-017-init.tar.gz",
    6. "prompt": "Add draft auto-save and restore to the existing app. If the app is reclaimed by the system and reopened, unsaved content should still be restorable.",
    7. "acceptance": ["build_pass", "draft_restore_test", "regression_suite"],
    8. "budget": {"max_turns": 200, "max_hours": 8.0, "max_cost": 3.0}
    9. }

    The prompt should only describe “what is wanted,” without writing acceptance details into it. Otherwise, the evaluation would shift from “understanding requirements” to “copying answers.”

    5.2 Environment isolation and reproducibility

    • Isolation: Each task should run in an independent containerized environment or simulator. Dependencies can be cached, but each run must start from a clean snapshot.
    • Fixed variables: Fix model versions, prompt templates, and random seeds. Run multiple times and report averages and dispersion. Randomness in long-horizon agent tasks cannot be eliminated, and a single run is insufficient to draw conclusions.
    • Archiving: Code artifacts, build logs, and test reports should be archived in object storage. Task states and metrics should be written to a database or data warehouse for cross-version comparison and traceback.

    5.3 Failure attribution

    Break “not passed” into actionable categories and track their proportions:

    • build failure;
    • feature test failure, further broken down to specific cases;
    • regression defects;
    • termination due to timeout or budget exhaustion.

    For configurations with strong scores, manually review several tasks to summarize reusable prompt rules and additional acceptance cases.

    5.4 Minimal harness implementation

    The following Python pseudocode only illustrates the evaluation flow and can be adjusted to match the team’s technology stack:

    1. from dataclasses import dataclass
    2. # 1) Hard guardrails: regardless of model behavior, resource and budget limits must not be exceeded
    3. @dataclass
    4. class Limits:
    5. max_turns: int = 200
    6. max_hours: float = 8.0
    7. max_cost: float = 3.0
    8. max_parallel: int = 4
    9. def guard_rail(state, limits: Limits) -> str | None:
    10. """Return termination reason; None means execution may continue"""
    11. if state.turns >= limits.max_turns:
    12. return "max_turns"
    13. if state.elapsed_hours >= limits.max_hours:
    14. return "max_hours"
    15. if state.cost >= limits.max_cost:
    16. return "max_cost"
    17. return None
    18. # 2) Single-task evaluation: pass/fail is determined only by executable acceptance, not by visual plausibility
    19. def evaluate_one(task, run_agent, check) -> dict:
    20. reason = guard_rail(guard_rail.init_state(), task.budget)
    21. if reason:
    22. return {"task_id": task.id, "passed": False, "completion": 0.0,
    23. "hours": 0.0, "cost": 0.0, "fail_reason": reason}
    24. result = run_agent(task) # The harness executes the task and checks guardrails step by step
    25. ok = check(result.workspace, task.acceptance) # Run hidden acceptance cases in an isolated environment
    26. return {"task_id": task.id, "passed": ok,
    27. "completion": completion_score(ok), # Convert to a proportional score
    28. "hours": result.elapsed_hours, "cost": result.cost,
    29. "fail_reason": None if ok else ok.reason}
    30. # 3) Aggregate metrics: four core metrics + overall achievement + cost per achievement
    31. def summarize(results: list[dict]) -> dict:
    32. n = len(results)
    33. pass_rate = sum(r["passed"] for r in results) / n
    34. completion = sum(r["completion"] for r in results) / n
    35. hours = sum(r["hours"] for r in results) / n
    36. cost = sum(r["cost"] for r in results)
    37. overall = pass_rate * completion
    38. return {"pass_rate": round(pass_rate, 4),
    39. "completion": round(completion, 4),
    40. "avg_hours": round(hours, 2),
    41. "cost_per_run": round(cost, 2),
    42. "overall_rate": round(overall, 4),
    43. # Cost per achievement: average cost required to deliver one complete task
    44. "cost_per_pass": round(cost / (n * overall), 2) if overall else None}

    5.5 Infrastructure

    An evaluation may run continuously for hours or even tens of hours, so the infrastructure must support long-running workloads:

    • Sandboxes should run on a container platform, with elastic scheduling of multiple parallel sandboxes.
    • Test device pools should manage simulators or real devices uniformly.
    • Object storage should be used to archive artifacts.
    • Message queues should decouple scheduling from execution.
    • Logging services and monitoring alerts should set thresholds for timeouts, cost anomalies, and consecutive failures.

    Enterprises can build this pipeline using general-purpose cloud products such as object storage, containers, message queues, and logging/monitoring, for example those offered by Baidu AI Cloud.

    6. Turn evaluation results into selection and collaboration strategies

    6.1 Three-layer selection framework

    • Ecosystem layer: look at the overall landscape. If an agent’s overall achievement rate is about 20%, higher than some general or lightweight solutions, this often comes from three factors: adaptation to the platform’s engineering pipeline, a tool ecosystem around mobile development, and optimization for long-horizon tasks. Repeated validation on your own task set is needed to determine whether the lead comes from engineering adaptation or model capability.
    • Scenario layer: match by task type. Production feature development and architecture migration require high completeness, so the long-latency configuration may be preferred. Prototype validation and batch refactoring are better suited to lower-cost solutions.
    • Budget layer: use guardrails instead of faith. Mechanisms such as task-level cost caps, timeout circuit breakers, and approval gates can be written into the same pipeline used for both evaluation and production.

    6.2 Layered collaboration: using engineering orchestration to offset model differences

    • Fast-slow combination: The lightweight solution first provides a runnable skeleton and initial implementation, while the heavyweight solution handles only tasks that did not pass.
    • Multiple sampling: Run multiple times within budget and select the best result to improve effective pass rate.
    • Expert review: High-risk changes, such as those involving payments, permissions, or data migration, should require manual review.

    Pseudocode example:

    1. def deliver(task, light, heavy):
    2. draft = light.run(task) # The lightweight agent first produces a draft: fast feedback, low cost
    3. if draft.passed: # A task counts as complete only if executable acceptance checks pass
    4. return draft
    5. # If it fails, the heavyweight agent takes over and receives failure information as context to avoid repeated mistakes
    6. final = heavy.run(task, context=draft.failure_report)
    7. return final
    8. # Batch tasks: use multiple small-budget attempts and select the best result instead of one large-budget run
    9. def batch_run(tasks, agent, attempts=3):
    10. return [best_of([agent.run(t) for _ in range(attempts)], key="completion")
    11. for t in tasks]

    6.3 Six principles for evaluation-driven tuning

    • Make requirements explicit: Write implicit constraints such as “backward compatibility” and “support for portrait and landscape modes” into task descriptions.
    • Make acceptance executable: Use builds, feature tests, and regression suites as criteria, rather than subjective impressions.
    • Manage long contexts: Use summarization and compression, retrieve information on demand, and avoid irrelevant history crowding out the context.
    • Classify tool permissions: Allow free read-only analysis, restrict write operations to sandboxes, and require manual confirmation for publishing operations.
    • Ensure artifact traceability: Record configurations and costs for each run, and archive them uniformly.
    • Close the operational loop: Feed online defects back into the evaluation task set, and continuously monitor drift in pass rate and cost.

    7. Conclusion

    The competition in mobile application development agents is shifting from “can it write code?” to “can it independently deliver complete functionality within a reasonable time and cost?” Public benchmarks put pass rate, completion, elapsed time, and cost in the same table. Their value is not only ranking, but also providing teams with a reference frame for calibrating their own evaluations.

    For development teams, the next step is not to chase a single highest score, but to turn real requirements into an executable task set, constrain costs with guardrails, and use layered collaboration to combine the strengths of different configurations. This makes every agent output verifiable, traceable, and measurable.