Mobile Development Agent Evaluation: Balancing Quality, Cost, and Time

    A practical look at mobile development agent benchmarks and how engineering teams can translate pass rate, completion, runtime, and cost into selection and governance decisions. It covers four essential metrics, stage budgets, L0–L3 task tiers, benchmark design, pipeline gates, and cost controls for production use.

    In mobile app development, AI coding agents are no longer limited to completing a few lines of code. They are beginning to take on long-chain tasks such as creating project skeletons, implementing feature modules, migrating legacy interfaces, and converting pages. Recently, the official developer channel of a major mobile ecosystem published a set of benchmark results for real development scenarios. Although the leaderboard itself carries vendor marketing overtones, the engineering problems it exposes are broadly relevant: among models that can all “do the work,” some achieve high average completion but require more than twenty hours and hundreds of dollars to finish one evaluation run; others have slightly lower pass rates and completion but can finish in about ten-plus hours at a cost of less than ten dollars. For engineering teams, the key is not chasing rankings. It is translating such results into actionable selection language: which tasks are worth handing to an agent, which tasks must retain human fallback, and how to use metrics to prevent a solution that “demos well but costs too much in production.”

    1. Turn the Leaderboard Back into an Engineering Problem: Four Metrics Are Indispensable

    Mobile development tasks differ from ordinary code generation. They are closer to a small software project: requirement understanding, project structure, resource files, build scripts, compatibility constraints, and runtime verification are all interlinked. A model may perform well on a single function but expose problems such as dependency chaos, broken page navigation, or missing permission declarations in a task like “generate a runnable app from an empty directory.” Therefore, an effective evaluation should look at at least four types of metrics.

    The first is task pass rate: whether the agent actually makes the app run and pass acceptance under defined conditions. It is closest to business outcomes, but it is also most easily polluted by “accidental success”: when the sample is too small, the difference between 24% and 17% may not be statistically stable. The second is average completion, which describes how much of the requirement was implemented even when the task did not fully pass. It is useful for measuring the value of a partial result, but it cannot replace runnability. The third is runtime. In long-horizon tasks, time represents interaction rounds, build counts, debugging loops, and team waiting cost. The fourth is single-run cost, including model calls, tool execution, sandbox builds, retries, and the amortized cost of human review. Looking only at pass rate overestimates “slow and expensive” options; looking only at cost brings unmaintainable partial work into the main branch.

    The overall experience can be written as an illustrative formula rather than a precise ranking:

    1. Effective output = pass rate × business acceptance weight
    2. + completion × handoverability coefficient
    3. - runtime × blocking cost
    4. - run cost × budget sensitivity

    This formula does not aim for absolute numbers. It emphasizes that weights change by scenario. For overnight batch migration of old projects, runtime is not sensitive, while completion and traceability matter more. For online assistance to new developers fixing bugs, response time and first-attempt success rate are more critical. For automated fixes in CI, cost guardrails must come first; otherwise, a retry storm can rapidly inflate the bill.

    2. Why “High Completion, Long Runtime” Is Not Necessarily a Defect

    Anonymized, one class of high-quality samples shows completion above 80%, a runtime of more than twenty hours, and a single-run cost of hundreds of dollars. Intuitively, it seems inefficient, but in complex migration tasks this curve may come from more cautious planning: read the build configuration first, then map page navigation and data flow, fill in permissions, resources, themes, and compatibility branches, and finally repeat build verification. Its value is not speed but reducing rework.

    Mobile engineering has several places where errors are amplified. First, declarative UI and imperative lifecycles coexist; missing one state save can cause data loss after screen rotation. Second, third-party SDKs and system APIs vary widely by version; a successful compile does not mean the app works on lower-version devices. Third, resource naming, internationalization, dark mode, and accessibility attributes are often not explicitly stated in requirements, yet they affect release quality. If a high-completion sample makes fewer mistakes on these implicit constraints, even if it takes longer, it may reduce the cognitive load of human review.

    But long runtime also signals engineering risk: when an agent falls into a “modify—build—fail—modify again” loop, cost does not grow linearly. It rises in steps with retries. A reasonable approach is to set stage budgets for long-horizon tasks: requirement clarification no more than 10% of total rounds, scaffolding and dependency resolution no more than 20%, core feature implementation about 40%, and verification and wrap-up about 30%. Once a stage exceeds its budget, a degradation strategy should be triggered: narrow the scope, generate a runnable minimal closed loop, and output a handover checklist instead of continuing blind retries.

    3. Why “Low Pass Rate, Low Cost, Fast Response” Also Has Strategic Value

    Another class of samples has a pass rate below 20% and completion around 70%, but runtime is about half a day and cost is less than ten dollars. If judged only by leaderboard rank, it is clearly behind. If positioned as an “exploratory assistant,” it is very practical. Suitable scenarios include generating prototype interfaces, comparing two implementation skeletons, quickly producing migration drafts, writing migration risk lists for legacy code, and previewing feature breakdowns locally. It may not deliver the final artifact, but it can significantly shorten the time people need to make decisions.

    The core here is distinguishing “authoring the main branch” from “assisting with proposals.” For main-branch code, low completion means high takeover cost and may not be worth it. For the exploration phase, a low-completion but clearly structured draft can prevent the team from getting stuck in details too early. From an engineering perspective, tasks can be divided into four layers: L0 Q&A and explanation, L1 snippet-level completion, L2 module-level implementation, and L3 project-level creation/migration/conversion. Low-cost fast models are suitable for L0–L2 and for L3 previews. High-completion, high-cost models are more suitable for constrained L3 execution and must be paired with human gates.

    One caution: low-cost models can encourage a “let’s try it” culture. Developers repeatedly submit similar prompts, using quantity to compensate for quality. The total cost may exceed one high-quality run, and it also creates many intermediate branches that need cleanup. Therefore, even if a single run is cheap, teams should set task-level budgets, duplicate-prompt detection, failure-mode classification, and cache reuse to avoid turning a cost-saving tool into a waste-generating tool.

    4. How Benchmarks Should Be Designed to Serve Selection Rather Than Hype

    Leaderboards often choose long-horizon tasks, a unified agent, and a fixed toolchain. This helps horizontal comparison, but it still differs from real teams. A more reusable evaluation should expand across three dimensions.

    The task dimension should cover typical mobile development pain points: empty project creation, feature increments, cross-version migration, UI paradigm conversion, build script repair, test completion, and performance regression localization. Each task type should have clear acceptance, not only “it compiles” but also “it starts without crashing, critical paths are reachable, resource references are complete, permissions are minimized, basic accessibility labels exist, and low-end devices do not ANR.” Runnability should be checked by scripts rather than human eyes, for example installation, startup, traversal of core pages, screenshot comparison, log assertions, and build-time limits.

    The environment dimension should fix the system version, build tools, dependency repository snapshot, and network policy; otherwise, “score differences” may simply come from dependency source jitter. For long-horizon tasks, record trajectories: planning rounds, tool-call distribution, failure types, recovery methods, and final diff size. Without trajectories, pass rate is only a black-box result. With trajectories, teams can judge whether the model implemented stably or passed by luck.

    The statistical dimension should avoid conclusions from small samples. At minimum, report mean, median, confidence intervals, and failure clusters by task type. Run high-variance tasks multiple times. Put “partially complete but valuable for takeover” into a separate category. For enterprises, it is best to use public benchmarks as external calibration while building a private regression set from their own codebase to protect business-specific risks such as legacy architecture, internal component standards, and compliance requirements.

    5. Implementation Strategy: Put Capabilities into the Pipeline, Not Just the Chat Box

    The biggest fear when using agents in mobile R&D is that they “can chat but cannot do, or can do but cannot be merged.” A safer path is layered gates.

    Before requirements enter the pipeline, use a structured template to constrain output: target pages, user paths, inputs and outputs, boundary conditions, compatibility scope, and acceptance commands. The closer the task description is to a testable specification, the less likely long-horizon execution will drift. An illustrative template is below:

    1. task: Add an "offline favorites" feature
    2. scope: list page, detail page, local data source
    3. acceptance:
    4. - Open saved details without network
    5. - Favorite state persists across process restarts
    6. - No access to sensitive data without authorization
    7. constraints:
    8. - Minimum supported version unchanged
    9. - Startup time increase within threshold
    10. deliverable: Buildable project + change notes + risk list

    During execution, use a rhythm of “plan—minimal closed loop—expand—verify.” First ask the agent to output an implementation plan and impact surface instead of rushing to change code. Then require a runnable minimal path. After it passes, expand boundary scenarios. Finally, run builds, static checks, unit tests, and smoke tests together. For high-risk operations such as deleting resources, modifying signatures, changing major dependency versions, or migrating storage structures, human confirmation should be required by default.

    After merge, feed results back into the evaluation set: successful patterns become templates, and failure patterns are classified as prompt defects, tool defects, insufficient context, or model capability boundaries. This way, selection decisions shift from “one impressive run” to “long-term repeatability.”

    6. Cost and Governance: The Impossible Triangle of Speed, Quality, and Budget

    Engineering experience repeatedly shows that speed, quality, and budget are hard to maximize at the same time. A high-completion, long-runtime solution is like a senior contractor: slow and expensive, but with complete delivery. A low-cost, fast solution is like an idea generator: cheap and agile, but it needs an owner to review it. Enterprises should not ask “which model is strongest” but “at which gate level, for which task type, and within which budget range is the cost of failure lowest.”

    A simple decision matrix can be established: L0/L1 prioritize speed and cost and allow lower completion; L2 prioritizes completion and testability; L3 prioritizes pass rate, auditable trajectories, and human takeover cost. For tasks blocking the mainline, choose slow and stable. For exploratory tasks, choose fast and disposable. For high-risk areas involving funds, privacy, compliance, medical, or financial scenarios, even a high pass rate should not exempt human review and least-privilege execution environments. When running evaluations in the cloud, teams can use object storage for snapshots, logging services for trajectories, monitoring and alerting for budget spikes, and container platforms to isolate build sandboxes. The focus should be on processes and metrics rather than binding to any single model.

    Conclusion

    The most valuable part of a public leaderboard is not ranking models first and second. It is pulling mobile agent development from a “demo culture” back to an “engineering culture.” High completion with long runtime reminds us that the hidden cost of complex migration is high. Low pass rate with low cost reminds us that exploration and mainline work must be treated differently. Mature teams treat pass rate as a gate, completion as an asset, runtime as a blocking signal, and cost as a guardrail, then continuously calibrate with their own task sets. The next time you evaluate any intelligent coding capability, ask less “what is the score?” and ask three questions: can we take over when it fails, is the trajectory auditable, and does total cost decline after it enters the pipeline? Only solutions that can answer these three questions deserve a place in the production mainline.