Evaluating AI Coding Models for Mobile Development: Success Rate, Time, and Cost Trade-offs

    Benchmarks based on single functions fail to capture how AI coding models behave in real projects. This article explains how to evaluate coding models on engineering tasks in mobile development, covering four core metrics—pass rate, average completion, wall-clock duration, and cost per run—along with the trade-off between high completion and long runtime, cost-per-valid-output analysis, and practical steps for building an internal evaluation framework.

    1. Why Evaluate Coding Models with Engineering Tasks

    Early code generation benchmarks mostly relied on single-function problems: given a description, the model had to produce code that passes unit tests. These problems are good for testing algorithmic understanding, but they say little about how work happens in a real project. In practice, a model has to locate files in an existing codebase, understand module dependencies, modify multiple files, resolve build errors, and keep the UI, permissions, and configuration consistent.

    Task-based evaluation raises the unit of assessment from “function” to “task.” A task typically includes a requirement description, an initial codebase, acceptance tests, and completion criteria. The evaluation system records whether the model passed acceptance, how many sub-goals were achieved, how long the work took, and how many resources were consumed. This closely mirrors how teams judge deliverables.

    Mobile app development is a typical scenario. Common tasks include creating a new project, implementing pages and business logic, upgrading framework versions, migrating to a new language standard, and adapting to different screen sizes and OS versions. These tasks share common traits: heavy dependencies, broad change surface, and the need for runtime verification.

    2. Four Core Metrics

    2.1 Pass Rate

    The pass rate is the share of tasks that pass all acceptance tests: pass rate = tasks passed / total tasks. It is the strictest metric and reflects whether a model can independently deliver a working result.

    The pass rate is highly sensitive to acceptance criteria. The more comprehensive the test coverage, the closer the pass rate reflects real quality. When coverage is insufficient, the pass rate overestimates performance, so evaluation designers should prioritize adding tests for edge cases and error paths.

    2.2 Average Completion

    Completion measures the proportion of sub-goals correctly implemented within a task. If a task contains five features and the model implements four correctly, completion is 80%—but because one feature is missing, the task counts as failed for pass-rate purposes.

    Completion distinguishes “nearly done” from “badly off track.” For deciding whether a model can serve as an assistant that humans finish up after, this metric is more informative than the pass rate.

    2.3 Average Duration

    Duration is the wall-clock time from receiving the task to producing the final result, including model inference, tool calls, compilation and execution, and self-correction loops. Duration is strongly affected by an agent’s operating strategy. Systems that repeatedly run tests and fix errors tend to achieve higher completion, but take longer.

    2.4 Cost per Run

    Cost includes model API charges, compute resources consumed by tool execution, and any necessary storage overhead. The same model can vary several-fold in cost across different agent frameworks. Therefore, when comparing results, you must record the full run configuration; otherwise the conclusions are not comparable.

    3. The Engineering Trade-off Between High Completion and Long Duration

    When a result shows high completion but long duration, it usually means the system invested more iterations in the task. Common reasons fall into three categories:

    • Reading the codebase more frequently to build a more complete context;
    • Running more verification steps before submitting a result;
    • Continuing to fix errors over multiple rounds instead of stopping early.

    This strategy suits scenarios with high correctness requirements where asynchronous waiting is acceptable—for example, overnight batch migrations or generating first drafts of large refactors. Interactive development values responsiveness; a task that takes too long breaks a developer’s focus, so a faster option may be the better fit.

    There is no universally optimal trade-off. Before choosing, teams should answer three questions: Can the task run asynchronously? How costly is a failure? Is there enough time for human review?

    4. Measuring Cost-Effectiveness per Valid Output

    Comparing total cost or average duration alone often leads to misleading conclusions. A more practical approach is to compute cost per unit of valid output, using three metrics:

    • Cost per pass = total cost / number of tasks passed
    • Effective throughput = tasks passed / total duration
    • Human rework cost = number of failed tasks × average cost per manual fix

    If you include human rework cost, then total cost = model cost + human rework cost. Only with this accounting can comparisons between options reflect the true expense.

    For example, one option may have a low unit price but a low pass rate: the model cost is small, yet it may generate substantial rework. Another option has a higher unit price but a higher pass rate, and ends up with a lower cost per pass. Only by computing the total cost of both can you make a sound decision.

    5. Steps to Build an Internal Evaluation Framework

    1. Assemble a task set: Layer tasks by type (new features, modifications, migrations, bug fixes) to ensure coverage of the team’s real workload.
    2. Write acceptance tests: Each task should come with automatically runnable tests, including edge cases.
    3. Fix the run configuration: Record the model version, prompt templates, tool permissions, and timeout settings.
    4. Run multiple times and take statistics: Agent outputs are stochastic; a single run is not enough to support conclusions.
    5. Produce a metrics report: Present pass rate, completion, duration, and cost together, along with a breakdown of failure causes.

    The following example shows how to aggregate these metrics. The data structures are only for illustrating the calculation; adjust the field semantics to match what your team actually records.

    1. from dataclasses import dataclass
    2. @dataclass
    3. class RunResult:
    4. passed: bool # Whether all acceptance tests passed
    5. completion: float # Completion, in the range 0.0 to 1.0
    6. duration_h: float # Duration, in hours
    7. cost: float # Model and compute resource cost
    8. def summarize(results: list[RunResult], rework_cost_per_fail: float = 0.0) -> dict:
    9. total = len(results)
    10. if total == 0:
    11. raise ValueError("Result set is empty")
    12. passed = sum(1 for r in results if r.passed)
    13. model_cost = sum(r.cost for r in results)
    14. rework_cost = (total - passed) * rework_cost_per_fail
    15. total_cost = model_cost + rework_cost
    16. return {
    17. "pass_rate": passed / total,
    18. "avg_completion": sum(r.completion for r in results) / total,
    19. "avg_duration_h": sum(r.duration_h for r in results) / total,
    20. "total_cost": total_cost,
    21. "cost_per_pass": total_cost / passed if passed else float("inf"),
    22. }

    6. Practical Recommendations

    • Start with a small task set to validate the evaluation process, and expand only after results are stable.
    • For high-risk modules involving security, payments, or permissions, keep human review in place no matter how good the metrics look.
    • Re-run evaluations after any model version or agent configuration change; do not carry over old conclusions.
    • Read the metrics alongside code review results and test coverage; do not make decisions based on model metrics alone.

    7. Limitations of Evaluation Results

    Benchmark results only represent performance within the task set used. The more the task distribution differs from a team’s actual workload, the less relevant the findings. In addition, cost data depends on billing methods and exchange rates. When comparing across teams or over time, use a consistent pricing basis and note when and under what configuration the data was collected—avoid putting incomparable numbers side by side.