Long-Horizon Coding Agents Benchmarked on Mobile Projects: Reading the Metrics and Weighing the Cost Trade-offs

    Long-horizon coding agents can now work autonomously across an entire codebase for hours, and mobile development is one of the hardest proving grounds for them. This article explains what long-horizon coding benchmarks actually measure and how to interpret their core metrics—pass rate, average completion, average wall-clock time, and cost per run—along with the often-hidden fifth dimension of stability. Drawing on a public A/B comparison as a case study, it shows how to weigh a high-completion, high-cost approach against a low-cost, fast one, then lays out a five-step framework for building your own evaluation system: task-set design, isolated execution, declarative task definitions, metric computation, and reporting gates. It closes with practical guidance on converting benchmark scores into business decisions through risk-based task routing, and the common pitfalls to avoid.

    1. Background: the task granularity of coding agents keeps moving up

    The capability boundary of LLM-assisted coding continues to expand. Early on, it could only offer function-level completions inside an editor; later, it could modify a single file; today, it can work autonomously across an entire codebase for hours—scaffolding an application, implementing features that span multiple files, migrating a legacy implementation to a new framework, and even converting one platform’s UI into another platform’s implementation.

    Mobile engineering is a typical proving ground for these capabilities, for four reasons:

    1. Long task chains: adding a single screen often requires coordinated changes to the data model, business logic, UI components, theme configuration, and tests—and a mistake in any one of them breaks the build.
    2. Automatically verifiable results: compilation, static analysis, unit tests, and UI automation cases can all run inside a container and emit clear pass/fail signals, which makes them ideal for benchmarking.
    3. High environment rebuild cost: you need to install dependencies, download toolchains, and start emulators or rendering engines, so a single task runs long and consumes heavy resources.
    4. Heavy legacy baggage: many requests are migrations and refactors that must make minimally invasive changes within an existing code style and an existing framework version, which tests the model’s context management and self-correction.

    These traits also explain the basic design of long-horizon coding benchmarks: assemble a batch of tasks that resemble real work, let the agent execute them autonomously in an isolated environment, then produce a quantitative score based on build and test results. Benchmarks aimed specifically at mobile engineering already exist, and they focus on application creation, feature development, code migration, and app conversion. In a recently published set of results, one approach reached a pass rate of roughly 1.4x its baseline, but at about 2.3x the average time and roughly 32x the cost per run.

    This article does not discuss any specific product or leaderboard ranking. Instead, it focuses on three questions with broader value: what long-horizon coding benchmarks actually test, how these metrics should be interpreted, and how development teams can use them to make technology choices and drive engineering adoption.

    2. Reading a long-horizon coding benchmark: four core metrics

    2.1 Pass rate: can it actually ship

    Pass rate is the proportion of tasks in a given task set whose final output satisfies all acceptance conditions. Acceptance is usually a binary judgment—for example, the build succeeds, all tests pass, or human review finds no blocking issues. It answers the question “can this be delivered,” and it directly determines how many requirements can be handed to an agent for independent completion, which makes it the most business-relevant metric in long-horizon evaluation. The problem is that the stricter the acceptance criteria, the lower the pass rate. At a scale of 30 tasks, each task accounts for about 3.3 percentage points, so the success or failure of a single task makes the pass rate jump noticeably; leaderboards based on small samples need to be read with particular caution.

    2.2 Average completion: is the gap at the start or the finish

    Average completion measures how far the agent still is from the task goal. Two common approaches:

    • Checkpoint method: break the task into several verifiable sub-goals (such as interface definitions, core logic, UI implementation, and test coverage) and count the proportion of sub-goals achieved;
    • Continuous scoring method: a scoring model or a set of rules gives a score between 0 and 1 against the requirement description.

    Whichever approach is used, what it captures is “partial progress.” A high-completion agent, even when it fails, often produces output that a human can use as a starting point to finish; a low-completion agent’s output may need to be thrown away and rebuilt. It also reflects the agent’s “sense of direction”—whether it can keep the goal from drifting across a long task chain.

    2.3 Average time: the time cost of long-horizon tasks

    Long-horizon agents typically take hours, because they repeatedly cycle through “read code → modify → run build and tests → analyze errors → modify again.” Two measurement conventions matter when interpreting it:

    • Per-task time: the time a single task takes from start to finish;
    • Cumulative runtime: the sum of execution time across all tasks.

    The former affects the developer’s waiting experience; the latter determines the peak and total duration of resource usage.

    2.4 Cost per run: converted from token usage

    An agent’s cost comes mainly from model calls. It can be priced separately for input, output, and cache-hit tokens, then multiplied by the corresponding unit price, where cache hits usually enjoy a lower price. In long-horizon tasks, the context keeps accumulating as tool calls proceed, so input tokens are often the main cost driver. As a result, “many calls” combined with “large per-call context” makes long-horizon tasks far more expensive than a single-turn conversation would suggest.

    2.5 The hidden fifth dimension: stability

    A leaderboard usually reports just one number, but what engineering adoption truly cares about is the distribution:

    • Is the pass rate across repeated runs of the same approach stable within a band, or does it swing wildly?
    • Are failures concentrated in a few task types, or do they appear at random?
    • Is the approach sensitive to prompt wording, framework versions, and dependency caching?

    A score without stability information is hard to use directly for production decisions.

    3. Case breakdown: two routes viewed through four numbers

    The following is an abstracted example based on a publicly reported A/B comparison, modeled as “Approach A” (high completion, high investment) and “Approach B” (low cost, fast response):

    Dimension Approach A (high-completion route) Approach B (low-cost, fast route)
    Task set size 30 long-horizon tasks 30 long-horizon tasks
    Pass rate 24.0% 17.3%
    Average completion 82.3% 71.2%
    Average time 27.4 hours 12.0 hours
    Cost per run ~$317.4 ~$9.9

    Reading 1: the pass-rate gap is smaller than it looks. The two differ by 6.7 percentage points, and at 30 tasks a single task accounts for about 3.3 percentage points, so the gap is only about two tasks. A single run is not enough to conclude that one approach is steadily ahead on pass rate. Also, 24.0% corresponds to 7.2 tasks and 17.3% to 5.19 tasks—neither is a whole number—which suggests the pass rate may come from a statistical result across multiple runs or a weighted calculation, so the measurement convention should be confirmed when interpreting it.

    Reading 2: re-measure with “successful tasks per dollar.” This metric equals pass rate divided by cost per run. Approach A comes to about 0.00076 tasks/dollar and Approach B about 0.0175 tasks/dollar—a gap of roughly 23x. If tasks can be split into reusable modules and failed tasks can be identified quickly, the low-cost route clearly delivers higher output efficiency under a budget constraint.

    Reading 3: the 32x cost buys “certainty at critical moments.” Approach A’s extra investment buys roughly 1.4x the pass rate and 11.1 percentage points of completion. This means that in scenarios with tight delivery deadlines, no easy human fallback, and high failure costs, a higher pass rate may be a necessary cost.

    Reading 4: the two approaches have different “time structures.” Approach A averages about 1.6 hours per task, Approach B about 24 minutes. Roughly estimating at 30 tasks with 10-way concurrency, Approach A’s wall-clock time is about 82 hours and Approach B’s about 36 hours. Approach B’s time scale lets developers submit the work and immediately continue with their own tasks, checking results when it finishes; Approach A must be scheduled as asynchronous batch processing, with a supporting task queue, failure retries, and progress notifications.

    Reading 5: a completion shortfall does not mean the output is useless. The gap between 71.2% and 82.3% means the low-cost approach’s output often needs more careful human review before adoption. The truly cost-effective route is usually not an either/or choice, but tiered routing by task—covered in detail below.

    4. Why “high time and high cost” are almost unavoidable

    The differences between the two approaches are not accidental; they follow from the nature of long-horizon tasks:

    • Many iterations: every error requires a full cycle of analyzing logs, locating files, modifying code, and re-verifying. The more hidden the defect, the more cycles, and the higher the time and token consumption.
    • Context keeps expanding: tool calls, file contents, and error messages continuously enter the context. Even agents with context compression or memory management incur extra computation from deciding what information to keep.
    • Retry costs multiply: the later an error occurs in a long flow, the greater the cost of rollback and rework.
    • Verification itself takes time: builds and emulator startup for mobile tasks often take minutes; this time counts toward the total but has nothing to do with model capability.

    From this, a rule of thumb follows: the economics of long-horizon tasks cannot be judged by model unit price alone; you must also look at the “total tokens required to complete the task” and the “proportion of effective output.” An approach that is cheaper per token but needs more iterations to converge may not have a lower total cost.

    5. Building your team’s own long-horizon agent evaluation system

    Public leaderboards offer reference coordinates, but each team’s business code, framework versions, and engineering conventions differ greatly, so you ultimately still need your own evaluation system. A five-step approach is recommended.

    5.1 Task set design

    • Real-world sourced: sample and desensitize from your internal requirement pool, covering the four task types—application creation, feature development, code migration, and platform conversion—and annotate each with difficulty and estimated manual effort.
    • Moderately sized: a core set of 20 to 50 tasks both covers the main scenarios and keeps the cost of a single evaluation run in check; for latency-sensitive scenarios, set up a separate smoke set of 5 to 10 tasks for fast regression after each prompt or framework adjustment.
    • Reproducible: pin the repository commit and lock dependency files to avoid dependency drift distorting results.

    5.2 Isolated execution environment

    Each task runs in its own container, so build caches do not interfere with one another, and a single execution is bounded by CPU, memory, and timeout limits. When tool calls involve external dependencies, use a proxy to control downloads and caching uniformly—this ensures reproducibility and prevents the agent from accessing unnecessary external resources.

    5.3 Task definition and acceptance

    Use a declarative file to describe each task, explicitly separating “what to do” from “how to judge”:

    1. # task-template.yaml: long-horizon evaluation task definition example
    2. task_id: MIG-001
    3. repo:
    4. url: https://example.com/demo-app.git # placeholder address, not a real link
    5. revision: 8c3f1a0 # pinned commit to guarantee reproducibility
    6. goal: "Refactor the network layer from callback-based to coroutine-based, keeping the existing error handling and retry logic"
    7. constraints:
    8. - "Do not change the publicly exposed interface signatures"
    9. - "New logic must include unit tests"
    10. acceptance:
    11. build: true # the build must succeed
    12. static_check: true # static analysis introduces no new warnings
    13. hidden_tests: true # all hidden tests pass
    14. budget:
    15. max_wall_hours: 8 # terminate and mark as failed after timeout
    16. max_usd: 5.0 # cost cap, useful for comparing the cost-efficiency of different routes

    Hidden tests are a key design. Judging success or failure only with tests the agent can see tends to encourage “coding to pass the tests,” such as returning hard-coded results directly. Combining hidden tests with human spot-check review is what reflects true quality.

    5.4 Result logging and metric computation

    Record the full trajectory of every run: the task definition, each round’s prompts and tool calls, the diff of changes, build and test logs, token usage, time, and cost. The trajectory is used not only for scoring but as the basis for attributing failures. Example code:

    1. import statistics
    2. def summarize(results):
    3. """results: each element is a dict with task_id / passed / completion / seconds / cost"""
    4. n = len(results)
    5. return {
    6. "pass_rate": sum(r["passed"] for r in results) / n,
    7. "avg_completion": statistics.mean(r["completion"] for r in results),
    8. "avg_wall_hours": statistics.mean(r["seconds"] for r in results) / 3600,
    9. "run_cost": sum(r["cost"] for r in results),
    10. }
    11. def summarize_with_ci(results, rounds=1, iters=2000):
    12. """Estimate the 95% confidence interval of pass rate over multiple rounds (bootstrap, illustrative)"""
    13. import random
    14. rates = [summarize(r)["pass_rate"] for r in results]
    15. stats = summarize(results)
    16. if rounds > 1:
    17. stats["pass_rate_std"] = statistics.stdev(rates)
    18. boots = sorted(
    19. statistics.mean(random.choices(rates, k=rounds)) for _ in range(iters)
    20. )
    21. lo, hi = int(0.025 * iters), int(0.975 * iters)
    22. stats["pass_rate_ci95"] = (boots[lo], boots[hi])
    23. return stats
    24. if __name__ == "__main__":
    25. # Example: run the same task set for 5 rounds to observe pass-rate variance and confidence interval
    26. results = [
    27. [{"task_id": t, "passed": True, "completion": 0.8,
    28. "seconds": 5400, "cost": 10.58} for t in range(1, 9)]
    29. for _ in range(5)
    30. ]
    31. print(summarize_with_ci(results))
    32. # Example output: pass_rate ~0.267, 95% CI ~ [0.200, 0.333]

    5.5 Reporting and gates

    Metrics should include at least: pass rate (mean and confidence interval), average completion, cumulative runtime, cost per successful task, and the distribution of failure types. On the cloud, this system can use a container platform to run sandboxes, object storage to archive logs and change artifacts, and logging services and monitoring alerts to track failure rates and timeouts. Baidu AI Cloud also offers general-purpose capabilities such as model services, containers, and observability products that can plug into a self-built pipeline. Mature teams wire core metrics into continuous integration and set regression thresholds: any prompt or framework change that causes a significant drop in pass rate or a clear rise in cost per successful task cannot be merged.

    6. Selection advice: turning benchmark scores into business metrics

    The essence of technology selection is translating “leaderboard ranking” into business language. A useful approach is to layer tasks along two dimensions:

    • Failure cost: does it block a release, does it touch a core path;
    • Fault tolerance: is there ample time for a human fallback, is partial delivery allowed.

    This yields three tiers of strategy:

    Scenario characteristics Recommended strategy Rationale
    Low-risk, high-volume, long-tail needs (e.g., copy tweaks, minor style adjustments) Low-cost fast approach as primary, with human spot checks Limited per-task value; budget and response time take priority
    Medium-complexity feature development Fast approach first, auto-escalate on failure Balances cost and delivery rate
    Core paths, migration/refactoring, near release High-completion approach with strict review and regression High failure cost; prioritize pass rate and completion

    Tiered routing and “try first, escalate later” are easy to put into practice. For example, run the low-cost approach across all tasks and hand only the failed ones to the high-completion approach for a retry. Roughly estimating from the example data in this article: if the fast approach solves half the tasks and the high-quality approach solves the rest in full, the overall cost is about $164 per round—about 48% lower than using the high-quality approach for everything—while the pass rate approaches 24%.

    1. # Illustrative flow: fast approach first, escalate failed tasks for retry
    2. for task in task_queue:
    3. result = run(agent="fast", task=task, budget=task.budget)
    4. if not result.passed and task.priority == "high":
    5. # Keep the first failed trace so the agent can locate the problem
    6. result = run(agent="strong", task=task, resume_from=result.trace)
    7. record(task, result)

    Supporting practices:

    1. Layered budgets: set token and cost caps per task; once exceeded, terminate and hand off to a human, to avoid long-tail tasks consuming budget indefinitely.
    2. Asynchronous execution: run the high-latency approach as a task queue and send notifications via a message queue when it finishes.
    3. Human review cannot be skipped: a benchmark pass only means automated acceptance passed; security review, code conventions, and architectural soundness still need human oversight.
    4. Watch for version iteration: both agent frameworks and model versions evolve quickly, so evaluation baselines should be updated regularly, and cross-comparisons must annotate the version and run date.

    7. Common pitfalls

    1. Pass-rate tunnel vision: ignoring completion and failure types. Among failures, “one test case short” and “the approach went completely off track” have vastly different impacts on downstream human cost.
    2. Drawing conclusions from a single run: the pass rate itself is stochastic; run at least 3 to 5 rounds and report the variance.
    3. Ignoring environment and tool differences: different agent frameworks vary greatly in tool design, context management, and failure recovery; if a comparison changes only the model without stating the runtime environment, the conclusion can be distorted.
    4. Using public problem sets to gauge private codebases: public tasks may have been “seen” by the model during training, so evaluation results on private tasks better reflect real productivity.
    5. Ignoring the quality and security of generated code: compiling does not mean the code is maintainable; keep watching for duplicated code, exception handling, and dependency security.

    8. Conclusion

    The evaluation of long-horizon coding agents is shifting from “can the model write a piece of code correctly” to “can the model independently complete an engineering task.” Pass rate answers “can it be delivered,” average completion answers “how far is it from done,” and time and cost answer “can we afford it and can we wait for it.”

    For development teams, rather than chasing the ranking on a single leaderboard, it is better to build a repeatable evaluation system around your own tasks and select by task risk and fault tolerance: let low-cost approaches handle the massive long tail, let high-completion approaches crack the critical tasks, and defend the quality baseline with strict human review and automated gates. That is how long-horizon agents truly unlock engineering efficiency.