AI Coding Benchmarks for Long-Horizon Tasks: What Leaderboard Results Reveal

    This article analyzes a mobile application development benchmark for AI coding agents by comparing two anonymized solutions. It explains how long-horizon task benchmarks evaluate app creation, feature development, code migration, and app conversion, and how to interpret pass rate, completion rate, runtime, and cost. It also provides practical guidance for choosing agents, building private evaluation suites, adding budget guardrails, and running reproducible benchmark pipelines.

    From Code Generation to Long-Horizon Engineering Tasks

    In recent years, large language models have improved rapidly in code generation and code understanding, and evaluation methods have evolved alongside them. Early code benchmarks often used competition-style problems or single-function completion tasks. Today, long-horizon engineering benchmarks, such as those for mobile application development, place models in a workflow that is closer to real delivery: building an application from scratch, adding features to an existing codebase, migrating code, and even converting an application from one platform to another.

    Recently, a mobile application development AI coding benchmark published a new set of results. The evaluated solutions were several mainstream agentic coding approaches. Two anonymized result groups are especially representative: one solution took longer and cost more, but achieved better completion and pass rates; another solution achieved a reasonably close pass rate at about 1/32 of the cost and less than half the runtime.

    This article uses those two result groups to break down the engineering trade-offs behind the leaderboard and to provide practical recommendations for solution selection and evaluation.

    What Long-Horizon Task Benchmarks Evaluate

    Common task types in mobile development benchmarks include application creation, feature development, code migration, and application conversion. They share one key characteristic: completing a single task often requires moving back and forth across dozens of files and going through multiple cycles of build, run, test, and repair.

    These tasks usually have the following features:

    • Ambiguous input: Requirements may be only one or two sentences long, so the agent must infer boundaries and implicit conventions on its own.
    • Expensive feedback: A single build or a round of UI testing on an emulator or physical device can take several minutes, and failure logs may not directly reveal the root cause.
    • Strict acceptance: Code is counted as passing only if it builds successfully and core test cases pass. Any failure in a critical area can cause the entire task to be marked as failed.

    As a result, these benchmarks evaluate more than whether a model “can write code.” They evaluate whether an agent can maintain long-chain planning within a limited context window, use tools correctly, and self-correct after repeated failures. This matches some of the hardest parts of real-world AI coding adoption.

    Reading Leaderboard Data: Four Metrics with Different Meanings

    In anonymized form, the two result sets can be shown as follows:

    Metric Solution A (higher completion; Agent Framework A) Solution B (lightweight and fast; Agent Framework B)
    Number of benchmark tasks 30 long-horizon tasks 30 long-horizon tasks
    Pass rate 24.0% 17.3%
    Average completion 82.3% 71.2%
    Average runtime 27.4 hours 12.0 hours
    Cost per benchmark run $317.4 $9.9

    The two solutions differ by 6.7 percentage points in pass rate, which is roughly equivalent to two tasks out of 30. Because the sample size is small, the result is highly sensitive to the outcome of any single task.

    For example, an agent with poor decoupling may stuff context, build logs, and test results into the session. As the number of conversation turns increases, the context expands quickly, which raises both inference cost and time to first token. This is one common source of high cost for long-running solutions.

    Therefore, leaderboard metrics should be interpreted together:

    1. Pass rate is a qualification metric. It is usually based on final acceptance and reflects the proportion of usable outputs reasonably well. However, it does not distinguish between “almost passed” and “not implemented at all.”

    2. Average completion is a progress metric. It shows how far tasks were advanced and can be useful as a process reference, but it cannot be directly equated with the proportion of usable results.

    3. Average runtime reflects the real delivery pace. When backend tasks run asynchronously, longer runtime means tasks occupy queues for longer, which can limit parallel throughput.

    4. Cost per benchmark run is the model and tool cost of completing all 30 tasks. It is converted using the published prices and exchange rates at the time of reporting. It is a useful measure for comparing “how much more expensive” one solution is, but it is not the same as the production cost of a single task.

    Core Question: What Engineering Trade-offs Are Revealed by Higher Completion and Longer Runtime?

    There is a systematic gap between “doing more and taking longer” and “doing it correctly.” At least four trade-offs are involved.

    1. Completion Rate Is Not Pass Rate: The Rigidity of Engineering Acceptance

    Solution A’s average completion is about 1.16 times that of Solution B, but the relative gap in pass rate is about 39%. The two metrics do not move in sync for three main reasons:

    • Long-horizon tasks often use veto-style acceptance. Even if 90% of the functionality is complete, the task is still counted as failed if the build fails or a critical test case does not pass.
    • Completion may include intermediate steps that were attempted but not validated. Acceptance usually recognizes only the final usable result.
    • More ambitious agents tend to do more. More changes increase the probability of regressions or violations of existing conventions.

    Engineering implication: Progress from AI-generated work must be mapped to executable and verifiable acceptance criteria, such as build success, static checks, unit tests, and UI automation scripts. Completion can be used for observation, but not for final sign-off.

    2. Time Is a Hidden Cost Lever: The Cost Gap Narrows Under Limited Budgets

    Model inference is generally billed by token usage. The longer a task runs, the more input context and output tokens are consumed. A simple estimate is shown below:

    1. Nominal cost difference: 317.4 / 9.9 ≈ 32.1x
    2. Cost-rate difference normalized by time: (317.4 / 27.4) / (9.9 / 12.0) ≈ 14.0x
    3. Under a 1-hour time budget: 32.1 / 2.28 ≈ 14.1x

    In other words, if a team limits each task to one hour of runtime and scales cost proportionally, the relative cost gap between Solution A and Solution B narrows from about 32x to about 14x.

    This simplified estimate assumes that cost grows roughly linearly with runtime. In practice, the result may vary because of different model pricing and caching mechanisms. Even so, it reveals an important rule: speed is an important adjustment variable for cost. Splitting tasks into smaller units and setting execution limits can often reduce cost more immediately than switching models.

    3. Diminishing Marginal Returns: The Marginal Cost of Improving Pass Rate

    Using Solution B as the baseline, Solution A spends about $307.5 more and about 15.4 hours more to improve pass rate by 6.7 percentage points.

    • Improving pass rate by 1 percentage point costs about $46 and 2.3 hours on average.
    • Tail costs deserve even more attention. Suppose a failed task still has 80% of its subtasks completed. Even if the team wants to verify that the final step failed and decide to abandon the task, it still has to pay for the previous 80% of the work. Then the task may have to be reassigned to Solution B and restarted from scratch. The overall cost could become several times higher.

    Therefore, stacking longer runtimes and larger models does not necessarily produce linearly increasing value. Diminishing marginal returns are especially visible in long-horizon tasks.

    4. Constraints Among Planning, Fault Tolerance, and Model Cost

    Higher completion and longer runtime usually correspond to stronger multi-step planning, more tool calls, and more thorough self-correction. Larger models are generally slower per inference and usually have higher token prices.

    To keep an agent from “getting lost” in a long chain, teams often need to invest more in three areas: context management, failure retry, and result verification.

    This creates a kind of impossible triangle: high completion, low runtime, and low cost are difficult to achieve at the same time. Teams can only choose their priority based on task characteristics and use task orchestration to approach a better solution.

    From Leaderboard to Decision: Choose Agents by Task Profile

    • Budget-sensitive and latency-sensitive tasks: Code repair in CI gates, routine feature increments, and rapid prototyping usually have clearer task boundaries and easier rollback. In these cases, a lightweight and fast solution is preferable, combined with explicit runtime limits and budget caps. A full benchmark run costing about $9.9 is more suitable for high-frequency, low-risk daily tasks.

    • Complex delivery tasks: Large-scale code migration, cross-platform application conversion, and legacy system refactoring may require several hours or even multiple days. These tasks are often reviewed by senior engineers, so the additional cost is easier to dilute. If the $317.4 cost of Solution A covers all 30 tasks, the average is only about ten dollars per task. A manual migration may consume several person-days, so the difference is not necessarily dramatic.

    • Hybrid routing: Dynamically select agents based on task type, historical success rate, and budget constraints. For example, simple tasks can use a lightweight solution, while complex tasks can be assigned to a higher-capability solution with timeout and budget guardrails.

    Engineering Implementation: Make Leaderboard Data Serve Your Own Evaluation

    1. Build a private evaluation set. Select 20 to 50 historical tickets from your own repositories as task samples. Cover application creation, feature development, code migration, and application conversion. Define an automatically executable acceptance criterion for each task.

    2. Establish layered acceptance and budget guardrails. In order: build and static checks → unit tests → critical-path UI automation. Stop immediately when a layer fails and retain failure logs. Also set a per-task runtime limit, maximum token consumption, and maximum cost to prevent runaway execution.

    3. Standardize the execution environment. Run each task in an isolated container. Use fixed dependency caches to improve build efficiency. Record execution traces and token consumption in a logging service for attribution analysis.

    4. Run continuous regression evaluation. After updating a model or agent framework, rerun the same private task set. Build trend curves for “pass rate — runtime — cost” so that decisions are not driven by a single leaderboard result.

    The following sample code shows how an evaluation framework can record all four metrics while enforcing budget guardrails:

    1. import time
    2. from dataclasses import dataclass
    3. @dataclass
    4. class BenchCase:
    5. task_id: str # Task ID
    6. goal: str # Natural-language task description
    7. checks: list # Acceptance checks ordered from low cost to high cost
    8. @dataclass
    9. class RunRecord:
    10. task_id: str
    11. passed: bool # Passed only if all acceptance checks pass
    12. completion: float # Ratio of executed subtasks, used as completion rate
    13. duration_h: float # Task runtime in hours
    14. cost: float # Cumulative task cost
    15. def run_case(agent, case: BenchCase, time_limit_h: float,
    16. cost_limit: float) -> RunRecord:
    17. started = time.monotonic()
    18. completed, total, passed = 0, 0, True
    19. def over_budget():
    20. return ((time.monotonic() - started) / 3600 >= time_limit_h
    21. or agent.total_cost >= cost_limit)
    22. # Phase 1: observe and execute; stop when the budget is exhausted
    23. while not over_budget() and not agent.finished():
    24. agent.step()
    25. completed += 1
    26. total = agent.total_steps()
    27. # Phase 2: layered acceptance; stop immediately on failure to save cost
    28. for check in case.checks:
    29. if over_budget() or not check.run(agent.workspace):
    30. passed = False
    31. break
    32. completion = completed / total if total else 0.0
    33. return RunRecord(
    34. case.task_id,
    35. passed,
    36. completion,
    37. (time.monotonic() - started) / 3600,
    38. agent.total_cost
    39. )
    40. def summarize(records: list):
    41. n = len(records)
    42. return {
    43. "pass_rate": sum(r.passed for r in records) / n,
    44. "avg_completion": sum(r.completion for r in records) / n,
    45. "avg_duration_h": sum(r.duration_h for r in records) / n,
    46. "avg_cost": sum(r.cost for r in records) / n,
    47. "cost_per_pass": (
    48. sum(r.cost for r in records)
    49. / max(sum(r.passed for r in records), 1)
    50. ),
    51. }

    Running an Agentic Coding Evaluation Pipeline in the Cloud

    Batch execution of agentic coding tasks is a typical “high-elasticity, long-tail, resource-heavy” workload. In a cloud environment, such a pipeline can be built from general cloud service categories. For reference, Baidu AI Cloud provides relevant product categories that can be used as an illustration:

    • Compute and scheduling: Evaluation tasks can be dispatched through message queues. Tasks can run in isolated environments on a container platform and scale automatically based on queue length. For batch regression jobs that require elastic resources, task queue services can help smooth out peaks and valleys.

    • State and collaboration: Code repositories inside sandboxes can be hosted on a code hosting service. Message channels and task orchestration for agent collaboration can be supported by stateless compute services such as function compute and container platforms. Cache services can help with cache hits and context compression.

    • Storage and data: Dependency caches, build artifacts, and execution traces can be written to object storage. Datasets and evaluation results can be stored in data lakes or data warehouses to support historical comparison and trend analysis.

    • Observability and security: Logging services can record execution traces and token usage. Monitoring and alerting services can set thresholds for timeout rate, failure rate, and per-task cost. Identity and access management, together with key management services, can ensure that agents access code and secrets only within the minimum required permission scope.

    Conclusion

    The most important value of this leaderboard is not that a particular model moved up one position. Its real value is the engineering facts it reveals: in long-horizon development tasks, there is a systematic gap between completion and pass rate; runtime amplifies cost along the token consumption chain; and the marginal cost of improving pass rate continues to rise.

    For technical teams, the rational approach is not to chase a single leaderboard ranking. Instead, teams should build their own task sets and acceptance standards, bring “pass rate — runtime — cost” into a unified evaluation loop, and then use task orchestration, budget guardrails, and human-agent collaboration to apply the right agent capabilities to the right tasks.