How to Read LLM Agent Benchmarks for Long-Horizon Tasks: Engineering Trade-offs

    This article walks through how to read an LLM agent coding benchmark built around long-horizon tasks, using a recent mobile-app-development leaderboard update as a case study. It breaks down the raw data behind two contrasting engineering profiles — a high-completion, long-duration route and a high-throughput, low-cost route — then clarifies metrics that are easy to misread: pass rate under a small sample of 30 tasks, completion rate as a process-quality signal, and the gap between cost per run and cost per pass. It closes with a trade-off framework for four common scenarios and practical engineering methods for making long-running agent tasks controllable and reproducible through task decomposition, context management, sandboxes, budget gates, and human review.

    Introduction: Behind the Leaderboard Is a Debate About Engineering Trade-offs

    A recent update to a coding benchmark focused on mobile app development (referred to here as “the benchmark”) added results for a new generation of large models. Two things stand out.

    First, a leading model passed a certain long-horizon task for the first time, suggesting that the ability to “deliver a working application from scratch” is crossing a threshold.

    Second, the participating systems show clearly different profiles. Some models trade longer runtime and higher cost for higher completion; others trade a lower pass rate for clear gains in speed and cost.

    For a development team, the point of a leaderboard is not “who ranks first” but seeing where each technical approach fits. This article first breaks down the public data, then clarifies a few metrics that are easy to misread, and finally lays out concrete methods for putting long-running agent tasks into engineering practice.

    First, Look at the Raw Data: Two Profiles, Two Engineering Approaches

    From the public data, the newly added entries fall into two typical profiles.

    High-completion, long-duration route:

    • Paired with a specialized agent, it achieved a 24.0% pass rate across 30 long-horizon tasks;
    • Average completion of 82.3%, a leading figure among the top results on the leaderboard;
    • Average runtime of 27.4 hours;
    • Cost per benchmark run of $317.4.

    Among the top few results on the same leaderboard, the highest pass rate was 28.7%, close to the 24.0% level. The other results falling between 24.0% and 28.7% belong to the same “high completion, long duration, high cost” category.

    High-throughput, low-cost route:

    • Built on a general-purpose agent framework, it achieved a 17.3% pass rate;
    • Average completion of 71.2%;
    • Average runtime of 12.0 hours;
    • Cost per benchmark run of just $9.9.

    The two routes compare as follows:

    Metric High-Completion Route High-Throughput Route
    Pass rate 24.0% 17.3%
    Average completion 82.3% 71.2%
    Average runtime 27.4 hours 12.0 hours
    Cost per run $317.4 $9.9

    Three points are worth noting.

    1. The two routes represent two different optimization goals. The former buys higher task-completion quality with more time and higher cost; the latter emphasizes return on investment. Neither is absolutely better — what matters is the scenario.

    2. These differences are driven by the agent harness, not just the model. The same model series often behaves differently across different agents. A benchmark result reflects the systematic performance of “model + agent + toolchain + runtime environment” and should not be equated with the model’s ability alone.

    3. On either route, the pass rate is still under 30%. Long-horizon, end-to-end development tasks are far from solved, which is why human-AI collaboration remains the primary working mode at this stage.

    Understanding Pass Rate: How Large Is the Sampling Error Across 30 Tasks

    Pass rate is the most intuitive metric — and the most easily over-interpreted. With a sample of 30 tasks, the pass rate fluctuates considerably.

    Assuming tasks are independent and share the same pass probability, the pass rate follows a binomial distribution, and its standard error is

    1. SE = √(p × (1 − p) / n)

    Plugging in the numbers:

    • For a 24.0% pass rate, the SE is about 7.8 percentage points;
    • For a 17.3% pass rate, the SE is about 6.9 percentage points.

    The corresponding 95% confidence intervals are:

    • roughly 9.2%–38.8%;
    • roughly 4.1%–30.5%.

    This means two things:

    • The gap between 24.0% and 28.7% falls within the confidence interval and is within the range of statistical noise;
    • Even though there is about a 7-percentage-point gap between 17.3% and 24.0%, the intervals still overlap somewhat, so a single run is not enough to conclude a “clear lead.”

    Exact figures vary slightly with the estimation method, but the conclusion holds: the sample is small and the error is large. In practice, long-horizon tasks often involve external factors such as environment setup and network dependencies, so failures are somewhat correlated and the real uncertainty is only greater.

    So when reading a leaderboard, look at three things:

    • Was it run multiple times?
    • Are averages and dispersion reported?
    • Are the agent configuration and scoring rules disclosed?

    On top of that, metrics like whether a run succeeds end-to-end, the pass rate of individual test cases, and the average cost per task passed reflect real-world capability better than a single pass-rate number.

    Understanding Completion: What Does 82.3% Mean

    Average completion reflects how far a task was pushed. A common approach is to break a task into verifiable scoring points, judged one by one by a scoring program or a human, with partial credit for partial completion.

    The high-completion route’s 82.3% is markedly higher than its 24.0% pass rate. This means that across the many tasks it did not pass, the agent was not producing nothing — it often completed most of the work but got stuck at a critical step.

    Typical situations include:

    • The core functionality is implemented, but there are edge-case errors;
    • The interface renders, but its behavior does not match expectations;
    • The build artifact runs, but it fails some of the test cases.

    A few conclusions follow.

    First, completion is a process-quality metric and cannot be used directly as grounds for delivery. In software engineering terms, a program that is “80% complete” is generally still not shippable. For migration and conversion tasks, the “last mile” is precisely where compatibility handling, test alignment, and performance tuning — the parts hardest to automate — tend to live.

    Second, high completion is good news for the agent approach. Work already written can be a starting point for a human to take over, reducing rework; low completion means a higher chance of redoing the work from scratch.

    Third, the same completion figure can hide very different failure modes. Some tasks deviate from expectations wholesale; others fail on just a few assertions. The former needs a full rewrite; the latter is often fixed with a few patches. When evaluating completion, it is best to look at it together with a breakdown of failure types.

    Calculating Cost: From “Cost per Run” to “Cost per Pass”

    The cost per run is often compared directly, but the huge gap between $317.4 and $9.9 hides another layer of information: the money is not spent only on tasks that pass — failed tasks consume cost too.

    If a team wants to complete many similar tasks in bulk, a more meaningful metric is cost per pass:

    1. Cost per pass = Cost per run ÷ (Total tasks × Pass rate)

    Plugging in the public data (for illustration only):

    Entry Cost per run Pass rate Expected passes Cost per pass
    High-completion route $317.4 24.0% 7.2 ~$44.1
    High-throughput route $9.9 17.3% ~5.2 ~$1.9
    Highest pass rate $317.4 28.7% 8.6 ~$36.9

    A few takeaways:

    • The high-throughput route’s cost per pass is only about 1/23 of the high-completion route’s; and even the highest-pass-rate result still costs about 19× more;
    • Going from 28.7% to 24.0%, the pass rate improves by less than 5 percentage points, yet cost per pass rises by nearly 20%. On this leaderboard, the marginal cost of “chasing the top spot” is quite high;
    • When the cost-per-pass gap exceeds an order of magnitude, the cheaper option tends to be the more natural choice for large-scale automated pipelines.

    Of course, these are rough estimates. An uneven distribution of task difficulty, whether environment setup is counted, and the retry strategy after failure all affect the final result.

    A more complete accounting should cover the whole pipeline:

    • Environment setup and image building;
    • Model invocation costs;
    • Compute resources for building and testing;
    • Human review and fix hours.

    Labor hours are often the hidden bulk of the cost: even for an unattended 27.4-hour task, reviewing a nearly complete artifact can take hours.

    Translating Metrics into Engineering Decisions: A Trade-off Framework for Four Scenarios

    Mapping these metrics onto engineering scenarios yields the following decision framework.

    1. Quality-sensitive, low-tolerance, mission-critical migrations and conversions: prefer the high-completion route.
    Typical scenarios include migrating core modules, framework upgrades, and changes involving compliance or stability. Here, higher completion means less rework, and runtime cost is usually secondary to labor cost.

    2. High-volume, homogeneous, verifiable batch tasks: prefer the high-throughput route.
    Typical scenarios include bulk adaptation of similar pages, filling in test code, and generating boilerplate modules. A cost per pass of about $1.9 makes large-scale trial and error economically feasible; a 12-hour average runtime also makes “start today, accept tomorrow” possible.

    3. Time-boxed delivery: runtime is a hard constraint.
    An average runtime of 27.4 hours means a single task often spans two working days. If the agent can only run serially, a two-week (10-working-day) iteration cycle fits at most about 9 attempts. In that case, shortening runtime matters more than lifting the pass rate by a few points.

    4. Technology selection and capability exploration: consider pass rate, completion, and cost together.
    When running a small pilot on a model-and-agent combination, record all three dimensions and then judge based on an analysis of failed samples, rather than looking only at the leaderboard rank.

    Whichever route you choose, a 20×+ gap in cost per pass signals that the way the agent is invoked, its retry strategy, and task orchestration still have a lot of room for optimization. Such optimization often affects the final result more than swapping the model does.

    Implementation: Making Long-Running Agent Tasks Controllable and Reproducible

    When average runtime is measured in hours, the key challenge for long-horizon agent tasks has shifted from “can a single run go through” to “can it run repeatedly, stably, and under control.” Here are several general engineering practices.

    (1) Task decomposition: cut the big goal into verifiable small steps

    Long-horizon tasks often fail not because the agent can’t write code, but because it gradually loses the goal and drifts from the constraints over a long context. Rather than asking the agent to “create a complete application” in one shot, break it into independently verifiable subtasks by acceptance criteria, for example:

    1. Set up the project skeleton and get the build passing;
    2. Implement the core data models and APIs;
    3. Build the main interface and interactions;
    4. Add unit and integration tests.

    Each step carries a clear acceptance criterion (such as “build succeeds” or “all tests pass”), and you proceed to the next step only after it passes. This catches deviations early and makes it easy to resume from a checkpoint after a failure.

    (2) Context management: give the repository a “project brief”

    Place a project brief file in the repository root that spells out the tech stack and versions, directory structure, naming conventions, build and test commands, and coding constraints. The agent reads it every time it starts, which effectively reduces rework caused by misunderstandings. At the same time, avoid letting a single session’s context grow without bound — summarize and archive it promptly.

    (3) Sandbox and validation: build, test, and regress in an isolated environment

    Code generated by the agent should first go through static checks, a build, and automated tests in a sandbox before entering code review. On the cloud, you can use container platforms for isolated runtime environments, object storage for build artifacts and test reports, and logging and monitoring with alerts to track long-running tasks.

    Tasks that need to run for a long time can be built as standalone containerized jobs that keep executing in the background, with results pushed out via event notifications.

    General-purpose cloud capabilities — such as containers, storage, and logging/monitoring offered by providers like Baidu AI Cloud — can support this kind of isolated, traceable execution, and teams can adopt them based on their own needs.

    (4) Budget and execution control: make tasks “spend transparently and stop reliably”

    Set a turn limit, a duration limit, and a cost limit for each task, and check whether a threshold has been reached after each model invocation. Example code (pseudocode):

    1. # Budget and quality gate for a long-running agent task (illustrative)
    2. MAX_TURNS = 300 # Maximum number of interaction turns
    3. MAX_DURATION = 3600 * 8 # Maximum runtime (seconds)
    4. MAX_COST = 15.0 # Per-task cost cap (USD)
    5. GATE_CMD = "check && build && test" # Quality gate: format check, build, test
    6. def guarded_run(agent, task):
    7. budget = 0.0
    8. start = now()
    9. try:
    10. for turn in agent.run(task):
    11. budget = turn.cumulative_cost
    12. if (turn.index >= MAX_TURNS
    13. or elapsed(start) >= MAX_DURATION
    14. or budget >= MAX_COST):
    15. # Abort on threshold, archive artifacts, keep the checkpoint for human handoff or retry
    16. archive(task.id, turn.snapshot)
    17. return "interrupted"
    18. yield turn
    19. finally:
    20. metrics.record(task.id, budget, elapsed(start))
    21. def accept(sandbox, task):
    22. code = sandbox.apply(task.patch)
    23. # Roll back if the quality gate fails, to avoid pushing half-finished work into the mainline
    24. return sandbox.sh(GATE_CMD) == 0

    A quality gate like “check first, then build, then test” can be written into local or repository-level hooks to run automatically, reducing human oversight.

    (5) Human-AI collaboration: treat it as a “fast intern,” not “automatic delivery”

    Isolate the agent’s output in a dedicated branch or workspace, and merge it only after a developer’s review. Focus on checking edge-case handling, error handling, dependency versions, and security, and keep a record of decisions through a code review process.

    (6) Failure attribution: build a taxonomy for failures

    Classify failed samples by cause, for example:

    • Misunderstanding of requirements;
    • Loss of context;
    • Environment and dependency problems;
    • Overly strict test assertions;
    • Genuinely beyond the model’s capability.

    Once you have categorized counts, the team can judge which problems can be solved by improving the agent configuration or refining task descriptions, and which must be handled manually.

    Build Your Own Benchmark: Let Your Team’s Data Do the Talking

    A public leaderboard’s evaluation set often differs from a team’s real projects. Building a small evaluation set of your own is the key to making selection data fit your scenario. Recommended principles:

    1. Stay close to reality: Pick 10–30 representative tasks from your history that cover app creation, feature development, code migration, and application conversion, and have humans define the reference answers.
    2. Keep the environment consistent: Fix the runtime versions, dependency sources, and network policy, and use a unified container image to guarantee reproducibility, so environment differences don’t pollute results.
    3. Align configurations: Keep agent configuration, tool permissions, and run budgets consistent so the comparison focuses on the model itself.
    4. Run multiple times: Run each task at least three times and report the median and dispersion, not the single best result.
    5. Automate judging: Assemble a judging pipeline from builds, static checks, unit tests, and end-to-end tests, outputting four metrics: pass rate, completion, runtime, and cost.
    6. Keep costs attributable: Measure model calls, compute resources, and human hours separately, then convert them into cost per pass using the method described earlier.

    When evaluation data accumulates continuously in “days” and can be customized to your team’s real scenarios, it is often more valuable than any public leaderboard.

    Conclusion: Treat the Leaderboard as a Starting Point, Not the Finish Line

    The newly added results on the benchmark show that long-horizon agents are moving from “it can run through” to “it can be measured.” The figures — 24.0%, 82.3%, 27.4 hours, $317.4 — describe a route that trades time and cost for completion quality; the figures 17.3%, 71.2%, 12.0 hours, $9.9 describe a route that emphasizes efficiency and value for money.

    For a development team, what matters is not remembering whose numbers are bigger, but understanding the engineering meaning behind each metric:

    • Pass rate tells you whether it can be delivered;
    • Completion tells you how far it is from delivery;
    • Runtime decides whether it fits into your delivery rhythm;
    • Cost decides whether it can be used at scale.

    On top of that, manage long-running tasks with decomposition, isolation, budget control, and human review, then continuously validate with your own benchmark — that is how agents become a predictable, reusable part of the engineering workflow.