Long-Horizon Coding Benchmarks: Reading the Engineering Tradeoffs Behind AI Coding Scores

    A recent leaderboard update for a mobile-development AI coding benchmark added scores for two new candidate model configurations. This article explains what long-horizon coding benchmarks actually measure, breaks down their four core metrics—pass rate, average completion, average wall-clock time, and cost per run—and shows why "cost per accepted deliverable" is the number that should drive decisions. It also covers the frequently ignored role of the agent scaffolding, then gives practical methods for building your own evaluation set, routing tasks by tier, computing cost per unit of valid output, and governing long-running agents inside CI, along with the capabilities a cloud evaluation platform needs to make this routine.

    Why a Benchmark Leaderboard Update Deserves an Engineering Team’s Attention

    A recent leaderboard update for a mobile-development AI coding benchmark added the scores of two new candidate model/configuration entries. The four numbers reported—pass rate, average completion, average wall-clock time, and cost per run—are far more revealing when read together than any single metric alone:

    • High completion does not necessarily mean high output. The configuration with an average completion above 82% had a pass rate of only about 24%, a cost of roughly $317 per benchmark run, and a wall-clock time of a little over a day.
    • The fastest, cheapest configuration came in at under $10 per run with about half a day of wall-clock time, but its average completion was about 11 percentage points lower and its pass rate about 7 percentage points lower.
    • Once you convert these figures into “how much cost and time it takes to deliver one accepted task,” the higher-tier entry’s threshold is amortized by more than ten times, and the gap is far smaller than the raw numbers suggest.

    This is exactly where long-horizon coding benchmarks are most often misread: a benchmark score is not the end of a report card but the starting point of a chain of engineering decisions. This article first explains what these benchmarks actually test, then unpacks each of the four metrics and what their combinations mean, and finally offers selection and evaluation methods you can apply directly.

    1. What a Long-Horizon Task Benchmark Actually Measures

    Most traditional code-generation evaluations test short, local, well-defined tasks with clear answers—“complete this function” or “fix this specific error.” Long-horizon coding tasks are the opposite. They typically require the system under test to make continuous progress within a single session for several hours, repeatedly reading and writing code across multiple modules, running builds and tests, self-correcting based on failure output, and ultimately delivering a complete application that can be installed, run, and meets the functional spec.

    What makes these tasks hard is not any single capability but four things:

    1. State consistency: After a session spans a large codebase, the model must remember which files it has changed and which assumptions still hold.
    2. Planning and self-correction: Build failures and red tests are the norm. What’s needed is the ability to compress a long stream of error logs into an executable next step.
    3. Resource and concurrency constraints: Builds, simulator launches, and dependency downloads in mobile projects are inherently slow, and poor planning leaves most of the time spent waiting.
    4. Convergence judgment: Deciding when “we’re done” is itself a difficulty. Stopping too early delivers a half-finished product; stopping too late burns budget for nothing.

    Scenarios such as app creation, feature development, code migration, and app conversion are all fundamentally testing the same question: can the model push a real engineering task to the acceptance line on its own, without a human reviewing it line by line?

    2. The Semantics and Traps of Each Metric

    Pass rate: the one hard threshold that cannot be compromised. Pass rate is defined as “the proportion of tasks in a given set whose final output passes all acceptance checks.” It is binary—either it passes or it doesn’t, and there is no “half done counts as half.” Precisely because it is strict, it is extremely sensitive to sample size: a pass rate of about 24% on 30 tasks corresponds to a 95% binomial confidence interval of roughly 10% to 45%. This means a two- or three-percentage-point difference between configurations in the same tier may well be just run-to-run noise and is not enough to support a conclusion about which one is “stronger.”

    Average completion: it describes progress, not delivery. Average completion measures how deep a task was carried—to what extent the output reached the target features and acceptance points. It is valuable diagnostic information that answers “where did the failure happen,” but it is not a quality verdict. A half-finished product with the UI and main flow in place, missing only the last two integration points, is equivalent on pass rate to an approach that doesn’t even compile. Both score zero.

    Average wall-clock time: real time and hidden cost. A wall-clock time of forty-plus hours means a single task spans more than one working day from submission to acceptance. This is not just waiting: the longer the session, the longer middleware and execution environments are occupied, the higher the cumulative cost of retries, and the harder it is to recover schedules that get disrupted while the team waits for results.

    Cost per benchmark run: the compute ledger, but don’t judge it at face value. The cost figure is “how much it cost to complete one round of the benchmark,” and it includes the repeated calls from reading files over and over and running builds and tests repeatedly. It matters especially for long-horizon tasks, because the cost of failure is multiplicative: if a task fails at 90% completion, all the prior calls are still billed.

    3. What High Completion and Long Duration Reveal About Engineering Tradeoffs

    Only when you convert the two configurations side by side does the tradeoff become clear.

    Table 1: Reported data and conversion examples (conversions are computed directly from the disclosed data and are for illustration only)

    Item Configuration A (high-completion tier) Configuration B (lightweight tier)
    Pass rate 24.0% 17.3%
    Average completion 82.3% 71.2%
    Average wall-clock time 27.4 hours 12.0 hours
    Cost per run $317.4 $9.9
    Cost per accepted deliverable ~$1,322 ~$57
    Time per accepted deliverable ~114 hours ~69 hours

    Three conclusions read directly from this:

    First, completion leads by about 11 percentage points at the cost of roughly 32x higher spend. If a team is chasing “looks like it did more,” this cost-effectiveness is hard to justify; if the goal is “must deliver,” the calculation should run through pass rate.

    Second, looking at unit price alone badly underestimates cost. Configuration B’s per-run cost is only about 3% of Configuration A’s, but because its pass rate is only about 0.72x, its cost per accepted deliverable is roughly 1/23 of Configuration A’s. In other words, when reviewing long-horizon coding solutions, the correct yardstick is “cost per unit of valid output,” not “cost per call.”

    Third, long duration is itself a cost item. The 27.4-hour versus 12.0-hour contrast shows that the output curve of a long-horizon agent is “front-heavy, thick at the finish”: it spends time on more complete codebase retrieval, more thorough test verification, and more exhaustive failure diagnosis, in exchange for a higher probability of passing. This pays off in “task-critical, allowed to run overnight” scenarios, but in “must produce continuously during the day” scenarios it quickly drives throughput to the floor.

    So the two configurations are not in a “which is better” relationship but represent two engineering profiles:

    • Heavy investment, stable delivery, expensive per unit—suited to critical paths and automated pipelines;
    • Light investment, high throughput, cheap per unit—suited to exploratory tasks and batch trial-and-error.

    4. The Third Variable You Must Weigh: The Agent Scaffolding

    One sentence in the original report deserves separate emphasis: the two configurations participated in the evaluation using different “AI coding clients / execution scaffolds.” This is often overlooked, but it directly determines whether the leaderboard can be compared horizontally.

    A model’s benchmark score is never a binary “model × task” result; it is a three-part “model × scaffold × task” result. The scaffold determines at least four things:

    1. Available toolset: Whether it can read files, run builds, take screenshots, and check logs directly sets the ceiling on self-correction ability.
    2. Context management: How to compress history and preserve key decisions after dozens of tool calls—in long-horizon tasks this is the dividing line between success and failure.
    3. Retry and verification strategy: Whether it defaults to “must run the tests after every change” or “submit as soon as it’s changed.”
    4. Budget and timeout strategy: Whether it runs until convergence or stops once it hits a call-count limit.

    The implication is clear: without fixing the scaffold, directly comparing two models’ scores yields a composite result of “model plus toolchain.” The more robust engineering approaches are two: first, hold the same scaffold, the same tool permissions, and the same timeout and budget caps across all candidates before comparing; second, acknowledge that you are comparing a “complete package” and include the scaffold’s replaceability, observability, and failure traceability in the selection evaluation. The latter is more realistic for most teams, but the prerequisite is to stop simplifying the conclusion into “model X beats model Y.”

    5. Converting the Leaderboard into a Team Cost Model

    When evaluating these solutions, it’s best to fix on a set of formulas rather than going by feel.

    Cost per unit of valid output

    1. C_eff = C_run / P
    • C_run: cost per benchmark run
    • P: pass rate
    • C_eff: expected cost per accepted deliverable

    Using the data disclosed in Table 1: Configuration A is about $1,322 per deliverable, Configuration B about $57 per deliverable. This ratio is the number that should appear in the review meeting.

    Queue and throughput impact

    1. T_wait = T_run / (number of concurrency slots)
    2. Effective daily throughput ≈ number of concurrency slots / T_run × working hours

    A 27.4-hour run means a single slot can complete fewer than one task per working day. If a team has only two or three slots, the number of long-horizon tasks it can advance concurrently is naturally limited—and this often becomes the bottleneck before model capability does.

    1. # Illustrative pseudocode: choose a tier based on SLA and budget
    2. SLA = {"max_hours_per_task": 12.0, "min_pass_rate": 0.15, "cost_cap_per_success": 600}
    3. PROFILES = [
    4. {"name": "heavy", "pass_rate": 0.240, "completion": 82.3, "hours": 27.4, "cost_run": 317.4},
    5. {"name": "lite", "pass_rate": 0.173, "completion": 71.2, "hours": 12.0, "cost_run": 9.9},
    6. ]
    7. def select_profile(profiles, sla):
    8. feasible = []
    9. for p in profiles:
    10. cost_per_success = p["cost_run"] / p["pass_rate"] # cost per unit of valid output
    11. if p["hours"] > sla["max_hours_per_task"]:
    12. continue # exceeds the time limit, exclude
    13. if p["pass_rate"] < sla["min_pass_rate"]:
    14. continue
    15. if cost_per_success > sla["cost_cap_per_success"]:
    16. continue
    17. feasible.append((cost_per_success, p))
    18. return min(feasible, key=lambda x: x[0])[1] if feasible else None
    19. print(select_profile(PROFILES, SLA))

    The point of this pseudocode is to turn “which one feels stronger” into “which tier falls within the constraint set at the lowest unit cost.” When neither candidate is feasible, the function returns an empty value—at which point the discussion should be about relaxing the time limit, adding concurrency slots, or breaking tasks into smaller ones, not about hunting for a higher score on the leaderboard.

    1. # Illustrative pseudocode: cost estimation under a budget
    2. N_SUCCESS = 100 # number of accepted deliverables required
    3. def plan_budget(profile, n_success, retry_ratio=0.25):
    4. # A lower pass rate means more raw attempts are needed; retry_ratio is the extra re-run proportion
    5. runs = n_success / profile["pass_rate"] * (1 + retry_ratio)
    6. return runs * profile["cost_run"], runs
    7. cost_a, runs_a = plan_budget(PROFILES[0], N_SUCCESS)
    8. cost_b, runs_b = plan_budget(PROFILES[1], N_SUCCESS)
    9. print(f"A: {runs_a:.0f} attempts / ${cost_a:.0f}")
    10. print(f"B: {runs_b:.0f} attempts / ${cost_b:.0f}")

    After the conversion, you can see that for the same 100 accepted deliverables, the lightweight tier requires more raw attempts, yet its total cost may be only a small fraction of the heavy tier’s. If the tasks themselves can be decomposed, tolerate failure, and accept multiple rounds of rework, the lightweight tier paired with automated retries is often the more economical combination.

    6. Five Practical Engineering Methods

    1. Build your own “small but real” evaluation set, and fix acceptance criteria before choosing a model. Don’t copy public leaderboard tasks directly. Pull 10 to 20 real tasks from your own demand pool over the past six months, covering the four categories of new app creation, feature development, code migration, and app conversion, and hard-code the acceptance criteria for each: build passes, core test cases pass, and the key path is reproducible from screenshots. Once the task set is fixed, reuse it long-term as an internal team baseline, and re-run it on the same set every time you switch models or configurations—only then are the scores comparable.

    2. Make decisions with “cost per unit of valid output,” not “unit price.” Convert pass rate, wall-clock time, and cost per run together into the “expected cost and expected time per accepted deliverable.” This yardstick automatically penalizes both extremes—“cheap but repeatedly failing” and “powerful but schedule-constrained”—and brings the discussion back to the team’s real constraints.

    3. Tiered routing: let the expensive and cheap tiers each do their own work.

    1. # Illustrative pseudocode: route by task complexity
    2. def route(task):
    3. if task.has_cross_module_refactor or task.affects_release:
    4. return "heavy" # critical path, cross-module migration: pursue delivery certainty
    5. if task.is_exploratory or task.batch <= 20:
    6. return "lite" # exploratory, small batch: pursue throughput and cost
    7. return "heavy"

    This way you neither sacrifice the first-pass success rate of critical tasks nor prevent batch, trial-and-error, and prototype-validation work from running at a very low unit price. The overall bill is usually an order of magnitude lower than “using the most expensive tier for everything.”

    4. Put the long-horizon agent into the CI pipeline: the three-piece kit of timeout, budget, and traceability. Long-horizon tasks must be governed as a “remote job with a budget cap”: set a wall-clock limit per job, set a call/cost limit, force archival of artifacts and logs, auto-retry on failure with a retry cap, and finally use “build succeeds + key cases pass” as the merge gate—not “looks like it’s done.”

    1. # Illustrative pipeline snippet (YAML)
    2. agent_task_job:
    3. timeout_minutes: 90 # per-task wall-clock limit; terminate on timeout
    4. budget_usd: 5 # per-task call budget cap
    5. max_retries: 2 # failure retry limit
    6. artifacts:
    7. paths:
    8. - build_output/**
    9. - agent_logs/**
    10. retention_days: 14 # retain logs and artifacts for failure review
    11. acceptance:
    12. must_pass:
    13. - build
    14. - critical_cases # critical cases must be fully green
    15. on_success: notify_team

    5. Classify failures in your retrospectives; don’t record just one number. Break failures into attributable categories: compile/dependency issues, mismatched test assertions, UI and state-management defects, migration omissions, runtime crashes, and timeout interruptions. Only with categorized statistics do you know whether to change the model, change the scaffold, change the task description, or change the acceptance criteria. The leaderboard gives an overall pass rate; what a team really needs is “which category are we stuck in.”

    7. Capabilities a Cloud Evaluation Platform Should Have

    To run the above evaluation workflow as a routine engineering capability, the platform side needs support in roughly a few general categories:

    • Elastic containers and isolated execution environments: Provide an independent build and run sandbox for each long-horizon task to avoid dependency conflicts and mutual interference, and support concurrent scaling to shorten queues.
    • Object storage and artifact archiving: Centrally store build artifacts, logs, screenshots, and trajectory files to keep results traceable and reproducible.
    • Message queues and task dispatch: Make batch evaluation tasks asynchronous, smooth out spikes, and support stable submission under high concurrency.
    • Log services and monitoring/alerting: Continuously observe call counts, durations, failure rates, and cost per unit of valid output, with automatic alerts on anomalies.
    • Secret and permission management: Centrally host model-call credentials and code-repository credentials with least privilege and rotation, avoiding evaluation scripts hard-coding secrets into the repository.

    One reminder: for long-horizon evaluation, reproducibility matters more than the score itself. Only when the same task, the same scaffold, the same budget cap, and the same acceptance script are run three times and the median is taken can the conclusion be confidently written into a technical proposal.

    8. Conclusion

    The real value of this leaderboard update is not who ranks first, but that it places “completion,” “wall-clock time,” and “cost per run” in the same table as pass rate for the first time. It lays bare a question the industry has long left ambiguous: improvements in long-horizon coding capability often come at the cost of multiples or tens of times the spend and a full day of time; and whether a lightweight solution that gets about 70% of the pass rate at less than one-tenth of the cost is a better deal depends on how many tasks the team needs to run, how long it can wait, and how costly rework is after a failure.

    The advice for engineering teams is straightforward: treat public leaderboards as a screening tool for the candidate pool, not as a final verdict; and treat your own task set, tiered routing, cost-per-unit-of-valid-output accounting, and pipeline governance as the real moat. The leaderboard will refresh every month, but the method of “making decisions by cost per unit of valid output” will not go out of date.