Long-Horizon Tasks: The Real Dividing Line for Evaluating AI Coding Ability
This article argues that the true dividing line for evaluating AI coding models is long-horizon task capability rather than single-pass accuracy. Drawing on a public case in which an open-source model iterated 600+ times to lift a vector-database retrieval throughput metric roughly 6x, it breaks coding ability into five levels by task span, explains why long-horizon optimization is the biggest gap between real work and benchmarks, and analyzes the structural challenges of context overflow, trustworthy feedback, and error accumulation. It then lays out four engineering requirements—context engineering, feedback loops, sandboxed rollback, and budget control—along with guidance on building your own evaluation set and the platform support needed to ship long-horizon agents.
A Counterintuitive Observation: The Edge Shows Up After the Hundredth Pass, Not the First
Most discussion of coding LLMs still centers on “whose first answer is more accurate.” But if you watch how real R&D actually unfolds, you notice a widely overlooked fact: most of the value chain in software engineering does not happen at the instant the first line of code is generated. It happens during the long convergence process that follows—run it, read the results, find the bottleneck, swap the approach, run it again.
A recently released model version from an open-source model team based in China happened to expose this “convergence ability” at an observable scale. In a public case record, facing the same database retrieval optimization task, the model iterated more than 600 times in a row, triggered over 6,000 tool calls, and ultimately pushed a retrieval throughput metric from about 3,500 to about 21,500—roughly 6x. Around the same period, its score on mainstream software-engineering benchmarks sat at around 58.4, slightly above the 57 range where the leading closed-source models landed at the time.
Scores carry information, but the signal that truly deserves developers’ attention is this: this capability gain landed on long-horizon tasks, not on single-point accuracy. That is the core of what this article wants to explore.
Beyond the Leaderboard: Breaking Capability Back Down by Task Span
If you evaluate coding models with a single leaderboard, you lose a great deal of key information. A more reasonable approach is to layer coding ability by “task span”:
- Single-function generation: given a signature and semantics, produce a correct implementation in one pass. This maps to traditional code-completion and unit-test pass-rate evaluations.
- File/module-level generation: given interface contracts, generate a set of interdependent files. This tests naming consistency, dependency management, and error handling.
- Repository-level generation: starting from a natural-language requirement, produce a complete project with directory structure, dependency manifests, config files, and tests. This tests planning and self-consistency.
- Environment-operation ability: executing builds, installs, troubleshooting, and fixes within a command line and runtime environment. This tests tool invocation and feedback interpretation.
- Long-horizon optimization ability: the goal cannot be reached in one pass and must be approached through multiple iterations. This tests strategy switching, state maintenance, and cost control.
Public benchmarks roughly map to the first four layers: repository-level generation scores are often significantly lower than single-function generation (in public evaluations, roughly the 40-point range versus near-saturated high scores), which shows that “building a runnable project from scratch” is still difficult. Tool-augmented complex-reasoning evaluations sit around the 50-point range, web-browsing tasks around 79, and terminal-operation tasks around 63—together these numbers say: there is still a clear gap from “independently completing multi-step engineering tasks,” but a practically usable prototype already exists.
The fifth layer—long-horizon optimization—is the biggest gap between real work and the evaluation system.
What Long-Horizon Tasks Are, and Why They Are the Real Battleground
Traditional coding evaluations are essentially “exam thinking”: one question, one answer, graded by the result. Real engineering is “delivery thinking”: the goal is a continuously verifiable metric (latency, throughput, memory usage, coverage), and acceptance happens through repeated measurement.
Long-horizon tasks therefore have three structural challenges:
1) Context is bound to overflow. The logs, source snippets, and performance reports from 600 rounds of interaction far exceed any model’s context window. The model must have compression and retrieval-style memory: retain conclusions and constraints, discard intermediate noise, and re-fetch details when needed.
2) Feedback signals must be trustworthy. Every step in a long-horizon task needs an objective referee. Without automatically executable tests, benchmark scripts, and correctness checks, the model cannot tell whether it is progressing or regressing, and iteration degrades into blind retries.
3) Errors accumulate. In a short task, one mistake affects only one answer; in a long task, one erroneous change may have dozens of subsequent rounds built on top of it, and often surfaces only much later. This requires the system to have rollback-able state management and staged checkpoints.
Understanding these three points is what lets you appreciate the value of that database optimization case.
Case Study: Hundreds of Iterations for 6x Throughput—What Made It Hard
The task setup was fairly typical: given a vector database skeleton implemented in a systems programming language, optimize the approximate nearest neighbor (ANN) retrieval algorithm, maximizing queries per second (QPS) under the hard constraint that recall stays no lower than 95%.
The difficulty is that it is simultaneously multi-objective optimization and long-chain causal reasoning:
- Common ways to speed up retrieval (reducing the number of probed candidate partitions, lowering the number of fine-ranking candidates, compressing distance-computation precision) almost all sacrifice recall; the model had to continuously probe the boundary around the 95% constraint line.
- The feasible space of candidate approaches is enormous—index structure, quantization scheme, distance-metric implementation, parallelism and memory layout, caching strategy, batching granularity—a change in any single dimension may yield nonlinear gains.
- Each step had to be measured with real benchmarks; the model had to read profiling output and judge whether the bottleneck was compute, memory bandwidth, or synchronization overhead.
The model’s actual behavior showed a three-stage trajectory similar to that of a senior engineer: first establish a measurable baseline with a relatively conservative approach; abandon the current route when returns diminish (switch ideas) rather than intensifying trials in the same direction; then turn to structural changes (such as index organization and memory-access patterns) and keep squeezing within the constraint boundary. The final 6x improvement did not come from a single “silver bullet” change, but from the stacking of multiple structural adjustments.
This is the essential difference between a long-horizon task and “retrying”: retrying is repeated sampling of the same strategy; long-horizon reasoning is a search over the strategy space.
Four Engineering Requirements for Long-Horizon Capability
If you want to build a usable coding agent in your own business, the following four things are no less important than the model itself:
Context engineering. Use a two-layer “summary + retrieval” structure: rolling-compress history into structured summaries (current goal, verified conclusions, list of failed paths), with raw details stored in retrievable external storage and fetched on demand. Explicitly maintain a “constraint list” to avoid forgetting hard conditions like recall, compatibility, and dependency versions over the long run.
Feedback loop. Prepare deterministic verification for each task type: compilation, unit tests, static checks, performance benchmarks, integration smoke tests. Verification must run automatically and output structured results, otherwise the model cannot converge.
Sandbox and rollback-able state. Run each task in an isolated environment, with version control serving as checkpoints so that any round of changes can be rolled back with a single action. This both ensures quality and prevents accidental impact on production.
Budget and cost control. The most realistic constraint on long-horizon tasks is cost. Explicitly set step limits, tool-call limits, token budgets, and timeouts, and switch to “return the current best solution” rather than extending indefinitely when nearing those limits. The skeleton can be expressed in pseudocode:
# Pseudocode: a long-horizon task execution framework with budget gatesdef solve(task, budget):ctx = ContextManager(max_tokens=128_000, summary_ratio=0.15)best = Nonefor step in range(budget.max_steps):plan = planner.reflect(goal=task.goal,constraints=task.hard_constraints, # e.g., recall >= 95%evidence=ctx.summary(),remaining=budget.remaining(),)if plan.should_switch_strategy:ctx.reset_from(plan.new_strategy) # switch strategy, not resample the same oneobs = tools.execute(plan.next_action) # read code / run tests / run benchmarksif verify.gate(obs, task.hard_constraints):return task.solution()if obs.improves(best):best = obs.snapshot(); checkpoint.commit()ctx.compress(obs, keep=plan.relevant_fields)return best or task.baseline()
How to Evaluate Your Own Long-Horizon Capability
Do not rely only on model leaderboards. Build a small evaluation set from your team’s real tasks, and measure at least these four dimensions:
- Final delivery quality: whether the output is correct, whether it meets the hard constraints, and whether manual rework is required.
- Iteration efficiency: the number of rounds, tool calls, and token consumption needed to reach the goal.
- Convergence stability: the variance across multiple runs of the same task—in long-horizon tasks, variance often affects usability more than the mean does.
- Failure diagnosability: whether the failure can produce a readable intermediate process, making it easier for humans to step in.
On task selection, prioritize problems that have “an objective referee, clear constraints, and cannot be completed in a single round”—database and middleware performance tuning, batch-job timeout troubleshooting, and test-coverage completion and refactoring of legacy systems are all good candidates.
Platform Support Needed to Ship It
Long-horizon tasks demand very different infrastructure from short interactions. In engineering practice, you typically need: containerized isolated execution environments to run multiple candidate approaches in parallel; object storage to keep task artifacts, benchmark reports, and intermediate checkpoints; logging services and monitoring/alerting to provide end-to-end observability, especially resource usage and failure attribution for very long tasks; and pipeline tools to solidify verification and benchmark scripts into reusable assets. On cloud platforms such as Baidu AI Cloud, these capabilities are typically offered as general product categories—container platforms, object storage, logging services, and monitoring/alerting—that teams can combine directly, without having to build the underlying infrastructure themselves.
Significance and Limits
The real significance of this progress has three aspects. First, the anchor of capability evaluation shifts from “single-question accuracy” to “long-horizon delivery,” finally aligning with enterprises’ actual procurement and acceptance criteria. Second, Chinese open-source models have entered the top tier in the long-horizon direction, which means this kind of capability is no longer monopolized by a few closed-source solutions, making privatization and cost control possible. Third, it validates the effectiveness of the engineering paradigm of “verification signals + context management + strategy switching,” so teams can invest in building accordingly rather than waiting for models to naturally get stronger.
At the same time, stay clear-headed: long-horizon task evaluation is costly and high-variance, and public benchmarks can hardly cover it completely; the inference cost of hundreds of iterations remains significantly higher than a single generation; and for legacy systems that lack automated verification, the payoff from model capability diminishes substantially. So the more pragmatic path is this—first build the verification system and observability, then bring long-horizon capability into them. The model sets the ceiling; engineering determines how much of it you actually capture.
