AI Coding Capability Evolution: From One-Shot Correctness to Long-Horizon Tasks
This article examines a shift in AI coding capability from one-shot correctness—whether a model can produce a correct patch or pass tests on the first try—toward long-horizon reliability, where a model sustains hundreds of iterations under incomplete information. Through a vector-database optimization case that lifted throughput from 3,547 to 21,500 QPS under a 95% recall constraint, it explains what long-horizon tasks are, why they are harder than exam-style benchmarking, and how to engineer for them. It also covers the mainstream evaluation dimensions used for coding models, a practical five-stage optimization workflow, architecture guidance for sandboxed execution, verifiable feedback, context and state management, budget scheduling, and observability, plus a sober look at why high benchmark scores do not equal production readiness.
Most evaluations of AI coding models still revolve around a single question: can the model get it right on the first pass? Given a bug report, can it generate a correct patch in one shot? Given a function signature, can it pass the tests on the first attempt? Yet recent public long-horizon task evaluations paint a different picture: under incomplete information, and without getting it right the first time, a model can keep reading feedback, adjust its approach, and close in on the goal step by step until the task is done.
This may be the most noteworthy change in the coding ability of China’s domestic large models. According to public information, domestic models have entered the global first tier on mainstream software engineering benchmarks, with some narrowing the gap to top overseas models to within one or two points, and a few even overtaking them. But more valuable than any single first-place finish is a shift in the center of gravity: from “single-point correctness” toward “long-horizon reliability.”
1. A Counterintuitive Signal: Models Are Starting to Iterate Through Trial and Error
Let me start with a concrete case. The task was to optimize a vector database implemented in a systems programming language, with a goal summed up in a single sentence: maximize queries per second (QPS) while keeping recall no lower than 95%. Starting from a baseline of 3,547 QPS, the model went through more than 600 iterations and over 6,000 tool calls, raising QPS to 21,500—roughly 6x. At that scale, it is comparable to adding several servers to scale out a read-heavy service holding tens of millions of records.
The number is worth breaking down.
- 600+ iterations means the model tried dozens of optimization directions in sequence. The room for optimizing vector retrieval is spread across indexing structure, distance computation, memory layout, and concurrency scheduling—no single spark of insight can cover all of them.
- 6,000+ tool calls means that on average each iteration involved reading and writing about a dozen files and running about a dozen benchmarks or build commands. Real performance optimization works exactly this way: change one thing, run once, look at the data, then decide the next step.
- The 6x improvement means the model did not spin in place. A common failure mode is repeatedly making the same kind of small tweak, with the metric plateauing early. Sustained progress shows the model identified a more effective optimization path and switched to it in time.
Even more telling is how it handled the constraint. The easiest way to improve retrieval performance is to lower recall—for example, by reducing the number of candidates visited during retrieval. That approach can push QPS up easily, but it violates the task requirement. The fact that the model completed the optimization while keeping recall no lower than 95% shows it did not game the metric; it understood the constraint.
This “measure, change, verify” loop is not new—engineers do it every day when optimizing performance. What is new is that a model can run hundreds of iterations in this loop without losing control: it does not forget the original goal, is not derailed by intermediate failures, and does not lose critical information as the context keeps growing. This is a capability of a fundamentally different nature from “getting it right in one shot.”
2. Behind the Scores: What Mainstream Benchmarks Actually Measure
Before discussing the deeper implications, let’s quickly review the mainstream dimensions used to measure coding models today. The table below only describes the general capabilities each type of evaluation targets; it does not reference any specific product or score.
| Evaluation dimension | Typical task format | Core capability assessed |
|---|---|---|
| Bug fixing | Generate a minimal patch from a bug report and pass the related tests | Locating root cause across files, following existing conventions, avoiding regressions |
| Repository-level generation | Generate a complete project from scratch from a natural-language requirement | Module partitioning, interface design, cross-file consistency |
| Terminal and command-line | Complete build, debugging, and operations tasks in a command-line environment | Environment awareness, command orchestration, diagnosing and fixing errors |
| General knowledge (tools allowed) | Answer hard subject-matter questions using retrieval, computation, and other tools | Combining domain knowledge with tool use |
| Web browsing | Locate information and complete multi-step operations in a web environment | Page understanding, task planning, error recovery |
The table leads to three takeaways.
First, scores are an indirect signal of capability. Climbing the leaderboard only shows how a model performs on public test sets, which can differ significantly from its performance in a team’s real codebase; internal scenarios need their own evaluation sets to validate.
Second, different evaluations cannot be directly compared. Test-set versions, whether tools are allowed, and whether the score comes from a single attempt or the best of multiple attempts all change what the results mean.
Third, open weights mean long-horizon capability can “come out.” Some domestic models are now released with open weights, letting developers deploy, fine-tune, and embed them into internal toolchains in their own environments. For teams whose code and documentation cannot leave the intranet, this point is especially critical.
3. What Is a Long-Horizon Task, and Why It Is the Real Battleground
A long-horizon task is one that requires many intermediate steps and repeated interaction with the environment to complete. It has three characteristics.
- State accumulates. Earlier outputs become inputs for later steps, and intermediate errors propagate forward, so each step needs to be locally correct.
- Feedback must close the loop. The result of each step can be objectively checked—for example, whether a unit test passes or whether a benchmark metric improves—and the model uses that to correct course.
- Goals must stay stable. New information keeps flooding into the context, but the original goal and constraints cannot drift.
The difficulty of long-horizon tasks comes down to four points.
- Error accumulation: at 90% per-step success, the probability of 50 consecutive correct steps is only about 0.5%. In long-horizon settings, whether the model can detect and correct errors matters more than whether a single attempt is impressive.
- Context expansion: the 6,000+ tool calls in the case generate massive logs and test output; how to filter and compress that information directly determines whether the model can keep its judgment later on.
- Exploration vs. exploitation: every iteration faces a choice—dig deeper along the current direction, or switch to a new idea. Being too conservative leads to a plateau; being too aggressive wastes the budget.
- Insufficient test coverage: some intermediate steps have no ready-made test to serve as a judge, so the model can only reason about correctness, and errors may stay hidden for a long time before surfacing.
By contrast, most mainstream coding benchmarks follow an “exam mindset”: a clear problem statement, a single goal, and room to try multiple times and take the best score. Real engineering is a “long run”: vague requirements, multiple goals (correctness, performance, maintainability, cost), and almost no clean first pass. A model that is slightly weaker on single-shot generation but can hold its direction and keep converging over hundreds of iterations may deliver more real value to an engineering team.
4. Case Breakdown: How Long-Horizon Optimization Is Built Step by Step
Taking retrieval performance optimization as an example, a reliable workflow typically includes the following stages.
- Establish a measurable baseline. Record QPS, recall, and the test load, and fix the machine environment and dataset so that every later comparison is fair and trustworthy.
- Locate hotspots. Use profiling tools to find the main sources of time and memory consumption, and determine whether the bottleneck is in distance computation, memory access, or concurrency scheduling—avoiding blind optimization.
- Optimize locally. Improve the hotspots step by step—for example, adjusting memory layout to improve cache hit rate, batching distance computations, and streamlining the candidate-filtering process.
- Switch strategy. When a direction yields diminishing returns, decisively turn to a new phase—for example, from single-threaded micro-optimization to concurrency and batching.
- Converge and validate. Confirm that metrics are stable and constraints have not been breached, then do final validation with an independent test set and a longer time window.
Comparing these five stages against the raw data reveals three ways long-horizon models differ fundamentally from traditional “one-shot generation.”
- Hypothesis-driven: form a hypothesis from profiling data first, then verify it with experiments, instead of changing things at random and hoping to get lucky.
- Eliminate approaches with data: failed attempts are valuable too—they rule out wrong directions and shrink the search space for later steps.
- Constrain exploration with constraints: the recall floor rules out shortcuts like “trade accuracy for speed,” forcing the model to search within the truly feasible region.
# General skeleton of a long-horizon optimization loop (pseudocode)task = read_task_definition(goal_metric, constraints, baseline_codebase)best_result = run_baseline() in a fixed environment, record a metric snapshottried_approaches = {} # record each hypothesis and its verification outcomewhile budget_not_exhausted and convergence_not_reached:hotspot = profile(best_result) # locate the next optimization point from datacandidate = propose_hypothesis(hotspot, relevant_code, past_conclusions) # specify "expected gain + verification method"if candidate.key_assumption in tried_approaches: # avoid burning budget on the same direction repeatedlycandidate = switch_direction(candidate)patch = modify_code(candidate)if not passes(static_check(patch)):record_failure(candidate, "failed static check"); continueresult = evaluate(patch) in an isolated sandbox # the only objective judgeif not satisfies_constraints(result): # e.g., recall below the floorrollback(patch); record_failure(candidate, "violated constraint"); continueif result.goal_metric > best_result.goal_metric and difference exceeds noise_threshold:best_result = result; save_checkpoint(best_result) # every checkpoint is rollback-capablecontext.compress(keep=[goal, constraints, current_best, summary_of_past_conclusions])return best_result
Three design points in the pseudocode are worth expanding on.
- Fixed environment: fluctuations in machine performance and changes in test data will contaminate comparisons—this is the most common hidden trap in long-horizon optimization.
- Fluctuation threshold: benchmark results inherently jitter, so only changes whose improvement clearly exceeds the noise range count as real progress.
- Checkpoint rollback: save a snapshot each time an improvement is accepted, so any step that goes wrong can fall back to the last stable state without destroying existing results.
One reminder: the carrier of this kind of long-horizon capability is the whole system made up of “model + tools + execution environment.” The model handles reasoning and decisions; the execution environment runs commands, executes tests, and saves state. Without a runnable sandbox and automated tests, even a strong model struggles to complete long-horizon tasks.
5. Putting Long-Horizon Capability to Work: Architecture Essentials
If you want to use this capability in your own development workflow, start by sorting out five areas.
1. Sandboxed execution environment
Long-horizon tasks execute a lot of code and commands, so isolation is the first line of defense. Dependencies should be pinned, and network access tightened as needed. When conditions allow, use a container platform to allocate an independent, disposable execution environment per task and destroy it when the task ends.
2. Verifiable feedback signals
The more objective the feedback, the less likely the model is to drift. In priority order: automated tests, performance benchmarks, static checks, and only then model self-assessment. For modules without tests, add basic tests first, then hand them to the model for long-horizon changes.
3. Context and state management
- Externalize state: write the task goal, tried approaches, and conclusions into a structured plan file instead of piling everything into the conversation, so the state can be recovered even after the context is compressed.
- Layered retrieval: put only the code and logs relevant to the current step into the context, rather than stuffing in the whole repository.
- Periodic summarization: keep decisions and data, discard redundant process output.
4. Budget and scheduling
The budget should include at least three things: token consumption from model calls, the number of tool calls, and execution time (wall-clock). It is advisable to set a per-task cap and support mid-run pausing and resuming. Choose an appropriately sized model per phase—for example, hand repetitive work like file exploration and log categorization to a lighter model, and design and hard decisions to a more capable one—to noticeably reduce cost.
For model integration, enterprises can use a model service platform such as the Baidu AI Cloud Qianfan platform to connect and schedule models of different sizes in a unified way, making it easier to manage call quotas and usage centrally.
5. Observability and human checkpoints
Record each step’s input summary, the reason for each decision, and the evaluation result, to support retrospectives and audits. Before code is merged, keep human review and automated gates, including code review, CI pipelines, and security and dependency scans. Operations inside the sandbox must be strictly separated from permissions outside it, and production credentials must not appear in the long-horizon task’s environment.
The quality of the task description also directly determines whether long-horizon execution succeeds or fails. A task ready to hand to a model should look close to the following example.
[Task] Optimize the query performance of the retrieval module[Current state] Baseline throughput: 3,547 QPS (fixed dataset and machine specs)[Goal] Raise throughput to more than 2x the baseline[Constraints]1. Recall no lower than 95%, measured on the fixed test set2. All unit tests pass3. Modify only the retrieval module, without changing the external interfaces[Working method] Profile to locate hotspots first, then propose an optimization plan[Deliverables] Change description + before/after comparison data + test report
Only when the goal and constraints are clear does long-horizon execution have a stable “judge,” and only then can the team objectively validate the final result.
6. A Sober Look: Benchmark Scores Do Not Mean Production-Ready
While staying optimistic about this kind of progress, it is worth watching a few issues.
- Evaluation contamination and overfitting: public test sets may already be in the model’s training data, so scores risk being inflated. When comparing across models, unify conditions such as test-set version, tool permissions, and number of attempts.
- Process opacity: hundreds of iterations mean enormous token consumption. Whether that high consumption buys stable returns needs to be weighed against the task’s value.
- Long-horizon does not mean zero errors: the more steps, the greater the risk of accumulated errors. Without rollback checkpoints and reliable tests, long-horizon execution can amplify problems rather than reduce them.
- Risk-heavy tasks still need human leadership: for tasks involving production changes, permission adjustments, and data handling, keep human confirmation and follow the principle of least privilege.
Conclusion
On balance, the significance of this progress can be summed up in three points:
- A shift in capability focus: from “writing it plausibly” to “actually getting it done”;
- A shift in engineering paradigm: the model goes from a completion tool to an executor that can take on a complete optimization task;
- Advancing technical self-reliance: with open weights, long-horizon capability has a better chance of entering enterprises’ own environments.
For developers and enterprises, the rational approach is to start small with well-bounded tasks—for example, single-module performance optimization, filling in legacy test coverage, and repetitive bug fixes—and build a measurable, rollback-capable, auditable process before gradually expanding the scope. A model being able to work continuously for hundreds of iterations does not mean people can stop watching the process. Quite the opposite: defining goals, setting constraints, and validating results become the more central work for engineers.
