Chinese LLMs Are Closing the Coding Gap with Leading Global Models
This article examines how coding evaluation for large language models is shifting from single-pass correctness to long-horizon engineering capability. It discusses why exam-style benchmarks fail to capture real software development, uses a vector-search optimization case with a reported 6x QPS improvement to illustrate long-running tool use, outlines six key capabilities needed for long-horizon tasks, and provides an execution-loop framework. It also covers practical evaluation design, enterprise adoption, infrastructure requirements, and the boundaries of current progress.
1. The Shift: From “Getting It Right Once” to “Continuously Getting It Right”
Coding ability has become one of the main battlegrounds for large language models. The dominant evaluation approach still resembles an exam: give the model a problem and see whether it can solve it correctly in one attempt. In early April, a Chinese LLM team released a new coding model. It arrived just over a month after the previous generation, was released as open source, and was made available to users enrolled in the team’s coding subscription program. In public evaluations, its programming performance is approaching the leading global level.
However, focusing only on benchmark scores can miss more valuable signals. The team emphasized a capability closer to real engineering work: with tool use and repeated verification, the model can continue iterating for more than 600 rounds, call tools more than 6,000 times, and improve query throughput from 3,547 queries per second to 21,500 queries per second, an improvement of roughly 6x. This exceeded the previous best result on that task.
These numbers come from controlled evaluation environments and should not be treated as direct evidence of production gains. Still, they point to a clear trend: competition in coding models is shifting from single-problem accuracy to long-horizon task completion.
2. Why Exam-Style Evaluations Miss Real Engineering Ability
2.1 What Single-Pass Evaluations Measure and Overlook
Typical coding evaluations follow two main lines. The first uses bug-fixing benchmarks based on real repositories, testing whether a model can understand a project, locate a defect, and produce a patch that passes tests. The second generates code from function signatures or natural-language descriptions and checks whether the output meets expectations. These evaluations are objective, reproducible, and low cost, making them important tools for measuring progress. However, they usually assume “one problem, one attempt.”
| Comparison | Single-Pass Evaluation (“Exam Mindset”) | Long-Horizon Task (“Engineering Mindset”) |
|---|---|---|
| Duration | One turn or a few interactions | Tens of minutes to several hours |
| Feedback | Whether tests pass | Continuous metrics such as performance and coverage, plus regression results |
| Cost of failure | Immediate failure | Can roll back or switch approaches |
| Main focus | Getting it right once | Planning, verification, error correction, and convergence |
| Deliverable | Code patch | Complete, runnable, measurable result |
2.2 Real Engineering Is a Closed Loop
Real software development is a loop of writing, running, measuring, and adjusting: implement a version, run it, find a bottleneck, change the approach, and verify with regression tests. If necessary, the work may be restarted. Any step can fail, and fixing those failures is part of engineering ability.
A single-pass evaluation is like a driving-school exam that checks whether a specific action can be performed correctly once. Real development is more like completing a long drive in an unfamiliar city: it requires continuous navigation, handling unexpected situations, and arriving on time.
3. What Long-Horizon Tasks Are and Why They Are Difficult
3.1 Definition: Long Decision Sequences with Sparse Feedback
A long-horizon task usually requires a model to make decisions across many consecutive steps with minimal external intervention. In reinforcement-learning terms, it can be abstracted as a trajectory:
s0 --a1--> s1 --a2--> s2 --> ... --aT--> sT
Here, s represents the state, such as the codebase, files, test results, or runtime logs. a represents an action, such as editing a file, running a command, executing tests, or calling an API. T is the number of steps.
Long-horizon tasks have three key characteristics:
- Many steps, often ranging from dozens to thousands.
- Sparse or delayed feedback; a reliable signal may require a full test run or benchmark cycle.
- State must be persistent; intermediate results cannot live only in the model’s context or “memory.”
3.2 Four Core Challenges
The difficulty of long-horizon tasks is not in a single step, but in connecting many steps together.
- Error accumulation. Suppose a model has a 99% success rate for a single step. The probability of completing 100 steps correctly is only about 37%. Without verification and rollback, errors can accumulate and grow. Conversely, if each step can be validated and corrected, the success rate can improve substantially.
- Context drift. As the number of turns grows, initial requirements and constraints can become diluted. A model may forget a hard requirement, such as “recall must not fall below 95%,” and eventually produce a fast but unusable result.
- Stagnation loops. A model can repeatedly tweak the same area without making progress: changing things again and again, without improving metrics, moving backward, or switching direction.
- Goal substitution. Optimization can become a loophole: skipping validation, simplifying input data, or otherwise making the metric look better while the real outcome becomes worse.
These problems share the same root cause: long-horizon tasks require more than a smarter next step. They require knowing where the task stands, how far it is from the goal, and whether the next step is worth taking.
4. Case Study: Why Vector-Search Optimization Achieved a 6x Improvement
The team’s representative case comes from a vector-search performance optimization evaluation. The benchmark provided a skeleton vector database written in a systems programming language. The task was to maximize query throughput per second, or QPS, while keeping approximate nearest-neighbor recall at no less than 95%. The delivered result improved from 3,547 QPS to 21,500 QPS. During the process, the model’s strategy changed several times.
4.1 Why This Type of Task Is Difficult
- Tension between constraints and objectives. Recall and throughput constrain each other. Expanding the search range can improve recall but increase latency. Compressing the index can improve speed but reduce precision. The real challenge is approaching the constraint boundary while continuously improving.
- Large search space. Tunable elements include index structure, quantization methods, distance-calculation implementations, batching and parallel strategies, and memory layout. The number of possible combinations is far beyond brute-force enumeration.
- Noise. Throughput is affected by machine load, cache state, and data distribution. A single measurement is not reliable, so repeated runs and statistical aggregation are required.
- Overfitting risk. Tuning only against the benchmark’s built-in dataset can produce a misleading result: fast on the evaluation set but slow on real data.
4.2 Reconstructing the Process: Performance Optimization Like an Engineer
Public information discloses the result and the overall process, but not every internal decision. The outcome aligns with common performance-optimization patterns and can be reconstructed into three phases.
| Phase | Typical Actions | Main Source of Gain |
|---|---|---|
| Baseline diagnosis | Run the skeleton, establish a reproducible measurement process, and use profiling to locate hotspots | Identifying the real bottleneck |
| Local optimization | Improve distance calculation, vectorization, cache friendliness, and parallel scheduling | Stable marginal gains |
| Structural adjustment | Change index structure or adjust the trade-off between quantization and recall | Breaking local optima and achieving order-of-magnitude improvement |
Many optimization attempts may reach a plateau around the 20th round, after which small tweaks produce little benefit. Breakthroughs often come from changing the level of abstraction, such as moving from micro-optimizations to index-structure redesign, or reallocating recall budget to more important areas.
This is what “not blindly retrying” means. The model should judge whether the current direction still has value. If not, it should switch paths rather than keep turning in the same place.
4.3 Lesson: Establish the Judge Before Relying on the Agent
A prerequisite for this case was that the task provided a clear judge: a recall threshold, a reproducible benchmark, and an explicit optimization objective. The more objective the feedback, the more reliably the model can self-correct.
For enterprises and teams, this means answering one question before introducing long-horizon agents: can my task be validated automatically and objectively?
5. Six Key Capabilities That Support Long-Horizon Tasks
5.1 Externalize State: Context and Memory Management
When context exceeds the window, two common approaches are truncation and summary compression. Both risk losing critical information such as requirements, constraints, or lessons from failed attempts. A more robust approach is to externalize state.
# Task: Optimize vector-search throughput## Goal and constraints- Goal: maximize QPS- Constraints: recall >= 95%; do not change the public API; use only the standard library and existing dependencies## Current best- QPS: 21500 (round N; index parameter set A + distance-calculation vectorization)## Failed hypotheses (avoid repeating)- Lowering the recall threshold only: recall fell below threshold; rolled back- Brute-force search without an index: latency too high; rolled back
With a structured experiment log for each round—hypothesis, change, result, and conclusion—the model only needs to load the plan, current best result, failed-attempt list, and recent logs into context to remain aligned across rounds. When more detail is needed, it can retrieve the original files by ID. This is a form of layered memory.
5.2 Tool Calling: The Quality of the Action Space Determines the Ceiling
- Tools must be composable. Reading files, editing code, running tests, checking logs, and calling APIs should cover the full task loop.
- Output should be structured. JSON or other structured returns are more stable than long natural-language text and are easier to compress.
- Errors must be readable. When a command fails, returning the exit code, key logs, and suggested next action is far more useful than simply saying “execution failed.”
- Tools must be protected. With container isolation, resource quotas, and timeout controls, a single failed or timed-out tool should not bring down the entire task.
5.3 Self-Evaluation: Make Verification Part of the Loop
After every change, the model should answer three questions:
- Did the metric actually improve? The improvement should be statistically meaningful, not a one-off fluctuation.
- Did the change break anything else? This requires regression tests, interface-contract checks, and boundary cases.
- Can the conclusion be reproduced? Random seeds and data should be fixed, and results should be repeated with medians reported.
Illustrative pseudocode:
def evaluate_change(workspace, bench, repeats=3, tolerance=0.02):# 1. Functional correctness: build, unit tests, and contract tests must all passok = run(["build"]) and run(["test"]) and run(["contract-test"])if not ok:return {"accepted": False, "reason": "regression"}# 2. Performance: run multiple repetitions and take the median to reduce noisescores = [bench.run(workspace) for _ in range(repeats)]score = median(scores)# 3. Constraint check: for example, recall must not fall below the thresholdif score["recall"] < bench.min_recall:return {"accepted": False, "reason": "constraint_violated"}# 4. Improvement check: a gain above the noise tolerance over the best value counts as validgain = (score["qps"] - bench.best_qps) / bench.best_qpsreturn {"accepted": gain > tolerance, "score": score, "gain": gain}
5.4 Strategy Switching: From Local Hill-Climbing to Global Search
When there is no meaningful improvement for several consecutive rounds, the model should switch strategies. Possible approaches include:
- Raising the abstraction level. Move from parameter tuning to algorithm or data-structure redesign.
- Changing the search strategy. Shift from improving one direction at a time, similar to hill-climbing, to parallel evaluation of multiple candidates, similar to beam search, or introduce controlled exploration.
- Rechecking constraints. Review whether hard requirements were sacrificed for the metric.
- Requesting human clarification. When objectives conflict or information is insufficient, ask rather than guess.
5.5 Budgets and Termination: Making Long Runs Cost-Aware
Long-horizon tasks can become increasingly expensive. Engineering practice requires setting budget limits for each task, such as maximum steps, tool calls, token consumption, and duration. Clear termination conditions are also necessary:
- The goal has been reached with sufficient margin.
- There has been no meaningful improvement for N consecutive rounds.
- The budget is exhausted.
- A blocker appears that the model cannot resolve itself, such as missing permissions or an unavailable external dependency.
6. A General Execution Loop for Long-Horizon Tasks
The following product-agnostic loop shows how these mechanisms work together. It places verification on equal footing with execution. Variable names and thresholds are illustrative.
from dataclasses import dataclass@dataclassclass Budget:max_steps: int = 1000 # Maximum number of iterationsmax_tool_calls: int = 10000 # Maximum tool callspatience: int = 50 # Trigger replanning after this many rounds without meaningful improvementmax_replans: int = 3 # Maximum number of replanning roundsdef solve_long_horizon(goal, workspace, verifier, budget):# 1) Externalize state: plan, constraints, current best, and failed attempts are written to the workspacestate = init_state(goal, workspace)last_check = 0for step in range(budget.max_steps):if step - last_check >= 20:# 2) Maintain context: replace long history with a structured summary to prevent requirement driftstate.context = summarize(state.journal, keep=[state.goal, state.constraints, state.best, state.failed])last_check = step# 3) Decide and execute: edit files, run commands, and collect metrics in an isolated environmentaction = plan_next_action(state)result = tools.execute(action, sandbox=workspace)state.journal.append(result)# 4) Self-evaluation: functional correctness, constraint satisfaction, and significant improvement are all requiredreport = verifier.evaluate(workspace)if report.accepted:state.best, state.stagnant = report.score, 0else:state.stagnant += 1state.failed.append(report.reason)# 5) Strategy switching: when stuck at a plateau, raise the abstraction level or change the search strategyif state.stagnant >= budget.patience and state.replans < budget.max_replans:state.plan = replan(state, reason="plateau")state.stagnant, state.replans = 0, state.replans + 1# 6) Terminate: goal met, budget exhausted, or human intervention neededif verifier.satisfied(workspace) or budget.exhausted(state):break# 7) Produce an auditable report: final metrics, key decisions, failed attempts, and costreturn build_report(state)
This skeleton has two design intentions:
- Fault tolerance is core, not an add-on. It addresses the noise problem discussed earlier. A failed evaluation does not terminate the task; it becomes input for the next decision.
- Observability determines optimizability. Recording metrics and decisions each round makes it possible to determine which strategies actually work, improving prompts, tools, or the evaluation itself.
7. How to Evaluate Long-Horizon Capability Scientifically
As single-benchmark scores converge, long-horizon capability needs new measurement methods. A practical evaluation system should cover three layers.
- Task layer: coverage breadth. In addition to bug fixing, tasks should include performance optimization, dependency migration, building small projects from scratch, terminal operations, and collecting and organizing information across web pages. Each task type needs an automatically verifiable success criterion.
- Process layer: efficiency curves. Record how metrics change over iterations and compare the time, tool calls, and token consumption required to reach equivalent quality. Two models may both reach 21,500 QPS, but if one does so in 50 rounds and another needs 600, their engineering value is very different.
- Stability layer: multiple runs. Long-horizon tasks are sensitive to randomness. They should be run multiple times, reporting means, medians, and variance. Drawing conclusions from one impressive run can easily be misleading.
For example, the reported results from the team discussed in this article include:
| Capability Dimension | Result | Brief Interpretation |
|---|---|---|
| Software engineering bug fixing | 58.4 | Slightly higher than two mainstream overseas models from the same period, at 57.7 and 57.3 |
| Generating a complete code repository from natural language | 42.7 | Still about 7 points behind a leading overseas model, at 49.8 |
| Terminal operations | 63.5 | Reflects operation and execution ability in command-line environments |
| Open-domain knowledge Q&A with tools | 52.3 | Tests the ability to use tools to solve difficult problems |
| Web browsing and task handling | 79.3 | Reflects cross-page information retrieval and task execution |
The first two items both involve complete tasks, but the conclusions differ. The model has surpassed leading global results on fixing existing projects, but still lags in building entire repositories from scratch. This suggests that long-horizon capability is not uniform. Incremental improvement on existing code is easier because it allows verification and iteration. Building from scratch requires stronger planning and architectural design.
Evaluation design also has a common pitfall: public datasets may have appeared in training data, and offline tests may be disconnected from production. Enterprises should build private evaluation sets that turn internal requirements, historical defects, and business metrics into repeatable tasks.
8. Enterprise Adoption: Applying Long-Horizon Capability to Real Business
8.1 Task Characteristics Suited to Long-Horizon Agents
Before assigning a task to a long-horizon agent, ask four questions:
- Can the goal be quantified? For example: throughput, latency, coverage, or pass rate.
- Can it be validated automatically? For example: tests, benchmarks, or contracts.
- Can failures be rolled back? For example: version control or snapshots.
- Can it run in isolation? For example: containers or sandboxes.
Only when all four answers are “yes” should the task be allowed to run autonomously for a long time. Otherwise, validation and isolation should be improved first, or human checkpoints should be retained.
8.2 Typical Scenarios
- Performance and cost optimization. Slow-query optimization, API latency reduction, and resource-usage improvement are essentially constrained multi-objective optimization problems, similar to the case discussed here.
- Legacy system modernization and dependency migration. Changes may span files and modules, with compilation and testing serving as validation.
- Test completion and defect governance. Tests can be added around failing cases, root causes located, fixes applied, and regression checks performed.
- Terminal and operations automation. Diagnostic commands and operations scripts can be generated and validated inside a sandbox.
- Information collection and organization. Information can be collected across multiple web pages and organized into reports with a unified structure.
8.3 Infrastructure: The Foundation for Long-Horizon Tasks
- Isolated execution environment. A container platform can provide each task with an independent workspace while restricting network access and resource usage.
- Reproducible verification loop. Builds, unit tests, integration tests, and benchmarks should be executable through a continuous-integration pipeline.
- State and artifact storage. Experiment logs, runtime logs, final code, and reports should be archived together for audit and review.
- Monitoring and metering. Logging, monitoring alerts, and usage statistics should be integrated, with cost thresholds set to pause tasks automatically when exceeded.
As one example, a cloud platform such as Baidu AI Cloud can offer general cloud product categories—such as model services, container-related products, object storage, logging, and monitoring/alerting—that can correspond to these needs. Enterprises can assemble a complete long-horizon task execution chain without building everything from scratch.
8.4 Human-in-the-Loop and Risk Governance
- Human in the loop. Key steps, such as solution approval, code merging, and deployment, should be reviewed by humans.
- Least privilege. Agents should only access repositories and data necessary for the task.
- Security review. Generated code should undergo static scanning, dependency analysis, and secret-leak detection to avoid introducing vulnerabilities or non-compliant dependencies.
- Prompt-injection protection. When tasks require reading web pages or external documents, external content should be treated as untrusted data, not instructions.
- Auditability. Every action and decision should be fully recorded so issues can be located, reviewed, and rolled back.
9. Significance and Boundaries
This progress can be understood at three levels.
- Technical level. Long-horizon tasks are closer to real software development. The corresponding optimization directions—context management, tool use, self-verification, and strategy switching—also feed back into model capability development.
- Industry level. On some programming tasks, the gap between Chinese models and international leaders has narrowed to within one point. Coding capability is an important foundation for agent deployment.
- Methodological level. It reminds us that AI should not be evaluated only by single-problem scores. The more important question is whether it can continuously accomplish real work under realistic constraints.
There are also clear boundaries:
- More than 600 iterations and more than 6,000 tool calls imply substantial computational cost. Cost-benefit analysis is necessary.
- Long-horizon capability depends heavily on verifiable objectives. Tasks with vague requirements or subjective standards still need humans to define and judge them.
- Public results come from controlled evaluation environments and cannot be directly extrapolated into enterprise-level returns.
For developers, the practical approach is to first build verifiable, rollback-capable, and isolatable infrastructure around key business tasks. Long-horizon agents can then be introduced gradually, starting with low-risk, measurable scenarios. For models, the next competition may no longer be about who answers the first question correctly, but about who can reliably carry a complex task through to the end.
