Domestic LLMs Are Advancing in Coding: Long-Horizon Engineering Intelligence Emerges as the Next Turning Point
Recent progress in a Chinese large language model highlights a broader shift in AI-assisted programming—from one-shot code generation to sustained software engineering. Public evaluations and a vector-search optimization case show stronger capabilities in repository repair, tool use, terminal operations, reasoning, web tasks, and iterative performance improvement. The next stage of competition may depend less on isolated benchmark scores and more on how reliably models can plan, execute, validate, learn from failure, and operate within secure engineering workflows.
In recent years, competition among large language models in code generation has shifted from “can the model write code?” to “can it complete complex engineering tasks over time?” A Chinese LLM recently released an improved version and published evaluation results covering software engineering and complex task execution. Public information indicates that the model performed strongly across code repair, repository generation, terminal operations, knowledge reasoning, and web-based task benchmarks. Its score on one software engineering evaluation reached 58.4, exceeding some internationally advanced models evaluated during the same period.
However, simply asking whether a model ranks first does not fully capture the technical significance of this progress. The more important development is the model’s stronger long-horizon task capability. Instead of only producing a one-time answer, it can understand objectives, create execution plans, call tools to verify results, analyze failures, and continuously adjust its approach based on feedback. This is closer to real software development and may become the central battleground in the next phase of AI coding.
1. From One-Shot Code Generation to Continuous Engineering Iteration
Traditional code generation usually follows a short interaction cycle. A developer describes a requirement, the model generates code, and the developer then compiles, tests, and modifies it independently. Under this model, the main evaluation criteria are whether the first response is correct, whether the code is properly formatted, and whether it covers the basic functionality.
Real software engineering is much more complicated than a single question-and-answer exchange. Even a seemingly simple performance optimization task may require the following steps:
- Understand the existing code structure and business constraints.
- Identify performance bottlenecks or potential defects.
- Design multiple feasible approaches.
- Modify the code and complete the build.
- Run tests and benchmark programs.
- Analyze the results.
- Balance functionality, performance, and resource consumption.
- Continue refactoring and validation until the target is reached.
Engineering capability therefore cannot be equated with getting the answer right on the first attempt. More important is whether the model can continue making progress when the initial result is unsatisfactory, distinguish environmental issues from logical errors and flawed approaches, and select an appropriate next action.
This is the core of long-horizon tasks. Their objectives usually cannot be completed within a few conversational turns. Instead, they require dozens, hundreds, or even more rounds of continuous operations.
2. Public Evaluations Show Broader Capability Coverage
According to publicly available evaluation information, the model scored 58.4 on a software engineering repair benchmark, 42.7 on a test for generating complete code repositories from natural-language descriptions, 63.5 on a terminal-operation benchmark, 52.3 on a tool-assisted complex knowledge-reasoning evaluation, and 79.3 on web browsing and task-execution tests.
These figures represent different dimensions of capability.
Software engineering repair benchmarks primarily test whether a model can read a large codebase, locate problems, and submit usable changes. This is closer to practical development than simple function generation because the model must handle contextual dependencies, legacy code, testing constraints, and hidden edge cases.
Repository generation tests focus more on turning a requirements description into a complete engineering structure. The model must do more than write several functions. It must organize directories, configure dependencies, define module boundaries, and make multiple components work together.
Terminal-operation tests evaluate the model’s ability to work in a command-line environment. Tasks may include inspecting files, searching code, building projects, running tests, and troubleshooting errors. This capability is particularly important in software engineering because many problems cannot be resolved by reading text alone; the program must be executed to produce reliable evidence.
Tool-assisted complex reasoning tests measure how well the model coordinates external information, computational tools, and multi-step analysis. Web task evaluations extend this assessment by examining whether the model can search, filter, compare, and organize information in an uncertain environment.
Together, these results suggest that model capability is expanding beyond standalone code completion into a broader system covering understanding, planning, execution, validation, and summarization.
3. The Most Representative Change: Long-Horizon Optimization
A publicly described vector-search optimization task is particularly representative. The objective was to increase queries per second while maintaining a recall rate of at least 95%. The model was not working with an isolated code fragment, but with a system-level engineering project that required compilation, execution, and repeated testing.
After more than 600 iterations and over 6,000 tool calls, query throughput increased from approximately 3,547 queries per second to about 21,500 queries per second—an improvement of nearly six times.
The value of this case lies not only in the final number, but also in the engineering reasoning reflected in the optimization process. Performance optimization rarely has a single correct answer. The model may need to explore several directions, including:
- Adjusting the index structure to reduce irrelevant candidates.
- Optimizing memory layout to improve cache utilization.
- Reducing unnecessary data copies.
- Improving concurrency strategies to reduce thread waiting.
- Adjusting batching methods to increase hardware utilization.
- Finding an acceptable balance between accuracy and speed.
- Using benchmarks to identify actual bottlenecks rather than modifying code based on assumptions.
If a model simply repeats the same type of change, running for a long time is unlikely to produce meaningful gains. A more valuable capability is the ability to change direction based on test feedback. When one approach has limited impact, the model can move to data structures. When data-structure optimization reaches its limits, it can examine concurrency or memory access. If performance improves while recall declines, it can revert or retune the relevant parameters.
This indicates that the model is moving beyond code generation and beginning to form a closed loop of “propose a hypothesis, run an experiment, analyze the result, and revise the approach.”
4. Tool-Call Volume Is Not the Main Metric; Closed-Loop Quality Is
The number of tool calls alone does not measure intelligence. If a model repeatedly runs the same commands without learning from the results, a higher call count may indicate lower efficiency rather than stronger capability.
What matters is whether each call produces useful information and changes the next decision. An effective engineering agent should provide at least the following capabilities.
1. Task Decomposition
It should break a complex objective into stages such as environment inspection, code understanding, baseline measurement, localized modification, regression validation, and final summarization. This helps avoid large-scale refactoring before the system is understood.
2. State Management
It should continuously record attempted approaches, test results, and failure causes to avoid repeating ineffective operations. In long-running tasks, losing state can significantly reduce efficiency.
3. Result Attribution
A slowdown may be caused by algorithmic complexity, build configuration, input size, resource contention, or the measurement method. The model must distinguish among these factors instead of attributing every result directly to a code change.
4. Strategy Switching
When local fixes no longer produce improvement, the model should be able to change the algorithm, restructure data, or reconsider the optimization objective.
5. Risk Control
Performance gains cannot come at the cost of correctness. Each optimization requires corresponding tests, with particular attention to boundary inputs, exceptional flows, concurrency safety, and resource release.
Future evaluations of coding models should therefore consider more than the final score. Relevant metrics may also include completion time, tool-call volume, scope of code changes, number of regression defects, and result stability.
5. What This Progress Means for Developers
For individual developers, a coding model with long-horizon capabilities can take on more repetitive work. Examples include analyzing unfamiliar code, generating test cases, locating build errors, writing data migration scripts, organizing technical documentation, and conducting performance experiments on specific modules.
For engineering teams, the main value lies in automating parts of the development process. A model can act as an execution-capable engineering assistant involved in requirements decomposition, implementation, testing, and issue documentation. This does not mean developers can leave the process entirely. Architecture, data security, business rules, and production changes still require human review.
For enterprises, the goal is not simply to deploy a chatbot. The more important task is to build an auditable, reversible, and controllable intelligent development workflow. For example, organizations can:
- Isolate code repositories, test environments, and permission systems.
- Record model operations in logs.
- Require approval and provide rollback mechanisms for automated changes.
- Validate results in an independent test environment.
- Set access boundaries for sensitive code and data.
- Use continuous integration systems to check code quality automatically.
Only by embedding model capabilities into disciplined engineering processes can demonstration-level performance become stable production value.
6. Limitations That Still Require Attention
Despite progress in long-horizon engineering, these systems still have clear limitations.
First, benchmark environments differ from real production environments. Public benchmarks typically have clearly defined objectives and automatically verifiable outcomes. Real projects often involve ambiguous requirements, complex constraints, and incomplete or unreliable automated tests.
Second, long-running execution increases cost and uncertainty. A model may spend excessive resources repeatedly exploring a local optimum, or forget early constraints as the context becomes longer.
Third, performance improvements require careful verification. An improvement measured in one test may not transfer to different hardware, data volumes, or production workloads. Results should be validated across multiple datasets, environments, and extended runs.
Finally, models may generate code that appears reasonable but contains security risks, such as insufficient authorization checks, missing input validation, sensitive-data exposure, or concurrency defects. Automatically generated code must therefore undergo static analysis, dynamic testing, and human review.
7. Future Competition Will Shift from Model Parameters to Engineering Systems
The deeper signal from this progress is that competition in AI programming is moving from individual model capability toward complete engineering-system capability.
Leading solutions may need to combine:
- Strong code understanding and reasoning.
- Stable multi-turn task planning.
- Efficient tool-calling mechanisms.
- Robust context and state management.
- Reliable testing and validation systems.
- Strict permission, security, and audit controls.
- Integration with enterprise software development workflows.
In other words, the model is the core component of an intelligent development system, but it is not the entire system. Organizations that can effectively connect models with code repositories, build environments, testing platforms, monitoring systems, and permission frameworks are more likely to gain an advantage in real engineering scenarios.
Overall, the progress of Chinese LLMs in programming is significant not only because one evaluation score exceeded those of certain competitors. More importantly, it shows the potential to complete complex engineering tasks continuously. For developers, AI assistants may gradually evolve from tools that answer questions into collaborators that participate in the full engineering loop. For enterprises, the most valuable investment will be in building intelligent development infrastructure centered on security, validation, auditing, and process transformation.
