Long-Horizon Software Engineering Tasks Are Becoming the New Battleground for Chinese AI Models
Chinese foundation models are moving beyond one-shot code generation toward sustained software engineering work. This article examines their progress in code repair, repository generation, terminal and web tasks, and performance optimization, while explaining why tool use, iterative testing, state management, and human oversight are critical to reliable engineering agents. It also discusses evaluation metrics, development workflow changes, and the remaining security and reliability limitations.
In recent years, competition among AI coding models has shifted from whether they can generate a piece of code to whether they can independently complete a complex software engineering task. The difference is significant. Code generation is closer to autocomplete, while real engineering requires a model to understand requirements, inspect an existing codebase, design a solution, run tests, analyze failures, and continuously revise its approach based on feedback.
A recently upgraded Chinese foundation model has reported new results across several areas, including software engineering repair, natural-language-to-repository generation, terminal operations, knowledge reasoning, and web-based tasks. Publicly available information indicates that the model scored approximately 58 points on a comprehensive software engineering benchmark and also demonstrated strong task-completion capabilities in terminal and web environments.
However, scores alone do not fully capture the value of this progress. More significant is the fact that these models are beginning to demonstrate behavior closer to real engineering practice. When facing a complex problem, they are no longer limited to a single generation and submission. Instead, they can perform hundreds of operations, invoke many tools, inspect intermediate results, change technical directions, and gradually approach a solution through repeated experimentation.
1. From One-Shot Generation to Long-Horizon Task Execution
Traditional code-generation benchmarks usually have clearly defined boundaries. A model may be asked to complete a function based on its specification or modify a particular file to fix a reported defect. These tasks are easy to evaluate and can measure basic coding ability, but they remain far removed from real software development.
A real engineering task often includes the following steps:
- Understand business goals and constraints;
- Read the dependencies between multiple directories and modules;
- Locate the code that actually affects the result;
- Design an initial implementation;
- Modify the code and run tests;
- Use error logs to identify problems;
- Analyze performance and resource requirements;
- Balance functionality, stability, and efficiency;
- Refactor repeatedly until the result reaches an acceptable quality level.
The challenge is therefore not simply to write correct code. It also involves context management, task decomposition, state tracking, error recovery, and solution selection.
Long-horizon tasks generally have three characteristics. First, they involve enough steps that they cannot be completed in a single response. Second, intermediate results influence later decisions, so the model must maintain an understanding of the current state. Third, there is often no single correct solution, which means the model must compare alternatives through experimentation.
For these reasons, long-horizon capabilities are closer to the working style of an engineer than that of a simple code generator.
2. What a Performance Optimization Case Reveals
In one vector retrieval optimization task, the model had to improve an approximate nearest-neighbor search algorithm within an existing system skeleton. The solution needed to maintain a recall rate of approximately 95% or higher while maximizing queries per second.
The difficulty was that performance gains could not come at the expense of result quality. Simply reducing computation might lower recall, while aggressively improving recall could significantly reduce query speed. The model therefore had to balance multiple metrics at the same time.
According to the publicly described case, the model completed more than 600 iterations and over 6,000 tool calls. It increased query throughput from approximately 3,547 queries per second to about 21,500 queries per second, representing an overall improvement of nearly six times. Regardless of the exact testing environment, the result illustrates an important trend: when a model can execute, measure, and correct its work, its approach to problem-solving changes fundamentally.
A typical process might look like this:
- Understand the data structures, index design, and query pipeline;
- Run a baseline test to identify the current bottleneck;
- Determine whether the bottleneck comes from distance calculation, memory access, thread scheduling, or index traversal;
- Modify one part of the implementation;
- Run the benchmark again and observe changes in throughput and recall;
- Revert the approach if the improvement is limited;
- Try alternative index strategies, caching methods, or parallelization mechanisms;
- Compare different parameters and implementation paths;
- Keep the version with the best overall performance.
The real value is not that the model guessed one effective optimization. It is that the model could organize optimization as a continuous experimental loop involving hypothesis formation, validation, result analysis, and strategy switching. This goes beyond the traditional meaning of code completion.
3. Tool Use Is the Foundation of Engineering Intelligence
Long-horizon programming tasks cannot be completed using only the model’s internal knowledge. The model must interact with an external environment to obtain reliable feedback. Common tools include:
- File reading and directory search;
- Code editing and patch application;
- Compilation, execution, and automated testing;
- Log inspection and error diagnosis;
- Performance benchmarking;
- Static analysis and dependency checks;
- Version and diff comparison;
- Data queries and result visualization.
Tool use alone does not constitute intelligence. The key questions are whether the model knows when to use a tool, which tool to select, and how to interpret the returned information.
For example, a compilation failure and a performance regression are different types of problems. The former usually requires checking syntax, types, and dependencies. The latter requires deeper analysis of time complexity, hot functions, memory allocation, and concurrency behavior. If a model merely repeats the same operation, increasing the number of tool calls may waste resources rather than improve the result.
An effective engineering model should be able to:
- Maintain task state: Know what has been completed and what is currently blocking progress;
- Control operation granularity: Avoid changing too many files at once, which makes diagnosis difficult;
- Interpret feedback: Distinguish transient errors, environment issues, and logic defects;
- Switch strategies when necessary: Adopt a different approach when the current one remains ineffective;
- Preserve recovery paths: Establish clear rollback points before making large-scale changes;
- Control cost and risk: Avoid running tests indefinitely or repeatedly modifying stable modules.
These requirements place greater demands on planning, context management, and self-correction.
4. How Should Engineering Benchmark Results Be Interpreted?
Software engineering benchmarks typically cover multiple dimensions. Code repair tests evaluate whether a model can understand a defect and submit an effective fix. Code repository generation tests examine whether it can build a relatively complete project from natural-language requirements. Terminal-operation tests measure whether it can complete a sequence of tasks in a command-line environment. Web-based tasks evaluate information retrieval, page understanding, and action execution.
Because these tests measure different capabilities, no single score should be treated as a complete representation of a model’s engineering ability.
For example, a strong code repair score indicates good code comprehension and problem-localization skills, but it does not necessarily mean that the model can design a large system from scratch. Strong terminal performance suggests that the model can use a development environment, but it is still necessary to examine whether it performs excessive or ineffective operations. Strong repository-generation results indicate some level of system-design ability, but the maintainability, security, and long-term stability of the generated project must also be assessed.
A more meaningful evaluation should consider:
- First-attempt success rate;
- Final completion rate for multi-step tasks;
- Average number of tool calls;
- Task completion time;
- Scope of code changes;
- Number of regression defects;
- Changes in test coverage;
- Magnitude of performance improvement;
- Resource consumption;
- Frequency of human intervention.
Human intervention is particularly important. If a model appears to complete a task but still requires frequent developer involvement at critical stages, it functions more as an assistant than as an autonomous engineering agent.
5. The Impact on Software Development Workflows
The progress of these models will first change how developers collaborate with development tools.
In the past, developers typically asked models to solve local problems, such as completing a function, explaining an error, or generating an API. In the future, they may provide a higher-level objective, such as reducing query latency without lowering result quality, and allow the model to perform analysis, implementation, and verification.
This does not mean that developers will be completely replaced. Instead, their work may shift toward:
- Defining requirements and acceptance criteria;
- Designing system architecture and key constraints;
- Reviewing the technical direction proposed by the model;
- Assessing performance, security, and maintainability risks;
- Establishing automated testing and monitoring;
- Reviewing the final changes and confirming responsibility for the result.
In other words, developer value will increasingly lie in problem definition, solution judgment, and quality control.
For organizations, long-horizon coding models may also enable more automation in research and development workflows. Potential applications include legacy system migration, test-case completion, log analysis, performance regression testing, documentation generation, and initial defect triage. These uses must be supported by permission isolation, operational auditing, data protection, and human approval mechanisms.
6. Important Real-World Limitations
Despite the progress in long-horizon task execution, these systems are still far from fully reliable.
First, errors can accumulate over long tasks. An incorrect judgment early in the process may be repeatedly reinforced over dozens of later operations.
Second, a model may optimize a local metric at the expense of system-level behavior. For example, improving throughput through changes to caching or parallel execution could introduce data-consistency issues.
Third, testing environments differ from production environments. An optimization that performs well on a fixed dataset and standard hardware may not work equally well in a real business system.
Fourth, tool calls can be expensive. Repeated execution, compilation, and testing consume computing resources and extend task duration.
Fifth, security risks cannot be ignored. If a model has permission to write files, execute commands, or access data, it could accidentally delete files, expose sensitive information, or introduce insecure dependencies.
In practical deployments, organizations should therefore use sandboxed environments, least-privilege permissions, operation allowlists, automated backups, change auditing, and staged approval processes. Database changes, production services, and core business code should continue to require human review before release.
7. The Broader Significance of This Progress
The improvement of Chinese foundation models in programming is not important merely because one model achieved a higher benchmark ranking. Its deeper significance is that it reflects the transition of AI from generating content to completing tasks.
If a model can only produce a plausible-looking code fragment, its primary value is improved input efficiency. If it can continuously understand a complex objective, invoke tools, verify results, and revise its approach, it begins to participate as an execution agent in the software engineering process.
In the future, important evaluation criteria for coding models may no longer focus only on whether the first attempt is correct. They may also ask whether a model can:
- Understand the context of a real project;
- Make trade-offs under complex constraints;
- Sustain tasks lasting several hours;
- Detect its own mistakes;
- Prove that a change works through verifiable results;
- Operate reliably within defined security boundaries.
From this perspective, long-horizon task execution represents an important direction for AI programming. It will not immediately replace experienced development teams, but it will gradually change the division of labor in software development. Engineering work will increasingly move from manual coding toward goal specification, automated execution, continuous verification, and intelligent collaboration.
