Evaluating Long-Horizon Coding Capabilities of Chinese LLMs
Leaderboard scores are not enough for production software engineering. This article explains how to evaluate Chinese open-source code models on long-horizon coding using legacy-system incremental work and from-scratch product builds, with process metrics and human gates for permissions, security, and final acceptance.
Why Long-Horizon Coding Matters More Than Leaderboards
Software engineering is rarely a single-file, single-turn task. Real projects usually include historical code, implicit conventions, cross-module dependencies, permission and security constraints, and delivery standards such as “must run, must be testable, must be acceptable.” When evaluating a new generation of code models, the focus should be on whether it can keep making progress on long tasks: understand complex context, plan the scope of changes, land in a runnable state, and remain restrained in security-related steps.
Leaderboard scores can reflect a model’s relative position in standardized environments, but they struggle to cover the key difficulties of production-grade software engineering: legacy system semantics, frontend/backend contracts, asynchronous task states, and permission isolation—issues that become high-priority defects once they go wrong. Only by shifting evaluation from “solving a few problems” to “bringing a real capability to acceptance” will the conclusions be useful for enterprise R&D decisions.
Evaluation Goal: Push the Ceiling to Runnable, Not Hand-Off Ready
A pragmatic conclusion can be stated this way: current leading open-source code models have pushed the ceiling for Chinese models in long-horizon coding further forward and are worth validating on real projects; but they are not yet at the point where “all tests green means fully hands-off delivery.” Permissions, security, and final acceptance still require human review.
This judgment comes from two very different types of tasks: incremental development in systems with existing code and security constraints, and from-scratch implementation of a fully experienceable product with only learning materials and requirements. The former tests understanding of old systems and security implementation; the latter tests completeness from material analysis to a product loop. Both being runnable does not mean human acceptance will pass—incremental projects may still expose top-priority issues such as permission isolation.
Understand Improvements Over the Previous Generation on the Same Baseline
Evaluate upgrades by fixing the baseline first. If the previous version could already work continuously in a monorepo, handling frontend/backend features such as contracts, data storage, task states, APIs, background workers, and the frontend and rendering pipeline, then the new version’s improvement should be judged by whether long tasks are more stable under the same complexity, cross-module changes are more coherent, and the fix loop is shorter.
A common upgrade path is not replacing the model with a larger base model, but scaling post-training: orders-of-magnitude more long-horizon task environments, richer environment types, and longer training cycles. Capability claims usually fall into three directions:
- Code engineering: more emphasis on terminal scenarios, repo-level changes, and production-grade software engineering, rather than single-point completion.
- Security-related capabilities: extending from “can write code” to white-box review and vulnerability reasoning; basic security tasks may approach frontier level, while full exploit chains still lag.
- Deployable and verifiable: before open weights, security evaluation and hardening need to be completed so teams can deploy, reproduce, and continue improving.
Evaluation suites can cover terminal environments, software engineering agents, automated workflows, tool-augmented hard Q&A, and economically valuable tasks. When reading scores, remember: leading scores do not equal passing acceptance.
Task 1: Secure Incremental Capability on Legacy Systems
A typical task is to add “secure user asset import and reference in dynamic page scenarios” to a project with existing code. The difficulty is not writing the import API, but satisfying several constraints at the same time:
- Understand old conventions: directory structure, naming, error handling, task state machine, and frontend/backend contracts must not be broken by new code.
- Put security constraints first: user uploads must validate type, size, content, and reference scope; untrusted content must not be directly concatenated into executable pages.
- Complete cross-module changes at once: storage, permissions, asynchronous processing, frontend references, and rendering isolation need the same semantics, not separate implementations.
- Runnable and testable: at minimum the main path must work, with automated or manual test entry points.
In practice, models often get the main chain running: import succeeds, scenarios can reference the asset, and pages can preview. But human acceptance may still find insufficient permission isolation—for example, asset visibility is too broad, cross-user references are not blocked, or rendering sandbox boundaries are unclear. Such issues are not obvious in feature demos, but they are blockers in security review. This shows that between “can make progress” and “can deliver” in long-horizon coding, there is still a layer of security semantics.
Evaluation should explicitly write constraints into the task: least privilege, default deny, reference allowlists, and untrusted content not executable. If the model only optimizes for “feature completion,” top-priority defects will surface at acceptance.
Task 2: From Materials to a Fully Experienceable From-Scratch Implementation
Another type of task has almost no legacy burden: only domain learning materials and product requirements are provided, and the model must build a knowledge-learning product that truly completes the full flow, including levels, feedback, progress, and completion conditions. It tests:
- Whether material extraction is accurate and whether it fabricates nonexistent knowledge points;
- Whether requirement breakdown covers main and exception flows;
- Whether interaction, state, save state, and judging form a closed loop;
- Whether “demoable” can become “completable.”
This type of task is easier to get running because there is no legacy security debt. It suits measuring product implementation completeness, but it cannot replace security evaluation on legacy systems. Both task sets must be done together for balanced conclusions.
Recommended Evaluation Method and Process Metrics
Do not only look at whether the final result compiles. Record process metrics:
- Context retention: after dozens of rounds, does the model still follow initial constraints and module boundaries?
- Change scope control: are there unrelated refactors, duplicate implementations, or destructive formatting?
- Fix loop: after browser or integration verification fails, can the model locate and converge instead of repeated trial and error?
- Security regression: are permissions, injection, path traversal, and untrusted rendering proactively considered?
- Testability: are assertions, fixtures, and minimal reproduction steps added?
- Human acceptance cost: number of P0/P1 issues and fix rounds.
Time dimension can also be compared: for the same cross-module feature, first implementation time versus subsequent verification and fix time. If verification and fix time approaches or exceeds implementation time, the model is still more “can write” than “can be stable.”
Delivery Boundary: Where Humans Cannot Let Go
Even if the model is suitable for real project trials, especially long-horizon coding, the following steps should still strongly require human involvement:
- Permission models and multi-tenant isolation
- Content security related to upload and rendering
- Keys, credentials, and audit logs
- Destructive migrations and data rollback strategies
- External contract compatibility
- Pre-release security and compliance checklist
The collaboration model can be defined as: the model handles cross-module progress and the main path, while humans handle threat modeling, acceptance tests, and release decisions. All tests green is necessary but not sufficient.
Practical Recommendations for R&D Teams
- Use real repositories, not toy repositories: include at least legacy burden and one security constraint.
- Use dual-task comparison: legacy increment plus from-scratch closed loop, scored separately.
- Write security as executable acceptance items: default deny, least privilege, and reference isolation need failure cases.
- Limit one-shot large refactors: require incremental commits, with each step rollbackable.
- Keep human gates: merge requests involving permissions and security must be human-reviewed.
- Continuously accumulate internal cases: turn this round’s P1 issues into a regression set, so the next evaluation does not only look at public leaderboards.
Conclusion
Evaluating a new generation of code models is not primarily about whether it “climbed the leaderboard again,” but whether it can continuously push tasks to runnable and testable states in complex contexts and make fewer structural errors under security constraints. The current level is worth validating long-horizon coding on real projects; but permissions, security, and final acceptance still need humans to handle the last gate. Design evaluation as two parts—“can run” and “can deliver”—so teams can use the model’s ceiling without bringing risk into production.
