Evaluating Long-Horizon Code Models on Real-World Projects
Benchmark scores are a weak proxy for whether a code model can sustain work on a real project with legacy code, complex dependencies, implicit constraints, and incomplete documentation. This article outlines a multi-dimensional, reproducible evaluation approach centered on complete software delivery: understanding the codebase, implementing cross-module changes, verifying behavior, and delivering safely. It examines two scenarios—retrofitting a security-sensitive asset import feature into a legacy project and building a learning game from source materials and requirements—then discusses security checks, long-horizon behaviors, three-layer acceptance, and a practical human-AI collaboration workflow.
When evaluating a code generation model, benchmark scores only show its average performance on standardized tasks. They rarely answer a more practical question: when the model faces a real project with legacy code, complex dependencies, implicit constraints, and incomplete documentation, can it keep working for hours, complete cross-module changes, and ultimately deliver a result that is runnable, testable, and maintainable?
From an engineering practice perspective, the competitive focus for truly valuable code models has gradually shifted from “can it generate a correct piece of code” to “can it complete a full software engineering task.” Such tasks typically include requirements understanding, code retrieval, architecture analysis, solution design, file modification, test writing, error localization, regression verification, and documentation updates. The model must not only write code; it must also maintain goal consistency over a long workflow, correct errors in time, and understand how a local change affects the entire system.
Therefore, evaluating long-horizon code models should not rely only on a single generation result. It should establish a multi-dimensional, reproducible testing system around real projects.
1. Evaluation Focus: From Code Snippets to Complete Delivery
Traditional code evaluation usually provides a function signature or a clear requirement, and the model only needs to generate a local implementation. This approach is suitable for measuring syntax ability, algorithm ability, and basic programming ability, but it cannot cover the main difficulties in enterprise development.
Requirements in real projects often have the following characteristics:
- Code is distributed across multiple directories and modules.
- Legacy code has inconsistent naming, missing comments, and historical compatibility logic.
- Requirement descriptions contain many implicit constraints.
- Changing one module may affect the database, APIs, frontend, and background tasks.
- Automated tests cannot cover all security and user-experience issues.
- A task may require multiple rounds of debugging rather than a single generation.
Therefore, evaluation should cover at least four dimensions.
The first is “can it understand.” Can the model quickly build an understanding of the project structure, identify entry files, core data flows, dependencies, and existing conventions?
The second is “can it implement.” Can the model break requirements into reasonable subtasks and keep interfaces consistent across multiple modules?
The third is “can it verify.” Does the model proactively run tests, builds, static checks, and necessary manual verification, rather than declaring completion based only on surface-level code correctness?
The fourth is “can it deliver safely.” Does the model consider permissions, input validation, file access boundaries, data leakage, exception handling, and backward compatibility, rather than pursuing only a working feature?
2. Test Scenario 1: Security-Sensitive Feature Retrofit in a Legacy Project
The first scenario suitable for testing model capability is adding a new feature to a project that has already been running for some time. For example, add user image asset import to a system and allow those assets to be referenced by HTML animation scenes.
At first glance, this requirement seems to be only “upload a file and display it,” but it actually involves multiple technical layers at the same time:
- Frontend file selection, upload progress, and error messages;
- Backend APIs, authentication, and permission checks;
- File type, size, and content validation;
- Storage paths, naming rules, and lifecycle management;
- Database records, status changes, and relationships;
- Asynchronous tasks or background processing;
- HTML scene rendering and resource references;
- Legacy data compatibility and exception rollback.
If the model only modifies the upload API and page component, it may produce a demo version that cannot meet production requirements. A more reliable implementation should first review the existing resource model, user identity model, and task state flow, then determine new fields, API boundaries, and resource access methods.
1. Understand the System Before Making Changes
When facing legacy code, the model should not immediately create new files. Instead, it should complete a round of structured exploration:
- Review project directories and module responsibilities;
- Search existing code related to uploads, resources, users, and permissions;
- Find database migrations and data model definitions;
- Check existing frontend and backend error-handling patterns;
- Confirm task states and asynchronous execution mechanisms;
- Run existing tests to establish a baseline before changes.
The value of this process is avoiding duplicate implementation. Many projects already have generic file handling, identity verification, or task scheduling mechanisms. If the model ignores them and introduces another set of logic, the result may run in the short term but increase maintenance costs over time.
2. Cross-Module Changes Require Contract Consistency
A complete feature usually requires changes to the data model, API layer, business service, background processing, and frontend display at the same time. If field names, state values, or error semantics are inconsistent at any layer, problems will appear at runtime.
For example, if the backend defines resource states as “processing, available, failed” while the frontend checks “pending, ready, error,” the page may load forever or fail to display error states.
A strong model should clarify the following contracts before making changes:
- Request parameters and response structures;
- The complete lifecycle of resource states;
- Failure scenarios and error messages;
- The difference between insufficient permission and a missing resource;
- Whether retries are idempotent;
- How historical references are handled after a resource is deleted;
- Whether older client versions can still work normally.
This is also an important standard for judging whether a model has engineering thinking. The amount of code does not represent completeness; API contracts and boundary handling determine whether a feature can run over the long term.
3. Security Cannot Be Replaced by Passing Tests
In actual testing, all automated tests passing does not mean the feature is secure. This is especially true for scenarios involving user uploads, resource references, and permission control, where security issues often hide in paths not covered by tests.
A typical risk is failed permission isolation: although the system verifies that a user is logged in, it does not verify whether that user owns the target resource. An attacker may then access other users’ files by modifying a resource identifier. This type of issue does not affect normal user flows, and regular functional tests may all pass, but it is a serious access control flaw.
For file import and resource references, at least the following checks should be performed.
1. Separate Authentication from Resource Authorization
“The user is logged in” only proves identity; it does not prove that the user has the right to access a resource. Every time a resource is read, modified, deleted, or referenced, the system should verify resource ownership, project affiliation, or an explicit authorization relationship.
2. Do Not Trust Paths Provided by the Client
File names, directory names, and resource identifiers should all be treated as untrusted input. The server should use internally generated unique identifiers and controlled directories to avoid path traversal, file overwriting, or access to arbitrary paths.
3. File Type Cannot Be Checked by Extension Alone
Extensions can be forged. The system needs to validate based on file size, content characteristics, and parsing results, and reject files that cannot be parsed safely. For content that may be executed by a browser, response headers, download policies, and isolation methods should also be considered.
4. Resource References Must Be Boundary-Limited
When HTML animation scenes reference external resources, the accessible scope should be limited. This prevents the resource reference mechanism from being used to read internal files, access unnecessary network addresses, or inject user input directly into page structures.
5. Exceptions and Logs Must Not Leak Sensitive Information
Error messages should be sufficient to help locate the problem, but they should not directly return server paths, internal object identifiers, database details, or other user information. Logs should also avoid recording full credentials and sensitive content.
These checks cannot be fully automated by the model. The model can assist with code review, test generation, and enumeration of attack paths, but developers still need to perform manual verification based on how the system is actually deployed.
4. Test Scenario 2: Building an Application from Scratch with Materials and Requirements
The second representative scenario is to provide only a set of domain learning materials and a product requirements document, and ask the model to build from scratch a knowledge learning game that can run completely and be completed.
This task differs from retrofitting a legacy project. There is little historical code to reference, so the model needs to complete requirements analysis, information distillation, gameplay design, page implementation, and test verification on its own.
When building an application from scratch, a common problem is not “it cannot write code,” but that the product loop is incomplete. For example:
- The page opens, but there is no clear start or end state;
- Questions can be displayed, but answer judgment is inaccurate;
- Users receive no feedback after completing an action;
- Scores exist but do not affect subsequent flow;
- Level design is disconnected from the learning materials;
- The game runs but cannot actually be completed.
Therefore, the evaluation focus should shift from the quality of individual pages to the complete user journey.
A deliverable learning application should have at least:
- A clear startup entry point;
- Understandable task goals;
- A continuous interaction flow with feedback;
- Correct knowledge-point or rule judgment;
- Hints and correction mechanisms after errors;
- Progress, score, or level state;
- Final completion conditions;
- The ability to run repeatedly and restart.
If the model can transform knowledge points from the materials into questions, rules, feedback, and level goals, it shows that it has completed not only code generation but also basic product abstraction.
5. How to Observe Long-Horizon Task Capability
Long-horizon tasks are the best way to expose the difference between a model and an ordinary code completion tool. A single task may last a long time, during which build failures, test failures, dependency conflicts, requirement ambiguity, and implementation-direction adjustments may occur.
When evaluating a model, focus on the following behaviors.
1. Does It Establish a Task Plan?
Does the model first decompose requirements and clarify the modules involved, implementation order, and verification methods? A reasonable plan does not need to be complex, but it should reduce the risk of omissions.
2. Does It Maintain Context Consistency?
When a task spans multiple files and multiple stages, can the model remember previously determined data structures, naming conventions, and business goals, and avoid contradictions with later changes?
3. Can It Fix Issues Based on Feedback?
Real engineering capability is not about generating everything correctly on the first try. It is about locating the root cause after an error occurs. The model should distinguish environment issues, API issues, state issues, and logic issues, rather than repeatedly modifying unrelated code.
4. Does It Proactively Verify Critical Paths?
Does the model run builds, unit tests, integration tests, and end-to-end flows? Does it check boundary conditions, rather than only verifying that “the page opens”?
5. Does It Know When Human Confirmation Is Needed?
When there is ambiguity in security, product, or architecture, the model should not make high-risk decisions on its own. Being able to clearly identify uncertainty and request confirmation is itself a sign of maturity.
6. All Tests Green Does Not Mean Ready to Launch
Automated tests mainly verify prewritten assertions, while production issues often come from unforeseen scenarios. This is especially true for permission, security, user experience, and compatibility issues, which cannot be fully covered by test results alone.
It is recommended to divide acceptance into three layers.
Layer 1: Machine Verification
This includes compilation, static checks, unit tests, API tests, integration tests, and end-to-end tests, used to confirm basic functionality and regression risk.
Layer 2: Scenario Verification
Execute key tasks according to real user flows, such as creating resources, uploading files, cross-user access, repeated submission, failure retry, and referencing a resource again after deletion.
Layer 3: Manual Review
Focus on permission boundaries, exception experience, data leakage, maintainability, code complexity, and whether unnecessary technical debt has been introduced.
Only when all three layers meet requirements is the feature suitable for release evaluation. For features involving permissions and sensitive data, dedicated security review and adversarial testing should also be added.
7. A Practical Human-AI Collaboration Model for Enterprises
Long-horizon code models are already suitable for participating in real development, but a more reasonable role is “high-intensity engineering assistant” rather than a complete replacement for developers.
The following working model is recommended:
- First let the model read the code and documentation and output its architecture understanding and risk list;
- Developers confirm task boundaries, technical approach, and security constraints;
- Let the model implement in stages, running verification after each module is completed;
- Require the model to record modified files, test results, and unresolved issues;
- Manually review code related to permissions, data handling, and external input;
- Run builds and tests in an isolated environment;
- Perform full regression and change audit before merging.
This process can leverage the model’s strengths in retrieval, coding, debugging, and documentation generation while preserving developer control over architecture judgment, security decisions, and final responsibility.
Conclusion
Evaluating a long-horizon code model should not ask only “can it write code.” It should further ask: can it understand a real system, maintain consistency across multiple modules, move requirements to a runnable state, pass tests and proactively identify risks, and expose problems in time when uncertainty exists?
From legacy project retrofits to new application builds, long-horizon code models already have the foundation to participate in real software engineering. They are particularly suitable for cross-file changes, repetitive implementation, test completion, error localization, and initial solution implementation.
However, for permission isolation, sensitive data, production release, and final acceptance, manual review remains irreplaceable. A more mature evaluation approach is not to pursue one-shot completion of all work by the model, but to establish a complete delivery process that includes planning, implementation, verification, security checks, and human confirmation. Only then can model-generated code move from “it runs” to “it is trustworthy.”
