How to Scientifically Evaluate Long-Horizon Coding Capability
Leaderboard scores offer only limited insight into production-grade coding. This article presents an evaluation framework for long-horizon coding capability that covers legacy-system changes under security constraints, from-scratch product implementation, process-based metrics, layered acceptance criteria, and human oversight. The goal is to assess whether a model can understand long contexts, advance work across modules, and bring results to a runnable and testable state without treating test success as delivery readiness.
In software engineering practice, the coding capability of generative models has moved from “completing a few lines of a function” to “continuously modifying code in a complex repository.” Many teams are now asking a more fundamental question: can a model understand long contexts, advance tasks across modules, and bring results to a runnable, testable state? Leaderboard scores provide a useful reference point, but they rarely answer the key questions that arise in production delivery: Does a change break existing contracts? Are security constraints enforced? Can final acceptance be entrusted to the model without human oversight?
This article does not focus on a specific model. Instead, it abstracts the evaluation target as “long-horizon coding capability.” Long-horizon means that a task spans multiple files, services, and stages, often lasting from tens of minutes to several hours. In the middle of the task, the model must repeatedly read the repository, run tests, and revise based on failure logs. Evaluating this capability should cover both legacy-system modification and from-scratch implementation. The former tests understanding of historical code and security boundaries; the latter tests completeness from material analysis to product delivery.
Why Leaderboards Are Not Enough for Production-Grade Coding
Public evaluations usually divide tasks into relatively independent units: given a repository snapshot and a test suite, measure the pass rate within a limited number of steps. This is valuable for comparing basic capability, but it differs systematically from production environments.
First, real repositories carry historical debt. Interface naming is inconsistent, implicit conventions live in comments, and critical logic is scattered across scripts and configuration. If a model can only “see the current file,” it can easily make a locally correct change that breaks the system globally. Second, the goal is often not “make the tests green” but “deliver acceptable behavior under constraints.” For example, importing user assets must enforce permission isolation, and rendering motion effects must not execute untrusted scripts. Third, long tasks accumulate errors: one bad abstraction can pollute subsequent commits. Evaluation should therefore focus on sustained progress rather than the impressiveness of a single generation.
A more engineering-oriented indicator is whether the model can touch the contract layer, data layer, task state machine, API, asynchronous workers, frontend, and renderer within a monorepo, and verify the result end to end in a browser or integration test. Only cross-layer changes show that the model understands a system rather than a fragment.
Evaluation Dimension 1: Enforcing Security Constraints in Legacy Systems
A typical task can be described as follows: in a project with a substantial amount of legacy code, add the ability to “securely import user image assets and reference them in HTML motion scenes.” This requirement touches storage, permissions, content security, and frontend rendering at the same time.
Evaluation should be broken into several layers.
Context understanding. The model needs to locate the existing upload path, storage bucket policies, authentication middleware, asset metadata tables, and frontend reference patterns. If it simply adds a parallel implementation without reusing existing audit fields and lifecycle management, subsequent operational cost will rise sharply.
Making security constraints executable. “Secure import” should not remain a slogan. It should be implemented as follows: only authenticated users can write to their own namespace; object keys cannot be arbitrarily specified by the client; content type and size are restricted by an allowlist; HTML scene references must use escaping or sandboxing, and user-controlled strings must never be concatenated into executable scripts. Evaluation should check whether the model proactively proposes permission isolation rather than patching after tests fail.
Cross-module consistency. Once a contract changes—for example, asset ID, checksum, or review status—database migrations, worker consumption logic, API error codes, and frontend loading states must all be updated together. A model with strong long-horizon capability will maintain a “change list” and revisit it in later steps rather than forgetting after the edit.
A common failure mode in practice is that the feature works, but manual acceptance reveals a P1-level permission issue—for example, presigned URLs can be enumerated, thumbnails can be read across users, or the motion page embeds an asset URL directly into an executable context. This shows that a gap remains between “runnable” and “deliverable.” When permissions and final acceptance are involved, humans cannot let go; code review, threat modeling, and penetration-test checklists should be retained.
An illustrative check can be written as policy pseudocode for reviewers to compare against:
on upload(user, file):assert authenticated(user)key = namespaced(user.id, random_id())assert mime in ALLOWED and size < LIMITstore.put(key, file, acl=private)db.insert(asset, owner=user.id, status=pending)on render(scene, asset_id):asset = db.get(asset_id)assert asset.owner == current_user or shared_with(current_user)url = signed_get(asset.key, ttl=short)html = template.escape(url) # never inject into script
If the implementation generated by the model lacks namespaced, lacks a short TTL for signed_get, or writes the URL into an inline script, it should be judged as failing security acceptance even if the demo page can play the motion effect.
Evaluation Dimension 2: Completeness from Materials to a Playable Product
Another type of task is as follows: provide only domain learning materials and a product requirements document, and require the model to implement from scratch a knowledge-learning game that can actually be “completed.” Here, “completed” means the rules are executable, progress can be saved, and levels can be verified—not that concepts are displayed on a static page.
Evaluation should focus on the following:
Mapping requirements to architecture. Does the model break the PRD into domain models—levels, knowledge points, assessment items, and session state—interaction flows—questioning, answering, feedback, and unlocking—and a persistence plan? Implementations without a state machine often can demonstrate only the first level.
A testable rules engine. Completion conditions should be written as assertable rules, such as completing specified experiment steps, passing a set of questions, or achieving a goal in a simulated environment. The application layer should be able to run rule tests without a UI, avoiding a situation where it “looks playable but the logic is hard-coded.”
Material digestion and hallucination prevention. Learning products easily paste source material directly into copy or invent knowledge points that do not exist. Evaluation should sample and check whether explanations match the materials, whether example code runs, and whether error feedback points to real concepts.
End-to-end runnability. From initializing data, entering a level, submitting answers, to settlement, the product should start with one command in a local or container environment. Long-horizon capability is reflected in whether the model fills in missing seed data and fixes frontend-backend field mismatches rather than stopping at “the core algorithm has been provided.”
The two task paths are complementary: legacy modification exposes respect for constraints, while from-scratch implementation exposes the pursuit of completeness. Being good at only one of them is not enough to qualify as production-grade long-horizon coding.
How Post-Training and Evaluation Environments Affect Long-Horizon Performance
When base model sizes are similar, differences often come from post-training: whether the long-horizon task environment is expanded, and whether environment types cover terminals, browsers, repository toolchains, and security review. The closer the training environment is to real software engineering, the more likely a model is to learn to “read before changing, test before committing.”
For users, there is no need to focus on internal recipes. Instead, they should reproduce a similar environment in their own pipelines: provide a complete repository checkout, package management, test commands, browser or API probing, and log return. The more complete the tool loop, the easier it is to elicit long-horizon capability and objectively record it—steps, failure types, and rollback counts.
Cybersecurity-related capability can extend from “writing code” to “reviewing code”: identifying injection, privilege escalation, and deserialization risks during white-box reading, and providing remediation suggestions. Even when basic review is close to usable, full vulnerability exploitation or adversarial scenarios should not be handed to a model for automatic execution. Defensive review and offensive exploitation carry different risk levels, and evaluation and usage policies must be separated.
Open-source weights and reproducible deployment help teams verify models in isolated environments, but openness alone does not mean production readiness. After security assessment and model hardening, it is still advisable to integrate the model into R&D workflows under an internal network, least privilege, and audit logs.
Recommended Testing Process and Acceptance Gates
Making evaluation a repeatable experiment is more convincing than a one-off trial.
Prepare differentiated task packages. At minimum, use two types: an incremental requirement with legacy code and security constraints, and a from-scratch product with only materials and a PRD. Task descriptions should clearly state non-functional requirements—permission model, auditing, performance limits—so the model does not optimize only for the happy path.
Record the process, not just the result. Include time to first runnable state, time spent on subsequent fixes, scope of cross-file changes, number of red-to-green test cycles, and severity of defects found by humans. Long-horizon value lies in “sustained progress,” and process metrics can distinguish luck from capability.
Use layered acceptance. All automated tests passing is only the first layer. The second layer is the security checklist: authentication, object-level authorization, injection, and secrets not stored in the database. The third layer is product acceptance: boundary inputs, concurrency, rollback, and observability. If any layer fails, the result should not be described as “safe to deliver.”
Divide work between humans and models. Models are suitable for reading repositories, drafting cross-module changes, and iterating based on logs. Humans are responsible for threat modeling, deciding the permission model, and release decisions. Changes involving user data and execution contexts must be reviewed by humans.
If a team uses cloud object storage, message queues, container platforms, logging services, and monitoring alerts, these general capabilities can be included in the task environment: assets go into private storage, asynchronous scanning goes through a queue, rendering tasks run in containers, and audits are written to logs with alerts configured. The evaluation target is whether the model can combine these capabilities correctly, not whether it is tied to a specific vendor’s interface names.
Conclusion: Test with Real Projects, but Do Not Outsource Acceptance
Overall, new-generation coding models have pushed forward the upper bound of long-horizon software engineering across both domestic and open-source model routes: complex contexts can be read, cross-module changes can be advanced, and tasks can be brought to a runnable and testable state. This is meaningful progress for R&D efficiency.
At the same time, all tests passing does not mean the result is safe to deliver. Permission isolation, content security, and final acceptance remain human responsibilities. A reasonable attitude is to test with real projects, especially for long-horizon coding and cross-layer refactoring; maintain review at the security and release stages; and treat the model as a high-intensity collaborator rather than an unattended delivery system.
Building an internal evaluation set according to the framework above and continuously tracking the gap between “runnable” and “deliverable” will provide better guidance for model selection and process design than chasing external leaderboards.
