Evaluating Long-Horizon Code Models: From Runnable Code to Deliverable Systems

    This article proposes a reusable engineering evaluation framework for long-horizon code models. Instead of comparing leaderboard scores, it distinguishes three levels—runnable, testable, and deliverable—and examines two representative tasks: enhancing a legacy system under security constraints and building a complete agent knowledge-learning game from source materials and a PRD. It covers long-context consistency, security review, environment feedback loops, testing quality, auditability, human acceptance cost, practical boundaries, and an implementation checklist.

    Discussions about the new generation of long-horizon code models can easily turn into comparisons of leaderboard scores. But for R&D teams, the real question is not “Can it write code?” It is “Can it keep making progress in complex contexts and bring the result to a state that is runnable, testable, and acceptable for delivery?” Between these two questions lie permission models, security boundaries, regression validation, and release responsibility.

    This article does not score a specific product or version. Instead, it abstracts two typical tasks and discusses a reusable engineering evaluation method: one is enhancing a legacy system with historical code and security constraints; the other is building from scratch, based on materials and a requirements document, a complete playable agent knowledge-learning game. The former tests long-context understanding, cross-module changes, and security; the latter tests requirements analysis, architecture design, and product completeness.

    1. Define the Evaluation Goal First: Runnable, Testable, and Deliverable Are Three Levels

    “The model got the project running” only proves the first level. For engineering teams, at least three levels should be distinguished:

    1. Runnable: Dependencies can be installed, the build passes, and core paths can start. This level mainly verifies whether the model understands the existing project structure and basic commands.
    2. Testable: Key modules have automated tests, exception paths and boundary conditions are covered, and the model can locate and fix failures based on logs. This level verifies continuous iteration capability.
    3. Deliverable: Permissions, security, auditing, rollback, compatibility, and operational constraints have all been manually confirmed. This level cannot be judged only by “all tests are green.”

    In one abstract evaluation, the evaluated models completed two tasks: enhancing a legacy system and implementing a project from scratch. Both projects could eventually run, but the legacy system enhancement exposed a P1-level permission isolation issue during manual acceptance. This result is representative: a model can significantly speed up implementation, but it does not necessarily understand implicit security requirements such as “cross-path authorization consistency.” All tests passing does not mean the system is safe to deliver.

    2. What to Focus on in Version Upgrades

    From an engineering perspective, changes in the new generation of long-horizon code models do not necessarily come from replacing the base model. More noteworthy is the expansion during post-training: whether there are more long-horizon task environments, longer training time, and coverage of builds, tests, log troubleshooting, and multi-round repair in real engineering. This can be summarized into three observation areas:

    First, long-horizon software engineering capability. The focus is not single-file completion, but maintaining consistent changes across multi-package repositories, legacy code, asynchronous tasks, and frontend-backend interactions. The model needs to understand existing abstractions and reuse existing patterns as much as possible rather than starting a separate implementation.

    Second, security review capability. Beyond generating code, it should also be evaluated on whether it can perform white-box code review and identify permission bypasses, missing input validation, unauthorized access, and injection risks. It is important to recognize that performance on basic security tasks is not the same as full vulnerability exploitation or final security sign-off. A model can help find problems, but it cannot replace the security owner in making release decisions.

    Third, openness and verifiability. Open weights help teams conduct private evaluations and reproduce experiments in controlled environments, but they also mean the team must build its own security baseline, dependency governance, and release process. Whether and when to open weights should not become a reason to bypass acceptance.

    Public evaluation sets can serve as screening signals, but they cannot replace an enterprise’s own acceptance criteria. Benchmarks for terminal operations, software engineering, and automation tasks only show relative performance on certain task distributions. Historical code, organizational permissions, and business rules in real projects are often more complex.

    3. Task A: Legacy System Security Enhancement—The Hard Part Is “Boundary Consistency”

    The first task is to add “secure user image asset import and HTML motion scene references” to a system with a large amount of historical code. The requirement looks like file upload plus page display, but it actually crosses multiple modules: API contracts, database, task status, background processing, frontend editor, HTML renderer, and permission system.

    The security surface includes at least:

    1. Import side: tenant isolation, upload permissions, file type identification, size limits, malicious file scanning, filename injection protection, and storage access control.
    2. Reference side: scene edit permissions, asset ownership validation, cross-tenant reference blocking, short-lived access credentials, and audit records.
    3. Rendering side: HTML escaping, attribute allowlists, script isolation, sandboxed containers, Content Security Policy, and prohibiting direct interpolation of unprocessed user content into pages.

    In this task, the evaluated model showed strong long-horizon progress capability: it could understand the legacy system’s layered structure, make cross-module changes across APIs, data tables, background jobs, and frontend entry points, and continue fixing issues based on build errors and test failures. The problem appeared during security acceptance: the model added permission checks to the import API, but did not abstract authorization rules into a unified policy and reuse them on the second path, “scene references asset.” As a result, an asset that had passed upload validation was checked only for existence when referenced, without revalidating tenant ownership and reference permission. The main-flow tests passed, and the potential horizontal privilege escalation risk was found only during manual review.

    The following illustrative pseudocode shows the correct approach. The point is not specific APIs, but “unified server-side validation, non-bypassable tenant boundaries, and required state”:

    1. // Illustrative pseudocode: the import path must validate identity, tenant, type, and state together
    2. function canImportAsset(user, file, tenant) {
    3. assertTenantMatch(user.tenantId, tenant.id);
    4. assertPermission(user, "asset:import");
    5. assertAllowlistedMime(file.declaredMime, file.sniffedMime);
    6. assertSizeLimit(file.size);
    7. const storageKey = randomKey(file.sniffedMime);
    8. return {
    9. ownerId: user.id,
    10. tenantId: tenant.id,
    11. storageKey,
    12. status: "pending_scan"
    13. };
    14. }
    15. // Illustrative pseudocode: the reference path cannot assume that "asset exists" means "the current scene may reference it"
    16. function canReferenceAsset(user, scene, asset) {
    17. assertTenantMatch(user.tenantId, scene.tenantId);
    18. assertTenantMatch(scene.tenantId, asset.tenantId);
    19. assertPermission(user, "scene:edit", scene.tenantId);
    20. assertPermission(user, "asset:reference", asset.id);
    21. assertState(asset, "SCANNED");
    22. }

    The lesson from this case is: if the acceptance criteria only say “users can import images and display them in scenes,” the model is likely to implement only the happy path. The task description must explicitly define the threat model, for example: “assets must not be referenced across tenants,” “all reference paths must re-authenticate,” “assets must not enter rendering before scanning completes,” and “audit logs must not be forgeable by clients.” Even when these are written clearly, manual review is still required, because the model may place validation in a controller and miss background jobs, batch processing, or rendering processes.

    4. Task B: From Materials and Requirements to a Complete Playable Product

    The second task provides only a set of technical learning materials and a PRD, and requires implementing from scratch a complete playable agent knowledge-learning game. It does not depend on legacy code, and the focus is on completeness from requirements analysis to product delivery.

    This task can be broken down into several key stages:

    1. Domain modeling: clearly define learning objectives, levels, questions, hints, actions, scores, progress, and completion conditions, avoiding a demo with only pages and no state.
    2. State machine design: define state transitions for start, answering, scoring, failure and retry, hint consumption, level completion, and final completion to prevent progress corruption caused by page refresh or duplicate submissions.
    3. Content generation and validation: extract knowledge points from materials and generate scenario questions, but retain a human review step to avoid packaging incorrect knowledge as game feedback.
    4. Data persistence: save user progress, answer records, and scores, and consider concurrent submissions, idempotency, and exception recovery.
    5. End-to-end testing: cover the complete playthrough path, wrong-answer paths, invalid operations, refresh recovery, and boundary conditions.

    In this task, the evaluated model showed good breadth: it could split modules according to the requirements document, design data structures and a state machine, add basic tests, and implement the complete flow from start to completion. During manual experience, the game could be played through, and level progression and scoring logic were basically valid. But it still required human review of the accuracy of learning content, difficulty progression, and feedback quality. Being “playable through” does not mean “pedagogically effective”; generating questions does not mean the questions match the target audience’s cognitive path.

    5. Build a Reusable Evaluation Framework

    Rather than looking only at public leaderboards, it is better to design internal evaluations around the following dimensions:

    1. Requirements understanding and plan quality. Observe whether the model first identifies constraints, lists change surfaces, and explains risks before writing code. The plan does not need to be long, but it must include acceptance conditions and a rollback approach.
    2. Cross-module consistency. Check whether API contracts, data models, background jobs, frontend calls, and tests are updated together. In legacy systems, especially prevent “new logic connected only at the entry point while old paths still bypass it.”
    3. Environment feedback loop. Give the model executable builds, tests, and logs, and observe whether it can locate problems based on failure information rather than repeatedly guessing. In long tasks, the ability to fix regressions is more important than one-shot correct generation.
    4. Security and permissions. Write the threat model first, then let the model implement. Acceptance must specifically check horizontal privilege escalation, vertical privilege escalation, file upload, template injection, sensitive information leakage, and missing auditing.
    5. Test quality. Look not only at whether tests pass, but also at whether there are negative cases, permission cases, exception states, and concurrency scenarios. Tests with only happy paths have limited reference value.
    6. Change auditability. Check whether changes are concentrated within a reasonable scope, whether they include explanations, whether unnecessary dependencies are avoided, and whether rollback is easy.
    7. Human acceptance cost. Record the number of issues found manually, their severity levels, and the number of repair rounds. Calculate how much implementation time the model saved together with how much review time it added.

    Simple metrics can be set, such as time to first runnable state, test pass rate, number of regression defects, number of issues found by human review, number of P1/P2 issues, average repair rounds, and human hours required for final delivery. The metrics do not need to be complex, but they must be recorded consistently to judge the model’s real benefit to the team.

    6. Applicable Boundaries: Good for Long-Horizon Coding, Not a Replacement for Sign-Off

    Based on these two types of tasks, long-horizon code models are already worth controlled validation in real projects, especially in the following scenarios:

    • Feature enhancements across multiple modules in legacy systems;
    • Refactoring and migration with clear test entry points;
    • From requirements documents to runnable prototypes;
    • Supplementing tests, fixing build issues, and organizing technical documentation;
    • Solution exploration and comparison in isolated branches.

    But humans cannot let go in the following areas:

    • Final confirmation of permission models and security boundaries;
    • Production releases, data migration, and irreversible operations;
    • Compliance, auditing, and external commitments;
    • Professional accuracy of educational content;
    • Final acceptance of high-risk business systems.

    A more reasonable collaboration model is “model advances, system constrains, human signs off.” The model is responsible for continuous implementation and repair, the system provides least privilege, isolated environments, automated tests, and security scanning as safeguards, and humans are responsible for threat modeling, critical code review, and release decisions.

    7. Implementation Checklist

    1. Write clear acceptance criteria before the task starts, including not only happy-path features but also prohibited behaviors and the threat model.
    2. Provide an isolated environment for the model and restrict access to networks, secrets, and production data.
    3. Require all permission checks to be enforced uniformly in the server-side domain layer, not by hiding frontend entry points.
    4. Conduct separate security reviews for file upload, HTML rendering, template concatenation, and external content references.
    5. Require at least happy-path, exception, and permission negative tests; passing tests still requires manual sampling review.
    6. Control the scope of each change, submit by module, and retain change descriptions and rollback points.
    7. For legacy system enhancements, focus on “new paths compliant, old paths bypassed.”
    8. For from-scratch implementations, focus on product completeness, state consistency, and content accuracy.
    9. Before release, perform dependency scanning, secret scanning, log masking, and permission regression.
    10. Record the net benefit brought by the model, including saved hours, review cost, and defect density, rather than looking only at generation speed.

    The conclusion is: the new generation of long-horizon code models has indeed pushed the upper limit of complex coding tasks further. It can understand longer contexts, continue advancing cross-module changes, and bring tasks to a runnable and testable state; in from-scratch product implementation, it also shows strong planning and execution capability. But it is not yet an automated engineer that can be trusted to deliver simply because “all tests are green.” Permissions, security, content quality, and final acceptance must still be owned by humans. The most reasonable way to evaluate these models is not “Can it replace humans?” but “Within clear boundaries and acceptance mechanisms, how far can it push the team’s productivity?”