How to Evaluate Long-Horizon AI Coding Models

    As new AI coding models shift from proving single-generation correctness to sustaining engineering tasks over longer interaction sequences, evaluation methods must evolve. This article proposes a practical evaluation framework using contrasting tasks—extending a legacy system and building an end-to-end product from documentation—then examines long-horizon progress, constraint adherence, security, and delivery readiness. It also highlights why green automated tests are not enough and recommends security-focused acceptance criteria, tiered review, and observability as delivery standards.

    1. Conclusion first: capability ceilings are rising, so evaluation must evolve

    A common trend in recent AI coding model releases is that capability improvements are shifting from “is a single generation correct?” to “can the model keep pushing a task forward across a longer span of time and context?”

    In one public evaluation, an open-source model—using the same base model and comparable parameter count—led on several task categories simply by expanding long-horizon training environments during post-training. The signal is clear:

    • Optimization focus has shifted: scale is no longer the only variable; training environments and task design matter more.
    • Evaluation criteria need updating: assess completion of long-horizon engineering tasks, not just the quality of isolated code snippets.

    My overall judgment is that long-horizon AI coding models can already handle implementation work in real projects, but teams need a stronger evaluation framework and must add security acceptance into the workflow. The article covers three areas:

    1. How to design effective evaluation tasks;
    2. What can be observed in real projects;
    3. How to turn those observations into process improvements.

    2. Designing the evaluation: why prepare both “legacy extension” and “greenfield build”

    2.1 Task 1: Add a security-sensitive feature to a legacy system

    Task 1 asks the model to add two related capabilities to a project with substantial legacy code: safe import of user-uploaded image assets, and referencing those assets in HTML animation scenarios.

    The difficulty is not technological novelty; it is constraint density:

    • Legacy constraints: existing layering, naming conventions, error handling, and data models should not be arbitrarily changed.
    • Security requirements come first: images are user-uploaded content, so the import path must cover file-type validation, size limits, malicious-file detection, and access control.
    • Clear reference semantics: animation scenarios reference assets by unique identifier; if an asset is replaced or deleted, the behavior of referencing parties must remain predictable.

    This task tests whether the model can understand an existing system, add new functionality without breaking structure, and include sufficient security protections in new code.

    2.2 Task 2: Build end-to-end from materials to a playable game

    Task 2 provides only two inputs: a set of learning materials about agents and a concise product requirements document (PRD). The goal is to implement a knowledge-learning game from scratch and make it fully playable from start to finish.

    It tests a complete chain:

    1. Extract knowledge points and assessment dimensions from the materials;
    2. Convert product requirements into playable and verifiable game mechanics and level design;
    3. Implement the frontend, backend, and data layer as an integrated system;
    4. Ensure the main flow from game start to completion works without interruption in real execution.

    This task measures a combination of capabilities: material understanding, requirement implementation, system implementation, and end-to-end runnability. It is hard to pass with only good local code quality.

    2.3 Evaluation dimensions: four dimensions, not one score

    Before drawing conclusions, define what is being evaluated and from which angles. A long-horizon coding model can be evaluated along four dimensions:

    • Long-horizon progress: Can it keep advancing a task instead of repeatedly trying small local fixes?
    • Constraint adherence: Can it respect the existing architecture and conventions instead of introducing a parallel design?
    • Security: Does the implementation include reasonable protections, rather than satisfying only functional requirements?
    • Delivery readiness: Can the output move directly into testing and acceptance?

    For execution, these can be expanded into five observation points:

    1. Context understanding: Can it identify the modules, interfaces, and data models that actually need to change before making modifications, rather than patching the wrong place?
    2. Cross-module consistency: Can interface contracts, database changes, task state, interface layer, background jobs, and frontend interactions evolve together?
    3. Self-verification: Can it proactively run, test, and fix code, instead of delivering something that only looks reasonable?
    4. Edge-case handling: Does it cover non-happy-path concerns such as exception branches, concurrency and failure retries, and permission checks?
    5. Regression awareness: Do new capabilities break existing functionality?

    3. What changed: under the same base model, post-training determines the ceiling

    Because base-model scale remains unchanged, the observed capability improvements come almost entirely from post-training, mainly in three areas.

    3.1 Scaling long-horizon task environments

    The scale and variety of training environments increase significantly, and training duration also grows. This corresponds to a typical engineering capability: maintaining goal alignment over long interaction sequences and preserving state during cross-module changes. In engineering terms, it can be viewed as an implicit task checklist:

    1. # Completion criteria for a long-horizon task (illustrative)
    2. - [ ] Requirements understanding: identify involved modules and impact scope
    3. - [ ] Interface contracts: define inputs, outputs, and error codes
    4. - [ ] Data changes: table structures, indexes, and migration scripts
    5. - [ ] State transitions: task state machine and logic for each state
    6. - [ ] Interface layer: parameter validation, authorization, and error handling
    7. - [ ] Background jobs: asynchronous execution, failure retries, and idempotency
    8. - [ ] Frontend interaction: page states and exception messages
    9. - [ ] Rendering output: artifact generation and resource references
    10. - [ ] Verification loop: run, self-test, and regression fixes

    The key difference between a “long-horizon model” and a strong code-completion model is whether the model can keep advancing along a path like this.

    3.2 Security: from generating code to understanding risk

    Security-focused training has expanded from “writing functional code” to white-box code review and vulnerability reasoning. Under a white-box assumption, the model can see the full source code, making it better suited to answering “does this code contain a risk?” rather than “how can an attack be carried out in a target environment?”

    Common capabilities to examine include:

    • Identifying typical risks such as injection, broken access control, and sensitive information exposure;
    • Understanding how inputs flow into dangerous operations through call chains;
    • Providing actionable remediation suggestions, not only warnings.

    3.3 Open weights: verifiable, deployable, and continuously improvable

    Open weights allow teams to deploy and evaluate the model themselves, adapt it to private codebases, and continue improving it. Two points deserve attention:

    1. Complete security assessment before deployment: public releases usually undergo a round of security evaluation and hardening. Even after private deployment, teams should conduct additional assessment for business scenarios such as data compliance, content safety, and abuse prevention.
    2. Build permission management into the toolchain: in shared environments, different roles can be given different tool permissions. For example, arbitrary command execution can be disabled by default, directory access can be restricted, and high-risk operations can require explicit confirmation.

    4. Observations from real tests: long-horizon progress is stronger, but security acceptance remains weak

    4.1 Long-horizon progress: from isolated edits to cross-package coordination

    In repositories with multiple modules, code is often organized by package or module, each with its own responsibility boundary. The value of long-horizon capability appears in:

    • Coordinating across multiple modules in one pass: interface contracts, databases, task state, interface layer, background jobs, frontend interaction, and rendering output are designed as part of the same chain, rather than disconnected pieces;
    • Sustained work: a single task can run for tens of minutes to a few hours, during which the model must run, verify, and fix on its own;
    • Frontend-backend collaboration: it can handle both server-side logic and frontend interactions, rather than completing only one side.

    In terms of time spent, for a feature spanning frontend and backend—using HTML animation rendering as an example—an earlier comparable model took about one hour to implement and about another hour to verify and fix. On the new-generation model, similar tasks were more coherent, with less rework.

    4.2 Constraint adherence: extend the existing structure instead of starting a new one

    In legacy projects, the most important issue is not “write fast” but “write in the right place.” Key observation points include:

    • Does the new functionality reuse existing layering and error-handling conventions?
    • Are database changes accompanied by traceable migration scripts?
    • Do new interfaces follow existing parameter validation and authorization methods?
    • Does the new code break existing callers?

    A common mistake is that models tend to complete the main functional path first while leaving cross-cutting concerns such as authorization, validation, and auditing until the end—or missing them entirely. This is the root cause of the issue discussed next.

    4.3 A key finding: green tests do not mean ready for delivery

    Both projects passed, but during manual acceptance, the first project exposed a priority 1 (P1) permission isolation issue.

    The specific defect here represents a common category of defects, not an inherent problem of a particular class of tools. Its typical form is missing object-level access control: any logged-in user who has a resource identifier can read or even modify another user’s resource.

    Example of the defect pattern:

    1. // Defective pattern: checks only whether the user is logged in, not whether the user has access to the resource
    2. public MaterialDTO getMaterial(Long userId, Long materialId) {
    3. // Does not verify whether the material belongs to the current userId
    4. return materialRepository.findById(materialId);
    5. }

    The key fix is that object-level access control must be enforced at the moment a resource is read, not left to hidden frontend entries or caller discipline.

    1. // Fixed pattern: after reading the resource, verify whether its owner matches the current user identity
    2. public MaterialDTO getMaterial(Long userId, Long materialId) {
    3. MaterialEntity material = materialRepository.findById(materialId);
    4. if (material == null) {
    5. throw new ResourceNotFoundException("Material not found");
    6. }
    7. if (!material.getOwnerId().equals(userId)) {
    8. // Do not distinguish between "resource does not exist" and "access denied" to avoid leaking resource existence
    9. throw new AccessDeniedException("Access denied to this material");
    10. }
    11. return MaterialConverter.toDTO(material);
    12. }

    Why did automated tests fail to catch it? Reasons:

    1. Homogeneous test data: test cases usually use the same account, role, and tenant data, so they naturally cannot expose cross-subject access;
    2. Functional assertions take priority: tests focus on “is the functionality correct?” rather than “who is allowed to access it?”;
    3. Cross-cutting concerns are easily ignored: permission checks are cross-cutting logic, often missed during rapid iteration, and not reflected in main-path test results.

    Conclusion: all tests passing only proves that “the functionality works on the expected path”; it does not prove that “the system has clear boundaries between different subjects.”

    5. Practical recommendations: incorporate security and acceptance into the evaluation system

    5.1 Evaluation dimension comparison

    Dimension Core question Observation method Common failure mode
    Context understanding Does it identify the right place to change? Review modules and file distribution involved in the change Patching at the wrong layer
    Long-horizon progress Can it advance the task to a runnable state? Task duration, interaction rounds, and rework count Repeated trial and error or stalled progress
    Constraint adherence Does it respect the existing architecture? Code review, convention checks, regression tests Introducing a parallel implementation
    Security Does it include protections by design? Dedicated test cases for permissions, inputs, secrets, etc. Object-level unauthorized access or missing validation
    Delivery readiness Can it enter acceptance directly? First-pass rate, defect severity distribution Functional tests pass but critical defects remain

    5.2 Three engineering recommendations

    1. Turn security-focused test cases into acceptance baselines.

      • Evaluation sets should contain adversarial cases, not only functional cases. They should cover three scenarios: cross-user/cross-tenant access, abnormal and malicious inputs, and privilege-escalation paths.
      • For high-risk paths such as file upload, export/download, and asynchronous jobs, maintain a fixed checklist and enforce it before every delivery.
    2. Establish tiered acceptance and define non-negotiable boundaries.

      • Acceptance intensity should depend on risk: changes involving permissions, security, data consistency, or external interfaces must receive final human acceptance;
      • Internal tools, prototype validation, and non-sensitive modules can be more relaxed;
      • Model-generated code should pass code review and automated quality gates by default before entering human acceptance.
    3. Make observability part of delivery standards.

      • Require audit logs for key operations: who performed what action, at what time, and on which resource;
      • Record errors and denied access clearly to support post-hoc traceability and quick identification of unauthorized access attempts.

    5.3 A reusable evaluation checklist

    If you need to evaluate a long-horizon coding model, use the following checklist:

    • Prepare two contrasting tasks: one legacy-system extension and one greenfield build;
    • Record key metrics for each task: time spent, interaction rounds, rework count, and number of human interventions;
    • Run existing regression tests to confirm no existing functionality is broken;
    • Run security-focused test cases, with special attention to object-level access control;
    • Have a human complete final acceptance and record the defect severity distribution.

    6. Conclusion: the capability boundary is moving, but the role of humans has not changed

    The progress of long-horizon AI coding models is real: they can understand complex context, sustain work across multi-module repositories, and push tasks to a “runnable and testable” state. However, there is still a gap between “runnable” and “deliverable,” and that gap is shaped by permissions, security, and acceptance standards.

    For teams, the right approach is not to wait until models are strong enough to remove human oversight, but to:

    • Quantify capabilities with contrasting tasks;
    • Fill gaps with security-focused test cases;
    • Control risk with tiered acceptance.

    Models make implementation faster; humans guard the boundaries. This is the most pragmatic division of labor at this stage and the most important position to maintain when evaluating this class of models.