Evaluating Long-Horizon Coding: Methods and Delivery Boundaries

    Long-horizon coding is the ability to work across files, modules, and historical constraints until software is runnable and testable. This article explains complementary evaluation tasks, process metrics, capability layers for implementation versus security, a reusable acceptance checklist, and human sign-off rules for production use.

    Long-horizon coding is a model’s ability to keep context, apply changes, and reach a runnable, testable state in multi-file, multi-module systems with historical constraints. It is not the same class of work as completing a single function: context is longer, dependencies are more implicit, errors spread across interfaces, and acceptance is behavioral correctness, unbroken constraints, and controlled regressions—not syntactic validity.

    Before putting this capability into a real codebase, probe it with two complementary tasks rather than relying only on public leaderboards.

    Designing tasks that actually measure the ceiling

    The first class is brownfield change. Use a project that already has history, data contracts, and an authorization model. Ask for a constrained business path, for example: uploaded image assets may enter only specified storage and rendering paths and may be referenced safely by motion-effect scenes. This tests three things at once: reading existing module boundaries, extending interfaces without breaking the current state machine, and encoding security constraints in the implementation rather than in comments.

    The second class is from materials to a working product. Provide only domain study materials and a requirements note, and require a completable, repeatably verifiable interactive application from scratch. This tests requirement decomposition, state design, content correctness, and end-to-end playability—whether analysis becomes executable structure instead of stopping at a design write-up.

    The two tasks should differ enough: one emphasizes constraints and compatibility, the other completeness and closed loops. Using only one class tends to overestimate or underestimate long-horizon skill.

    Process metrics carry more information than a final score

    Record process facts, not only whether something “ran”:

    • Time for the first implementation versus the share spent on verification and repair. If repair time approaches or exceeds the first pass, the model can start work but does not converge well.
    • Whether it can continuously change contracts, persistence, task state, APIs, background jobs, and front-end rendering in a monolith or multi-module repo, rather than only the presentation layer.
    • Whether context is used correctly: historical names, error codes, permission models, and existing tests are continued, not replaced by a parallel implementation.
    • After failure, whether it narrows the search or repeatedly rewrites large regions.
    • Whether the result is runnable and testable: critical paths have automation or repeatable manual steps, not merely “looks finished.”

    Long-horizon training usually grows environment scale, type, and duration so models get used to multi-step tool use and delayed feedback. Evaluation should match that: real dependencies, observable failures, and specific feedback. Overly clean toy repos do not measure production engineering skill.

    Capability layers: generation, review, and exploitation are not the same

    Strong code generation is not strong security. In practice, split at least three layers:

    1. Functional implementation: can the required path be completed.
    2. White-box review and vulnerability reasoning: can it point out issues such as authorization bypass, path traversal, unsafe rendering, and sensitive data in logs, with reasons tied to code locations.
    3. Full attack chain: from discovery to stable reproduction, bypassing defenses, and impact assessment. Many general coding models can approach usable quality on the first two layers; the third remains unstable and must not be treated as a substitute for penetration testing.

    Security requirements therefore need their own acceptance items and cannot be covered by “functional tests passed.”

    A reusable acceptance checklist

    For a requirement such as “import assets and reference them in motion-effect scenes,” minimum acceptance should include:

    • Identity and authorization: only the resource owner or an authorized role can import, list, and reference; cross-user identifiers must fail.
    • Type and size: allowed media types, dimensions, and volume are validated on the server, not only in the client.
    • Storage isolation: object keys cannot be chosen arbitrarily by the client; path concatenation must not read other tenants or system files.
    • Reference safety: addresses in scene descriptions may point only to registered resources; the renderer rejects arbitrary external scripts and unescaped markup.
    • State consistency: if upload, transcode, or registration fails, task state, database records, and object storage must not leave orphans.
    • Regression: existing motion-effect generation, query, and permission cases still pass.

    Illustrative server-side constraints can be written as policy rather than scattered checks:

    1. allow import if
    2. caller owns target_project
    3. and media.type in allowlist
    4. and media.size <= limit
    5. and object_key matches tenant_prefix(caller)
    6. deny otherwise

    If the model only adds a login check in a controller while the renderer trusts addresses in the scene, treat the work as incomplete even if playback appears to work.

    Boundaries humans cannot hand off

    All-green tests only mean covered assertions hold. In long-horizon changes, permission isolation, default deny, error branches, and historical-data compatibility are often untested. A real trial can show a working functional path while human review still finds high-priority defects such as cross-user resource references. That can justify trying the model on real projects, especially for long-horizon coding and cross-module work; authorization, security, and final acceptance still require human review.

    Suggested release rules:

    • Suitable for the model to draft and iterate: boilerplate, cross-module wiring, filling in tests, and bounded fixes from failure logs.
    • Must be human-signed: authentication and tenant isolation, secrets and tokens, externally exposed surfaces, data deletion and retention, and any change to trust boundaries.
    • Do not expand privileges after one successful demo: run in an isolated environment first, then widen data and account scope gradually.
    • Keep small, rollback-friendly commits so model changes can be distinguished from environment drift.

    Comparing versions

    If base-model scale is unchanged, gains usually come from post-training: longer tasks, more environment types, longer training and feedback. When comparing, do not cite only composite leaderboards. Fix the same repository, the same requirement, and the same checklist; record first-pass time, repair rounds, failed security items, and human rework. That measures change in engineering usefulness, not a shift in question mix.

    Public evaluations can cover terminal use, software-engineering tasks, tool use, and office automation, and they are useful for relative standing. Go-live decisions should still follow failure modes on your own code. If open weights are released after security evaluation and hardening, teams can reproduce experiments, but the environment, prompts, and tool permissions must match the report or results are not comparable.

    Practical rollout

    Establish a baseline with one moderately complex brownfield change and one small product built from scratch to a closed loop. Write failing cases for authorization, input validation, and state consistency. Treat model output as a patch under review, not a delivered version. Do not merge while security items remain open. Under this standard, long-horizon coding models can speed implementation, but delivery responsibility stays with people.