How to Evaluate the Long-Horizon Coding Ability of Large Language Models

    As open-source models close the gap with frontier proprietary systems, the real question is no longer whether a model can ace a single benchmark task but whether it can carry a full, continuously evolving codebase. This article explains why benchmark scores fail to capture long-horizon coding ability, then presents two complementary test projects—adding a feature to an existing codebase and building a learning game from scratch. It shows how one project passed every self-test yet still exposed a P1 access-control flaw during manual review, and closes with a reusable evaluation framework covering requirement understanding, cross-module progress, self-verification quality, deliverable state, and security, along with test templates and practical guidance for keeping humans in the loop on permissions and final acceptance.

    1. The Short Version: Long-Horizon Coding Is Usable, but You Cannot Let Go of Security Acceptance

    Large models are iterating quickly, and open-source models are closing in on frontier proprietary ones. This has shifted what developers care about: whether benchmark scores on short tasks still mean anything, whether a model can own an entire project on its own, and under what conditions its output is safe to hand to users.

    The version used in this comparison kept the same base model and the same model size as the previous generation. The upgrade came mainly from scaling the post-training stage: the long-horizon task environment expanded to tens of times its former size, the variety of environment types increased, and training ran longer. The significance is that the model’s coding ability has moved from “completing a single function or interface” to “maintaining a codebase that keeps evolving.”

    The published capability descriptions reflect this direction:

    • Coding: emphasis on long-horizon tasks and near-production-quality software engineering, rather than just accuracy on algorithm puzzles.
    • Safety: the scope extends from generating code to white-box code review and vulnerability reasoning. It approaches frontier proprietary models on basic security tasks, but still lags in full end-to-end exploitation scenarios.
    • Openness: the full weights are planned for release after safety evaluation and model hardening are complete, so developers can validate and improve them in their own environments.

    The verdict after testing:

    The new generation of open-source models has coding ability worth putting to the test in real projects, especially for cross-module, long-cycle development. It raises the ceiling for long-running tasks, can read complex context inside a monorepo, keeps pushing cross-frontend-and-backend changes forward, and brings tasks to a runnable, testable state. But when it comes to permissions, security, and final acceptance, a human still cannot let go.

    Two projects were run end to end in testing. One of them exposed a P1 permission-isolation issue during manual acceptance. Below I first explain the cause, then give a reusable evaluation method.

    2. Why Benchmark Scores Do Not Represent Long-Horizon Coding Ability

    Benchmark scores usually measure abilities that can be judged quickly: algorithm problems, single-file fixes, small-scope refactors—all with highly standardized scoring. Long-horizon coding tasks are more like real delivery, and the difficulty concentrates in a few places.

    1. Requirement understanding and scope definition

    Real requirements mix in business intent, historical constraints, and implicit assumptions—for example, “an imported image may only be referenced in scenes created by its owner.” Without this information, a model tends to implement along the shortest path, producing a solution that is formally correct but semantically off.

    2. Cross-module consistency

    A single change often touches data storage, interface contracts, database schemas, task state machines, background execution, frontend pages, and final rendering. If just one piece fails to line up—say, the values of a task-status enum differ between the database and the background executor—the problem only surfaces at runtime.

    3. Self-verification loop

    Developers actively reason about “which cases haven’t been covered yet” against the requirement boundaries. If a model only generates positive-path cases, the verification chain stops at “the feature works” and lacks adversarial and exception-path testing.

    4. Effective context length

    Being able to fit an entire codebase in the context window is not the same as using it effectively. As the context grows, early constraints can get diluted, and the model tends to imitate local patterns already present in the context. This tendency is especially strong in legacy projects, where security requirements are easily weakened under the banner of “staying consistent with existing code.”

    So evaluating long-horizon coding ability needs at least four dimensions: requirement understanding, cross-module progress, self-verification quality, and deliverable state. Deliverable state means whether the code compiles, starts, and is testable—not whether it survives a demo.

    3. Designing Two Categories of Test Tasks

    Dimension Project A: Incremental feature on an existing codebase Project B: Build a learning game from scratch
    Starting point A sizable legacy codebase with existing architecture and security constraints Only learning materials and a requirements document
    Core ability Understanding an old system, cross-module changes, honoring security constraints Analyzing materials, product design, and end-to-end delivery
    Verification Functional testing, security testing, manual acceptance Build and run, full playthrough walkthrough, functional checks
    Main risk Constraints getting diluted, permission boundaries ignored Misreading materials, messy progress and state management

    Cross-validating with two categories of tasks separates two abilities:

    • Incremental work on an existing project tests “understanding and restraint.” The model must read the existing design, accept new constraints, and avoid casually rewriting existing logic.
    • Building from scratch tests “planning and convergence.” The model must split modules, design data structures, define acceptance criteria, and converge to a usable state within a limited number of rounds.

    The previous generation was also run on a similar task: adding an HTML animation-generation pipeline to an open-source motion-effects project, touching interface contracts, database, task status, API, background execution, the web client, and the renderer across multiple modules. Implementation and browser-based verification fixes were split across several rounds and completed over a span of hours. It could already work continuously across a multi-module monorepo and handle cross-frontend-and-backend features. The new generation’s improvement is a further extension of that baseline.

    4. Project A: Secure Image Import and Motion-Reference on a Legacy Project

    4.1 Requirement Breakdown

    The requirement was to let users upload image assets in the project and later reference those images in HTML motion-effect scenes they create. The full pipeline includes:

    • A frontend upload and selection component;
    • Access control on object storage;
    • Persistence of asset metadata and its association with scene records;
    • Editing and previewing of motion scenes;
    • Generation and rendering of the final HTML output.

    The task description spelled out three security constraints:

    1. Only image types are allowed; the real file content must be validated, not just the extension;
    2. A per-file size limit is set, and storage defaults to denying anonymous access;
    3. When users reference assets in the generated HTML, script injection must be prevented.

    4.2 How the Model Performed

    • Context localization: It located the relevant data models, interface layer, and frontend entry points from the task description, and its changes did not drift into unrelated modules.
    • Cross-module progress: Storage, metadata, interface, pages, and rendering were wired together in one pass, and details like task status and error codes stayed largely consistent with the existing code.
    • Security awareness: It proactively added a file-type allowlist, file-header validation, and a size limit, and at the HTML generation stage it escaped both the injected image URLs and user-controlled text.
    • Self-verification and fixes: It could build, start the service, and use browser automation to validate the upload, preview, and reference flows; on hitting errors it could locate and fix them, ultimately bringing the task to a runnable, testable state.

    Below is a reasonable example of security controls in the import pipeline. It reflects general engineering practice and is not a proprietary implementation of any particular product:

    1. # The following is illustrative code for general engineering practice,
    2. # independent of any specific framework.
    3. import imghdr
    4. from pathlib import Path
    5. from urllib.parse import quote
    6. MAX_SIZE = 5 * 1024 * 1024 # 5 MB
    7. ALLOWED_FORMATS = {"png", "jpeg", "gif", "webp"}
    8. # The allowlist is only for display and routing; the security decision
    9. # is based on the real content.
    10. ALLOWED_MIME = {"image/png", "image/jpeg", "image/gif", "image/webp"}
    11. def validate_image(raw: bytes) -> str:
    12. """Validate the real file type and size; return a normalized MIME type."""
    13. if len(raw) == 0 or len(raw) > MAX_SIZE:
    14. raise ValueError("File is empty or exceeds the size limit")
    15. fmt = imghdr.what(None, raw)
    16. if fmt not in ALLOWED_FORMATS:
    17. raise ValueError(f"Unsupported image format: {fmt}")
    18. return f"image/{fmt}"
    19. def store_private(bucket, key: str, raw: bytes, mime: str):
    20. """Store the image privately, deny anonymous reads by default,
    21. and do not expose a directly accessible URL."""
    22. # Use an unguessable unique identifier for the object key so it is
    23. # not associated with the original filename.
    24. bucket.put_object(key, raw, content_type=mime, acl="private")
    25. def render_src(user: User, asset: Asset) -> str:
    26. """Produce the image URL used in HTML; only reviewed assets may be referenced."""
    27. if asset.owner_id != user.id:
    28. # Do not leak other users' asset information in the output.
    29. raise PermissionError("Not allowed to reference this asset")
    30. if asset.review_status != "approved":
    31. raise PermissionError("Asset has not been reviewed; reference is forbidden")
    32. # The URL points to a backend-controlled endpoint, not a direct storage link.
    33. return f"/api/assets/{asset.id}/content?token={quote(user.session_token)}"

    4.3 What Manual Acceptance Exposed

    After functional and routine security tests passed, a manual review against a permission matrix uncovered a P1 issue:

    • Symptom: After logging in, an ordinary user could read other users’ private uploaded images simply by changing the asset identifier in the request.
    • Cause: The read endpoint only checked login state, not asset ownership, and the storage side did not re-confirm the requester’s identity.
    • Impact: This is a classic broken-access-control case that could leak users’ private assets. P1 means it needs priority fixing and should not ship in this state.

    5. Project B: Building a Playable Agent Learning Game from Scratch

    The second task provided only one Chinese-language learning resource aimed at a mainstream backend language, plus a brief requirements document, and asked for a complete, playable Agent-knowledge learning game built from zero.

    The product difficulty here was not the amount of code but three things:

    • Distilling knowledge points from an unstructured long document and organizing them into levels from easy to hard;
    • Designing a deterministic, reusable level-scoring mechanism;
    • Managing learning progress and completion state so the flow closes properly.

    The model’s execution path was fairly complete:

    1. Material analysis: Split the material by chapters and topics, map out the knowledge structure, and derive a level sequence.
    2. Product delivery: Set up the frontend/backend structure and implement the level list, answer interactions, hints, and feedback.
    3. Progress and scoring: Design progress storage and scoring logic so users can resume after leaving.
    4. Closed-loop verification: After building and starting, walk through each level along the actual player path to confirm the whole game can be completed, and fix issues found along the way.

    Unlike Project A, there was no legacy-code burden here, so performance depended more on whether the planning was sound and whether the levels mapped accurately to the material. The final result was functional and complete. But the retrospective surfaced:

    • Some levels’ scoring conditions were too strict; phrasings close to the expected answer could still be marked wrong;
    • A few knowledge points summarized the source material inaccurately.

    Neither is a blocking problem, but both show that the model’s understanding of the input material still needs human spot-checking.

    6. Why “All Tests Green” Can Still Hide a P1 Issue

    When tests are all green but production breaks, the cause is usually not that the model “isn’t capable enough,” but that there are gaps in verification design and acceptance design. Drawing on both projects, there are four main reasons.

    1. Coverage blind spots in test cases

    When self-testing, models tend to write “normal path” cases first—correct formats upload, logged-in users can read. Broken-access-control cases require switching user perspectives or constructing a precondition like “someone else’s resource.” Such cases usually do not exist in the default test data, so they are hard to think of.

    2. Coupling between permissions and data-access design

    A model more easily frames security as “input validation” and “output escaping,” both of which it does well. But “who is allowed to access which record” belongs to data ownership and access control, which must stay consistent across the data model, interface layer, and storage layer. That consistency is hard to guarantee through local patches and has to be settled at the design level.

    3. Adherence bias

    When a security constraint in the task description conflicts with an existing “logged-in means read access” pattern in the codebase, the model tends to default to the existing pattern in the context and treats the security constraint as a “suggestion” rather than a “hard requirement.”

    4. Missing acceptance criteria

    If the requirement description does not clearly state “who can access what” and “how unauthorized requests should be handled,” the model can only guess, and humans lack a basis for judging.

    7. A Reusable Long-Horizon Coding Evaluation Scheme

    7.1 Evaluation Table

    Dimension What to observe Scoring focus
    Requirement understanding Whether it restates constraints and proactively clarifies ambiguity Number of missed key constraints
    Cross-module progress Contract consistency, error handling First-pass success rate of cross-module integration
    Self-verification quality Whether cases cover exceptions and boundaries Share of broken-access-control and bad-input cases
    Deliverable state Build, start, test, end-to-end walkthrough Number of manual interventions to get from delivery to runnable
    Security and compliance Permissions, input/output, storage access Number and severity of high-risk issues

    7.2 Broken-Access-Control Test Template

    Below is a set of broken-access-control test templates that can be dropped straight into a regression suite. They are independent of any specific framework:

    1. # The following is illustrative code for broken-access-control regression tests,
    2. # independent of any specific testing framework.
    3. def make_user(name="user", uid=1001):
    4. return {"name": name, "id": uid}
    5. def import_image(user, filename="photo.png", data=b"\x89PNG\r\n\x1a\n..."):
    6. # Simulate a real upload endpoint and return the asset identifier.
    7. return {"asset_id": 9001, "owner_id": user["id"]}
    8. def get_asset(token, asset_id):
    9. """Access asset content with the given session; return a response object."""
    10. ...
    11. def create_scene(user, asset_id):
    12. return {"scene_id": 7001, "owner_id": user["id"]}
    13. def list_scenes(token):
    14. ...
    15. def patch_scene(token, scene_id, payload):
    16. """Update a scene; the payload may contain ownership fields."""
    17. ...
    18. # 1. Horizontal access control: another logged-in user cannot read someone else's asset
    19. def test_cross_user_read_asset():
    20. owner = make_user("alice", 1001)
    21. attacker = make_user("bob", 1002)
    22. asset = import_image(owner)
    23. resp = get_asset(attacker["token"], asset["asset_id"])
    24. assert resp.status in (403, 404), "Broken access control: unauthorized read possible"
    25. # 2. Anonymous access: storage denies by default
    26. def test_anonymous_read_asset():
    27. asset = import_image(make_user())
    28. resp = get_asset(None, asset["asset_id"])
    29. assert resp.status in (401, 403), "An anonymous user should not read a private asset"
    30. # 3. Horizontal access control + enumeration: bulk traversal of resource IDs must also be denied
    31. def test_enumerate_assets():
    32. import_image(make_user("alice", 1001))
    33. attacker = make_user("bob", 1002)
    34. for asset_id in range(9000, 9010):
    35. resp = get_asset(attacker["token"], asset_id)
    36. assert resp.status in (403, 404)
    37. # 4. Vertical access control + field tampering: an ordinary user must not
    38. # reassign an asset's ownership to someone else
    39. def test_patch_ownership():
    40. owner = make_user("alice", 1001)
    41. asset = import_image(owner)
    42. scene = create_scene(owner, asset["asset_id"])
    43. resp = patch_scene(owner["token"], scene["scene_id"], {"owner_id": 1002})
    44. assert resp.status in (400, 403), "Ownership fields must not be client-modifiable"
    45. # 5. Scene-level isolation: the list endpoint returns only data the current user may access
    46. def test_scene_list_isolated():
    47. alice = make_user("alice", 1001)
    48. bob = make_user("bob", 1002)
    49. create_scene(alice, import_image(alice)["asset_id"])
    50. resp = list_scenes(bob["token"])
    51. assert all(item["owner_id"] == bob["id"] for item in resp.body)

    The value of this template is that it turns security acceptance into a repeatable test. Run it after each model output, and permission-related regressions get caught before manual acceptance.

    7.3 Requirement Description Template

    For the task description given to a model, I recommend always including four parts:

    1. Business goal: one sentence on the problem to solve;
    2. Functional scope: what to do and what not to do;
    3. Constraint list: security, performance, compatibility—each one verifiable;
    4. Acceptance criteria: expected behavior for the normal path, exception path, and broken-access-control path.

    For example, Project A’s constraints could be written as:

    1. Constraint 1: Assets may only be image formats, judged by real file content; the extension is for display only.
    2. Constraint 2: A single file must not exceed 5 MB; anything larger is rejected with a clear error.
    3. Constraint 3: Assets are private by default; only the owner may read or write them, and read requests must verify identity and ownership.
    4. Constraint 4: All user-controlled content in the generated HTML must be escaped.
    5. Constraint 5: Unauthorized access returns 403 or 404 and must not reveal whether the resource exists.

    Once constraints are written as decidable sentences, the model finds it easier to turn them into test cases, and humans find it easier to verify them item by item.

    8. Practical Recommendations for Users

    Based on this hands-on test, here are a few actionable suggestions:

    • Start small, then scale: First take on a feature with clear boundaries in an existing project, observe how well it understands existing code and how much restraint it shows in its changes, then gradually widen the task scope.
    • Write permissions as acceptance items: In the requirements, clearly state “who can access what,” and fold broken-access-control cases into the regression suite.
    • Distinguish “runnable” from “deliverable”: The model delivers a runnable version; security review, data compliance, and manual acceptance remain steps that cannot be skipped.
    • Build a staged acceptance checklist:
    Stage What to check
    Code review Whether permission checks cover read, write, delete, and export
    Data storage Deny anonymous access by default, least privilege, access logs retained
    Input/output Real-content validation on uploads, output escaping, error messages that do not leak internal structure
    Interface contracts Whether request fields can be tampered with to escalate access, whether state changes verify ownership
    Test coverage Normal, exception, and broken-access-control cases all present
    Before release Sensitive-data masking, dependency and configuration review, rollback plan
    • Understand the boundaries of the model’s security ability: The new generation shows clear gains in white-box code review and vulnerability reasoning, making it suitable for a first pass—for example, asking the model to run a permission review over the interfaces it changed. But for full exploitation and attack-and-defense validation scenarios, it still trails frontier proprietary models, so key modules need to combine human review with dedicated security tooling.

    9. Summary

    This test shows that the coding ability of open-source models has crossed the threshold of “can it complete one long task” and entered the stage of “can it steadily drive multi-module changes.” Both Project A and Project B ran end to end; the former even passed all self-tests before manual review, yet still exposed a P1 permission-isolation issue.

    The problem is not the model itself, but that the evaluation and acceptance methods need to keep up:

    • Test with real projects, real constraints, and real acceptance criteria;
    • Turn security requirements into executable tests;
    • Define the division of labor between humans and machines clearly.

    For developers, the more pragmatic stance is: treat the model as a collaborator with strong execution but little security intuition. It handles breadth and speed; the human handles boundaries and the bottom line.