Long-Horizon Coding: Evaluating Code Models with Real-World Projects

    For code-focused large language models, metrics like "can it write a function" or "single-shot code pass rate" say little about real engineering value. This article looks at long-horizon coding—where a model must read an existing system, work continuously for hours, and complete changes across multiple modules until the task is runnable and testable. Drawing on two complementary real-world projects (an incremental feature added to a legacy codebase under security constraints, and a playable learning game built from scratch from an agent knowledge document and a PRD), it analyzes what the model does well and where it falls short. It covers the role of large-scale long-horizon post-training, a P1 permission-isolation defect surfaced during manual acceptance, the full pipeline from source material to a finished product, and practical recommendations on building your own scene sets, writing explicit constraints, using CI gates, strengthening acceptance checklists, and dividing work between humans and the model.

    Introduction: Why Long-Horizon Coding Needs a New Way to Evaluate

    For code-focused large language models, metrics such as “can it write a function” or “single-shot code pass rate” have long ceased to reflect their real value in engineering. What R&D teams actually care about is this: given a requirement, can the model understand an existing system, work continuously for tens of minutes or even hours, make changes across multiple modules, and finally push the task to a runnable, testable state?

    This is the core proposition of long-horizon coding. Evaluating this capability cannot rely only on leaderboard rankings; it must be tested with tasks that closely resemble production environments. This article is based on two complementary real-world project tasks. It observes how a new generation of code models performs on long-horizon work, analyzes the reasons behind both the strong results and the shortcomings, and offers actionable engineering recommendations.

    1. Two Complementary Task Designs

    This evaluation deliberately avoided “algorithm-puzzle” style point tasks and instead set up two distinct goals:

    • Task 1 (incremental change): In a project with a large body of legacy code and security constraints, add the ability to “securely import user image assets and reference those assets in HTML animation scenes.”
    • Task 2 (from-scratch build): Given only an agent knowledge learning document and a product requirements document (PRD), build a learning game with a complete knowledge structure that can genuinely be played from start to finish.

    The capability dimensions the two tasks examine barely overlap:

    Evaluation dimension Task 1: Incremental change Task 2: From-scratch build
    Core difficulty Understanding legacy code and existing conventions Distilling material, decomposing requirements, delivering a product
    Main risk Missing permission and security constraints Ambiguous requirements, incomplete content, unable to complete the game
    Key output Cross-module, regression-testable features A well-structured app that can be fully completed
    Typical duration Hours of continuous work Hours of continuous work

    In terms of task duration, long-horizon tasks like these typically run for hours. This is exactly what exposes problems in the second half of a run: attention decay, loss of context, and “self-verification that becomes a formality.” Next, we look at where the capability jump comes from, then review each task in turn.

    2. Where the Capability Jump Comes From: Same Base Model, Amplified by Long-Horizon Post-Training

    A notable characteristic of this new generation of code models is that they did not swap out the base model. Instead, the improvement is concentrated in the post-training stage.

    Post-training refers to the process after a base model finishes pretraining, in which the model is made stronger in specific capability directions through techniques such as supervised fine-tuning, preference alignment, and reinforcement learning. Put simply, the base model determines the “knowledge foundation,” while post-training determines whether the model “can actually do the work.”

    The core of this upgrade can be summarized as the scaling of long-horizon post-training:

    1. Larger task scale: The number of end-to-end task environments used for training reached an order of magnitude of tens of times that of the previous generation. What the model had to complete during training shifted from “a few rounds of dialogue” to “complete workflows of dozens or even hundreds of steps.”
    2. Richer environment types: Training environments cover more tech stacks, directory structures, and engineering conventions, reducing the model’s tendency to feel out of place in unfamiliar codebases.
    3. Longer training time: The model is given a fuller cycle for trial and error and feedback, giving the “plan—execute—verify—correct” loop a real chance to form during training.

    The official summary groups the changes into three directions:

    1. Coding capability: Positioned as a leading open-source model for code, it ranks among the top open-source models across several software engineering and agent task evaluations, with greater emphasis on long-horizon execution and production-grade software engineering.
    2. Security capability: The scope extends from “generating code” to “reading code and reasoning about vulnerabilities.” On white-box code review and foundational security tasks it approaches leading closed-source models, though it still lags in complete exploit-chain scenarios.
    3. Open weights: The weights will be fully released after security evaluation and model hardening, making it easier for developers to self-deploy, independently verify, and make secondary improvements.

    One caveat: benchmark leaderboards (such as task sets for terminal operation, benchmarks for real software engineering tasks, and benchmarks for agent tool use) should be treated only as references. Benchmark problems differ greatly from real projects in code scale, number of constraints, and acceptance criteria, so your own scenario set should always be the ultimate standard.

    3. Task 1: Incremental Change in a Legacy Project

    3.1 What the Model Got Right

    In this older project with security constraints, several aspects of the model’s performance are worth recognizing:

    1. Understanding existing conventions: It first mapped out the project’s directory structure, data models, and interface conventions before touching the code, rather than starting a separate approach based on “general best practices.”
    2. Sustained cross-module progress: Changes flowed through interface contracts, database schema, task state transitions, server-side APIs, background jobs, and frontend pages, without stopping at an isolated code snippet.
    3. Proactive self-verification: After implementing the feature, it ran the project’s existing test commands on its own; if a run failed, it traced back through the errors and made fixes until the task passed.
    4. Usable delivery state: The final result was not code that merely “looked right,” but functionality that could start up, be operated, and be verified.

    These four points are exactly what distinguishes long-horizon coding from single-turn code generation. Of course, the time spent on verification and repair is often comparable to—or even greater than—the implementation itself. So when planning schedules, “verification and repair” should be treated as a necessary part of the task.

    3.2 P1 Issues Exposed by Manual Acceptance

    However, “all tests green” does not equal “ready to ship.” During manual acceptance, this change exposed a P1-level permission-isolation defect:

    A logged-in user could craft a request to read, or even replace, another user’s material files.

    Under the team’s standard defect classification, P1 typically refers to a high-priority issue that affects data security or core functionality and must be fixed as soon as possible.

    The problem was in the resource access layer: the code queried data based only on the passed-in resource identifier, without checking “whether the current user is authorized to access that resource.” This pattern often does not raise errors in a demo environment, but in a multi-user setting it becomes unauthorized access. The fix replaces an unconditional query with one that carries ownership conditions:

    1. // Problem: querying by ID only lets any logged-in user access the resource
    2. public Material getMaterial(Long id) {
    3. return materialMapper.selectById(id);
    4. }
    5. // Fix: include ownership in the query; if nothing is found, treat it as unauthorized
    6. public Material getMaterial(Long id, Long currentUserId) {
    7. Material material = materialMapper.selectByIdAndOwnerId(id, currentUserId);
    8. if (material == null) {
    9. throw new ResourceNotFoundException("Material not found or access denied");
    10. }
    11. return material;
    12. }
    13. // Write operations such as update and delete need the same ownership check to prevent tampering
    14. public void deleteMaterial(Long id, Long currentUserId) {
    15. int affected = materialMapper.deleteByIdAndOwnerId(id, currentUserId);
    16. if (affected == 0) {
    17. throw new ResourceNotFoundException("Material not found or access denied");
    18. }
    19. }

    Unauthorized access is hard to catch automatically because it is usually not “invalid input” but “valid input + wrong target.” Functional tests only verify “can my own data be accessed”; they do not proactively verify “other people’s data cannot be accessed.”

    3.3 Root-Cause Analysis

    1. Implicit requirement expression: The requirement mentioned “secure import” but did not write the permission model as verifiable clauses, such as “who can create, who can view, who can delete.”
    2. The model’s tendency to round to the nearest goal: To get the feature working quickly, the model is more inclined to implement the explicitly requested functional path and defer implicit security constraints to “later.”
    3. Blind spots in verification scope: The automatically generated tests mostly revolved around the happy path and lacked negative cases like “user A accessing user B’s resource.”
    4. Inertia of legacy code: If the old system itself had no unified permission-check layer, the model easily follows the old pattern and copies the defect into the new feature.

    3.4 Recommendations

    Turn security constraints from “adjectives” into “verifiable clauses.” In the requirements document or project specification file, explicitly write out a permission matrix, for example:

    Action Material owner Other project members Unauthenticated users
    Upload material Allowed Allowed (with project authorization) Denied
    View material Allowed Allowed (with project authorization) Denied
    Delete material Allowed Denied (requires project admin authorization) Denied

    At the same time, establish a unified security baseline for file-import features. The following pseudocode shows the validation order (regardless of the actual tech stack used, the order is the same):

    1. Validation flow:
    2. 1. Authentication: confirm the request is logged in and has a valid session
    3. 2. Quota check: verify storage quotas at the user and project level
    4. 3. Size check: limit single-file size and total size
    5. 4. Type check: inspect the real type in the file header; do not trust the extension or declared type
    6. 5. Content processing: decode and re-encode (e.g., normalize the format), strip metadata and embedded content
    7. 6. Safe storage: generate a random filename and isolate storage paths by user or project
    8. 7. Allowlist rendering: if rendering on a page, allow only controlled tags and escape when necessary
    9. 8. Ownership check: always include resource ownership conditions when querying, updating, or deleting
    10. 9. Audit logging: record the actor, target, source address, and result for traceability

    On this basis, turn “unauthorized access” into negative tests that run automatically, for example:

    1. def test_user_cannot_read_others_material(client, user_a, user_b):
    2. material = create_material(owner=user_a)
    3. resp = client.get(f"/materials/{material.id}", headers=auth(user_b))
    4. assert resp.status_code == 404 # Do not reveal whether the resource exists
    5. def test_anonymous_cannot_upload(client):
    6. resp = client.post("/materials", files=fake_image())
    7. assert resp.status_code in (401, 403)
    8. def test_oversized_file_rejected(client, user_a):
    9. resp = client.post("/materials", files=big_file(limit_mb + 1), headers=auth(user_a))
    10. assert resp.status_code == 413
    11. def test_spoofed_type_rejected(client, user_a):
    12. # Extension disguised as an image, but the content is a script
    13. resp = client.post("/materials", files=script_as_image(), headers=auth(user_a))
    14. assert resp.status_code == 415

    Cases like these can go directly into a CI pipeline and act as a mandatory gate before merges, alongside unit tests, static code scanning, and dependency vulnerability checks.

    4. Task 2: Building a Playable Learning Product from Scratch

    4.1 The Full Pipeline from Material to Product

    The second task provided only two kinds of input: an agent knowledge learning document and a product requirements document. The model had to complete an end-to-end pipeline:

    1. Material distillation: Organize scattered knowledge material into a hierarchical, ordered knowledge system.
    2. Requirement decomposition: Turn vague descriptions in the PRD into concrete feature points, pages, and interaction flows.
    3. Architecture design: Decide module boundaries, the data model, and how level state is managed.
    4. Content production: Organize knowledge into questions, hints, explanations, and feedback.
    5. Testing and verification: Ensure the full flow works end to end, especially the main path “from the first level to completion.”

    4.2 Key Takeaways

    1. Content as configuration: Organize questions, options, explanations, and hints into structured configuration (such as JSON or YAML) rather than conditionals scattered through the code. That way, adjusting difficulty or adding and removing levels later does not require changing core logic.
    2. A level state machine: Manage progress with explicit states (not started, in progress, passed, failed) to avoid conflicting flags and to make save and restore straightforward.
    3. Testable completion path: Design two kinds of end-to-end cases—“shortest completion path” and “common failure paths.” The former guarantees the game can always be completed; the latter guarantees that error feedback and retry mechanisms work.
    4. Built-in observability: Record each answer, the score, and the time spent, making it easier to analyze teaching effectiveness later.

    The configuration example below reflects the “separate content from logic” approach:

    1. {
    2. "levels": [
    3. {
    4. "id": "L1",
    5. "title": "Understanding the core components of an Agent",
    6. "knowledge_points": ["Perception", "Planning", "Action", "Memory"],
    7. "questions": [
    8. {
    9. "id": "Q1",
    10. "type": "single_choice",
    11. "stem": "The ability of an Agent to devise execution steps is usually called?",
    12. "options": ["Perception", "Planning", "Memory", "Output"],
    13. "answer": "Planning",
    14. "explanation": "Planning lets an Agent break a goal into executable steps.",
    15. "hint": "First think about which ability 'what to do first and what to do next' belongs to."
    16. }
    17. ],
    18. "pass_rule": { "min_score": 60, "allow_retry": true }
    19. }
    20. ],
    21. "progress": { "max_attempts": 3, "hint_cost": 1, "score_weight": 0.6 }
    22. }

    4.3 Usage Recommendations

    When building from scratch, the model tends to drift between “looks finished” and “can actually be completed.” It is recommended that the team walk through the main flow end to end themselves before acceptance, and check the following:

    1. Completeness of knowledge coverage: Does every core concept in the material have corresponding questions and explanations?
    2. Reasonableness of the difficulty curve: Are there sudden difficulty jumps, or levels that can be “skipped without thinking”?
    3. Effectiveness of failure feedback: After a wrong answer, is there an explanation and a path to keep trying?
    4. Data portability: Are learning records tied to a device or account, and can they sync across platforms?

    5. Conclusions and Boundaries of Use

    Combining the performance across both tasks, we can draw the following conclusions:

    1. Long-horizon capability: Commendable. The model can work continuously within an existing codebase, complete cross-module changes, and push a task to a runnable, testable state. This is the most obvious change of the new generation of code models relative to the previous one.
    2. Production readiness: Close to usable, but not yet at the point where “all tests green means you can ship with confidence.” Implicit requirements around permissions, security, and data boundaries remain the most easily compromised area.
    3. Product delivery capability: Given clear requirements and explicit acceptance criteria, the model can build a reasonably structured, fully runnable application from scratch.

    So rather than asking “can it replace engineers,” it is more useful to ask “which parts should be handed to it.” A practical division of labor between humans and the model looks like this:

    Stage Suitable for the model Still needs human oversight
    Requirement clarification Organizing requirements, raising open questions Confirming business boundaries and priorities
    Solution design Providing options and trade-off analysis Making architecture and security review decisions
    Coding Cross-module changes, boilerplate, test additions Line-by-line review of critical-path and security-related code
    Verification Running tests, reading logs, fixing regressions Designing negative cases such as unauthorized access and abnormal input
    Release Generating deployment scripts and configs Final acceptance of permissions, data, and compliance

    6. Practical Recommendations for Teams

    1. Build your own scene set: Turn your team’s real, high-frequency long-horizon tasks into an evaluation set. It reflects actual gains better than any public leaderboard.
    2. Write constraints clearly: In the project specification file, explicitly define the permission matrix, file upload rules, and error-handling conventions so the model has something concrete to follow.
    3. Guard the baseline with gates: Bring unit tests, static scanning, dependency vulnerability checks, and unauthorized-access cases into CI as hard conditions for merging.
    4. Strengthen the acceptance stage: Add three fixed check items to the manual acceptance checklist: “multi-user isolation,” “abnormal input,” and “resource exhaustion.”
    5. Continuously observe results: Track metrics such as task duration, rework count, and defect density to assess whether a model upgrade delivers real gains, and to inform later prompt and process optimization.
    6. Consider deployment form: For data-sensitive scenarios, keep an eye on self-hosted deployment options for open-weight models. Platforms typically offer common capabilities such as model serving, container platforms, logging, and monitoring and alerting, making it easy for enterprises to combine them as needed.

    Overall, the new generation of code models has advanced “can write code” to “can push a task forward,” but before reaching “can ship with confidence,” security constraints and final acceptance still need humans to backstop them. Placing the model in the long-horizon execution work it excels at, while keeping key decisions and risk gating on the human side, is the more prudent approach at this stage.