Long-Horizon Coding in Practice: How to Evaluate the Real Delivery Capability of AI Coding Models

    Benchmark scores are a poor proxy for whether an AI coding model can actually ship a production project. This article argues that long-horizon coding—sustaining coherent progress across many steps and multiple modules—is the real dividing line, and that capability gains increasingly come from post-training scaling rather than swapping the base model. It presents a two-task evaluation using deliberately contrasting scenarios (an incremental change in a legacy codebase and building a complete product from scratch), reports the results, and offers practical guidance on where to apply these models while keeping security sign-off in human hands.

    1. Why Benchmark Scores Don’t Tell the Whole Story

    When people evaluate an AI coding model, their first instinct is usually to look at its ranking across various benchmarks. Those scores do have reference value, but they measure “how a model performs on a fixed set of tasks”—not “whether a model can actually take on the delivery of a production project.” The gap between these two things is often much larger than expected.

    In real projects, a model doesn’t face a clean, self-consistent question bank. It faces a mix of legacy code, tightly coupled modules, explicit business constraints, and security boundaries that can trigger production incidents if handled poorly. These dimensions are hard to cover fully with static evaluation. A model might rank high on a general benchmark yet expose serious permission flaws when modernizing a legacy system with security constraints; it might also gradually lose its way during cross-module integration and end up leaving behind a half-finished product that runs but can’t be maintained.

    For this reason, I prefer to test a coding model with two very different kinds of real tasks. One is incremental modification within an existing codebase with substantial history—this tests its ability to understand a legacy system and respect existing constraints. The other is building a complete product from scratch—this tests the full chain from understanding requirements to delivering something runnable. The first exposes weaknesses in “understanding existing code”; the second exposes weaknesses in “going from zero to one.” Only by combining them can you assemble a picture that comes close to a real production environment.

    2. Long-Horizon Coding Is the Real Dividing Line

    If single-turn code generation is a coding model’s basic skill, then “long-horizon coding” is what truly sets models apart. Long-horizon here means the model has to sustain a large number of steps within a single task: reading multiple files, understanding dependencies between modules, modifying several places across the front end and back end, running tests, fixing issues based on feedback, and continuing until the task reaches a “runnable and testable” state.

    These tasks demand all-around capability. Beyond writing correct code at individual points, the model has to maintain goal consistency across a long chain—it can’t forget the original requirement halfway through, can’t break existing contracts when making cross-module changes, and can’t fall into a “the more you fix, the messier it gets” loop when it hits errors. Compared with the one-shot question answering on a leaderboard, long-horizon tasks are closer to an engineer’s real working rhythm and are harder to fool with “recitation-style” capability.

    One notable pattern is this: many teams, when they first start using a coding model, only care about whether it can quickly generate a snippet of code. But once they get into production, the value actually concentrates in long-horizon scenarios—batch refactoring, cross-file feature development, legacy system migration, and so on. This is exactly why, when evaluating a coding model, you should deliberately design tasks that “take several hours and span multiple modules,” rather than only looking at its answer to a single function.

    3. What Drives the Capability Jump: Post-Training Scaling, Not a New Base

    It’s worth noting that a significant upgrade to a coding model doesn’t necessarily come from swapping out the base model. A common approach in the industry is to keep the base model size unchanged and instead scale up post-training to raise capability. Concretely, this means expanding the task environment the model trains in by tens of times, introducing a richer variety of environment types, and extending the training duration.

    The gains from this “post-training scaling” can be grouped roughly into three directions:

    • Coding and software engineering capability: Making the model better at long-horizon tasks and production-grade engineering practices, evolving from “can write code” to “can complete end-to-end software engineering tasks in a terminal environment.”
    • Code review and security reasoning: Extending the capability boundary from “generating code” to “white-box code review” and “vulnerability reasoning,” gradually approaching frontier closed-source levels on basic security tasks, though a clear gap remains in complete exploit-chain scenarios.
    • Openness and verifiability: Opening up full weights after passing a security assessment, making it easier for developers to self-deploy, verify, and make secondary improvements.

    This line of thinking has direct implications for team selection: don’t fixate only on “how big the base is, how many parameters it has.” Pay more attention to how much post-training the model has received in real engineering environments. A model with a moderate base size but an extremely rich training environment is often more capable at long-horizon tasks than a product with larger parameters but no engineering-oriented training.

    4. Evaluation Design: Stress-Testing with Two Sharply Contrasting Tasks

    For an objective assessment, I prepared two kinds of tasks for the model under test, deliberately designed to contrast in difficulty distribution and technical focus.

    Task 1: Incremental modification in a legacy project. In a project that already has a substantial amount of historical code, add the capability for “secure user image asset import and reference in HTML animation scenarios.” The difficulty of this task lies not in writing new code, but in this: the model must understand the existing architecture and conventions and integrate the new feature while respecting the original security constraints. Any damage to the old logic might only surface during manual acceptance.

    Task 2: Building a complete product from scratch. Given only a piece of learning material and a requirements document (PRD), have the model implement from scratch a small knowledge-learning application that can genuinely be “played to completion.” This task tests the full chain: from material analysis and requirement breakdown, to interface design, to core gameplay implementation and final playability. It’s more like assessing “a junior engineer independently delivering a small project.”

    One tests “understanding a legacy system and handling security constraints”; the other tests “the complete journey from materials to product.” This combination can expose a model’s true level across both “maintaining existing code” and “creating from zero” at the same time.

    5. Results: Clear Progress, but Security Acceptance Can’t Be Left to Chance

    The from-scratch task ran through end to end. The model was able to distill a usable knowledge structure from scattered materials and turn it into a logically coherent product that can be completed in full. This shows it has reached a usable level of completeness in “going from zero to one”—for scenarios like building small tools independently, prototyping, and teaching demos, it can significantly speed things up.

    The legacy modification task also ran through, but exposed a critical problem. The program ran and the tests passed, yet during one round of manual acceptance, a P1-level permission isolation defect surfaced. This is a very typical lesson: the model-generated code passed on “functional correctness” but slipped on “security correctness.” Permission isolation is a classic non-functional requirement—it doesn’t get satisfied automatically just because all unit tests are green, and it often only shows itself under a real attacker’s perspective and an acceptance perspective.

    Here’s a rule every team should take seriously: passing tests ≠ ready to ship. Especially for security properties like permissions, authentication, and data isolation, you must bring in independent human review and dedicated acceptance testing. Treating the model as an “efficient junior executor” is reasonable, but treating it as a “security owner who can sign off independently” is still premature.

    6. Practical Recommendations: Apply the Model Where It Counts, Keep Security in Human Hands

    Based on the above tests, here are my practical recommendations:

    1. Prioritize long-horizon coding scenarios. Cross-module refactoring, extending legacy system features, and rapid prototyping are where these models deliver the most value. Letting them push tasks forward over long spans frees engineers from tedious, repetitive work.

    2. Evaluate with real projects, not leaderboards. Rather than agonizing over a ranking on some benchmark, stress-test with your own team’s real, constraint-laden projects—especially long tasks that span multiple modules and require multiple rounds of fixes.

    3. Keep permission and security acceptance in human hands. For tasks involving authentication, data isolation, and sensitive operations, be sure to establish an independent code review and a dedicated security test suite. Don’t rely on “the model says it passed.”

    4. Build a “generate–verify–fix” loop. Have the model work in a runnable, testable environment, combined with browser-based hands-on testing and automated checks. This effectively surfaces problems that static checks can’t cover.

    5. Focus on post-training, not just parameter count. When selecting a model, treat “whether the engineering-oriented training environment is rich” as an important dimension—it often matters more for real long-horizon performance than base size alone.

    Conclusion

    Evaluating an AI coding model is, in essence, evaluating whether it can reliably create value within your real workflow. This generation of models has already taken a solid step forward in long-horizon coding: it can understand complex context, sustain cross-module changes, and bring tasks to a runnable state. But when it comes to permission isolation, security protection, and final acceptance, human engineers remain the indispensable “last gate.” Using it rationally—letting the model handle efficiency and letting humans handle security—is the most pragmatic way to put it into practice at this stage.