How to Evaluate the Long-Horizon Coding Capabilities of Code LLMs
Benchmark rankings only capture isolated, well-defined coding problems, so they fall short of measuring how a code LLM performs on real engineering work. This article argues for evaluating models along three production-oriented dimensions—context understanding depth, long-horizon progress, and security-constraint adherence—using two complementary tasks: retrofitting a legacy project under strict constraints, and building a complete product from scratch. It explains why "all tests green" does not equal "ready to ship," illustrated with a P1-level permission-isolation gap, and clarifies what it means when capability gains come from post-training rather than a new base model. It closes with practical recommendations for teams on evaluating with real projects instead of leaderboards, treating long-horizon progress as a core metric, and setting up dedicated human review gates for security.
1. Why Leaderboard Scores Don’t Reflect Real Engineering Ability
When evaluating a code LLM, the most intuitive approach is to look at how it ranks across various benchmarks. But these benchmarks mostly test narrowly scoped, single-point problems: given a problem description, the model generates code that passes a set of test cases. In such tasks the information is complete, the context is bounded, and the acceptance criteria are explicit.
Real engineering is the opposite. Developers face a system burdened with legacy code, implicit conventions, cross-module dependencies, and multiple constraints. Requirements are often vague, and acceptance criteria are scattered across code, documentation, and human experience. Changing one piece of code can ripple through the database, task state, APIs, front-end rendering, and more. So whether a model is actually useful comes down not to whether it can solve a single problem, but to whether it can continuously understand context in long tasks, steadily advance cross-module changes, and bring a project to a “runnable, testable” state. This is the essence of “long-horizon coding capability.”
2. Three Dimensions Worth Focusing On
Setting leaderboards aside, I suggest observing a model through three dimensions that are closer to production:
- Context understanding depth: Faced with a large, structurally complex codebase, can it accurately identify relevant modules and understand implicit conventions, rather than fixating only on the currently open file?
- Long-horizon progress: In tasks that require multiple rounds and cross-module collaboration, can it keep the goal from drifting, continuously push forward until the feature is complete, rather than stalling midway or repeatedly reversing itself?
- Security-constraint adherence: When a task involves constraints such as permission isolation, input validation, and access to sensitive data, can it proactively recognize and correctly implement them, rather than bypassing these boundaries just to “get it running”?
Of these three dimensions, the first two determine whether the model can “finish the job,” while the third determines whether what it finishes is “usable.” Experience shows a model can perform excellently on the first two dimensions yet leave serious risks on the third.
3. Evaluation Scheme Design: Two Complementary Tasks
A single task easily leads to one-sided conclusions. To evaluate comprehensively, I designed two tasks of very different natures, each stress-testing a different facet of the model.
3.1 Retrofitting a Legacy Project: Understanding Old Systems and Security Constraints
The first task is to add a new capability to a mature project that has already accumulated a large amount of legacy code: support the secure import of user image assets and their correct reference in animation/motion scenarios. This task is characterized by:
- The system contains many existing conventions and styles, which the model must read, understand, and stay consistent with;
- It involves file uploads, which inherently carry security constraints: type validation, size limits, path traversal protection, and storage isolation;
- The change spans multiple layers: data models, task state, APIs, background tasks, front-end pages, and rendering logic.
This kind of task tests the model’s robustness when “dancing in chains”—whether it can complete the feature without breaking the existing structure or introducing security vulnerabilities.
3.2 Building from Scratch: From Material Analysis to Product Delivery
The second task is the exact opposite: given only a public technical learning resource and a product requirements document, the model must build, from zero, a knowledge-learning mini-game that can actually be played through to completion.
With no existing code to read, this task tests the model’s ability to complete the full chain of “material → structured knowledge → product logic → runnable code.” It must plan the data structures, design the level flow, organize state management, and ultimately deliver a complete closed loop that a user can play from start to finish. This task tests the completeness of “zero-to-one” productization.
One task tests “retrofitting under constraints,” and the other tests “building on empty ground”; only combined can they more fully reflect a model’s capabilities.
4. Key Observation Points for Long-Horizon Coding
When actually running these two tasks, I focused on the following signals—they say more about a model’s real level than “whether it can run in the end”:
- Goal retention: Across multiple rounds and modules, does it stay centered on the original goal, or gradually drift away?
- Change coherence: Is a single cross-layer change (for example, changing the database, API, and front-end at once) self-consistent, or are there inconsistencies between front-end and back-end?
- Self-verification awareness: Does it proactively drive the task to a “runnable, testable” state, and is it willing to discover problems through actual runs and iterate on fixes?
- Proactive constraint recognition: Facing security-related requirements, does it proactively consider boundaries, or only implement the “happy path”?
In the tasks I set up, the model could continuously advance cross-module changes on both projects and bring them to a running state, which shows it already has the foundational ability to take on long-horizon coding tasks in real projects—especially development scenarios with complex structures and long dependency chains.
5. Why “All Tests Green” Doesn’t Mean “Ready to Deliver”
This is the point that most needs emphasizing in the entire evaluation. Although the first legacy project eventually ran, manual acceptance revealed a P1-level permission-isolation problem: the model implemented the feature but left a gap in permission checks.
The following sample code illustrates this kind of problem:
# The "happy path" implementation a model tends to writedef import_asset(user_id, file):path = save_to_disk(file) # only cares that the save succeedsrecord = create_asset(user_id, path)return record# But production requires these additional constraintsdef import_asset(user_id, file):assert_valid_type(file) # type allowlist validationassert_within_size_limit(file) # size limitsafe_path = sanitize_path(file.name) # prevent path traversalassert_user_can_write(user_id, safe_path) # permission isolationpath = save_to_disk(safe_path)return create_asset(user_id, path)
The root of the problem is that automated tests usually only verify “whether the functionality matches expectations” and do not proactively construct attack scenarios like “unauthorized access” or “path traversal.” So “all tests green” can only prove that the functional logic is correct; it cannot prove that the system is complete on its security boundaries. Problems like permission isolation, data overreach, and input pollution often hide in the blind spots that test cases fail to cover, and must be specifically reviewed by people familiar with the business and security.
So a pragmatic conclusion is: the model is worth running on real projects, especially long-horizon development tasks; but for the parts involving permissions, security, and final acceptance, people cannot let go. The model is responsible for reducing the workload, and people are responsible for holding the line on risk.
6. What Post-Training Extension and an Unchanged Base Model Mean
An easily overlooked detail is that a new-generation model, compared with the previous generation, often does not replace the underlying base model but instead improves capability through the extension of post-training—scaling up the size of training environments for long-horizon tasks, increasing the diversity of environment types, and lengthening training time. Official sources usually summarize these improvements into a few directions: coding capability places more emphasis on long-horizon and production-grade software engineering; security capability extends from “generating code” to “code review and vulnerability reasoning”; and full weights are released after security evaluation is complete, making it convenient for developers to deploy and verify on their own.
The lesson of this technical approach for users is: capability improvements don’t necessarily come from a new base model—engineering-oriented training can also significantly improve long-horizon performance. When evaluating, don’t just fixate on “how big the base is”; look more at its actual progress on long tasks and in real engineering, as well as its performance boundaries in security review.
7. Practical Recommendations for Teams
Based on this evaluation, here are a few recommendations for teams preparing to adopt code LLMs:
- Evaluate with real projects rather than leaderboards: prepare a legacy-retrofit task and a build-from-scratch task—the former tests constraint adherence, the latter tests productization completeness.
- Treat long-horizon progress as a core metric: observe whether it can keep the goal across modules and multiple rounds and advance to a runnable state.
- Set up a dedicated human acceptance gate for security: treat permission isolation, input validation, and data access as independent review items, and don’t rely on the natural coverage of automated tests.
- Define a clear division of labor between humans and machines: let the model take on large-scale, repetitive coding work, and let people focus on security boundaries, architecture decisions, and final delivery oversight.
Code LLMs are rapidly approaching the level where they can take on real engineering tasks, and they are already quite practical in long-horizon coding scenarios. But “getting stronger” and “safe to leave alone” are two different things. Only by designing the evaluation method well and setting a firm human acceptance gate can we both enjoy the efficiency gains and hold the bottom line on engineering quality and security.
