How to Evaluate AI Coding Tools: Why Long-Horizon Coding Matters More Than Benchmark Scores

    A practical look at why public benchmark scores overstate the real capability of AI coding tools, and what a better evaluation looks like. It explains the difference between single-point problem solving and sustained engineering work, how capability gains often come from scaled post-training rather than a bigger base model, and how to design two contrasting evaluation tasks — a security-constrained feature change in a legacy system and a from-scratch build from raw material plus a PRD. It also covers a typical failure mode that automated tests missed (broken permission isolation), the current boundary between code generation and code review, and concrete recommendations for dividing work between humans and models.

    Bottom Line First: Evaluate the Tasks, Not the Leaderboard

    If you judge an AI coding tool only by its score on a public benchmark, you will probably come away too optimistic. Benchmark problems are mostly self-contained: clearly bounded, limited in context, and easy to score with automated scripts. Real engineering delivery tests a different set of abilities — whether the tool can read a large body of historical context, keep changes logically consistent across multiple modules, and carry code that “looks like it runs” all the way to a state that is runnable, testable, and deliverable.

    My criterion is fairly simple: can the tool hold up on long-horizon coding tasks? “Long-horizon” here means the whole chain — understanding the requirement, locating the relevant code, implementing across modules, and then running and verifying the result. The longer the chain, the higher the demands on the model’s ability to retain context, decompose a task, and correct itself. If any link in that chain gives out, the final artifact cannot pass acceptance.

    Why Benchmark Scores Don’t Tell the Whole Story

    A benchmark score reflects single-point problem solving. Engineering delivery requires sustained execution. The gap shows up in three main areas:

    • Context scale differs. A benchmark prompt usually comes with a one-off, static context. A real repository has hundreds or thousands of files, tangled dependencies, and legacy conventions. The model has to find the code that actually matters in an environment with a lot of noise.
    • Task continuity differs. A benchmark is a single question-and-answer exchange. Real development is continuous work over tens of minutes or several hours. Across many rounds of edits, the model has to keep the goal from drifting and not forget the original constraints halfway through.
    • Correctness criteria differ. A benchmark is scored by whether tests pass. Real acceptance also covers things tests don’t reach: permission isolation, security boundaries, maintainability.

    Because of this gap, “all tests green” is not the same as “safe to ship”. A real acceptance review often surfaces serious security or design defects even after every automated test has passed.

    From Swapping the Base Model to Scaling Post-Training: The Real Path of Capability Gains

    To judge how much a coding tool has actually improved, don’t read the version number — look at what changed. A common assumption is that a version bump means a larger base model underneath. That is often not the case. In many releases the base model stays the same, and the real jump in capability comes from large-scale investment in the post-training stage.

    This kind of evolution typically involves:

    • Scaling up task environments. Training the model on far larger and more varied coding environments, covering more real engineering scenarios instead of concentrating on a handful of benchmark tasks.
    • Longer training runs. More thorough post-training that improves stability and consistency on long-horizon tasks.
    • Emphasis on long-horizon, production-grade capability. Shifting the training objective from “can produce a correct snippet” to “can complete production-grade software engineering”, with an emphasis on sustained progress across files and layers.

    In this kind of evolution, capability usually expands in a few directions: first, coding ability itself, with an emphasis on long-horizon tasks and production-grade engineering quality; second, a move from generating code to reviewing code, including white-box code auditing and vulnerability reasoning; third, gradually approaching leading levels on basic security tasks, though still lagging in complex scenarios such as full attack-chain reasoning. This distinction matters: it determines where these tools should sit in your workflow, and which steps must keep a human in the loop.

    Evaluation Methodology: Designing Two Contrasting Tasks

    A single task can’t represent a tool’s capability well. When I evaluate, I prefer to design two engineering tasks with very different natures that both stay close to real business work, so that different sides of the model get exposed.

    Task 1: A Security-Constrained Feature Change in a Legacy System

    The first task adds a capability to a project with a large amount of existing code — something like “secure user asset import with front-end animation scene references”. The difficulty is not in writing new code:

    • Understanding the existing architecture conventions and implicit rules, without introducing an implementation that clashes with the project’s style.
    • Supporting user-uploaded assets while handling security constraints correctly — file type validation, path traversal protection, content injection risks.
    • Making sure the change fits cleanly with existing modules for the database, APIs, and task state.

    What this task really tests is the combination of understanding a legacy system and holding the security boundary — exactly where long-horizon coding tends to lose points.

    Task 2: From Source Material to a Working Product

    The second task is the opposite: you provide only a piece of learning material and a product requirements document (PRD), and the model builds an interactive learning application from scratch that is logically complete and can genuinely be played through. This task doesn’t depend on any existing code and tests:

    • Extracting knowledge points and rules from unstructured material.
    • Breaking product requirements down into implementable modules and data structures.
    • Independently completing the full loop from design to implementation to a runnable, verified result.

    One task changes an existing system from the inside; the other builds something new from the outside. Together they cover the key dimensions of long-horizon coding: legacy system understanding, security-constrained handling, requirement decomposition, and shipping from zero.

    What a Real Project Exposed: Permission Isolation and Security Gaps

    Interestingly, both tasks ran through. Both produced working software, and both passed the tests I had prepared. But during manual acceptance of the first project, a fairly serious permission isolation problem came to light: users at different permission levels had no strict boundary when accessing asset resources. The automated tests didn’t catch it, because those tests were built around “pass a set of normal cases” rather than “exhaustively try privilege-escalation paths”.

    A simplified sketch shows how this kind of defect arises:

    1. // Flawed access control: checks "is logged in", not "is allowed to access this resource"
    2. function getMaterial(id, user) {
    3. if (!user.isLoggedIn) {
    4. throw new Error("Unauthorized");
    5. }
    6. // Defect: any logged-in user can read someone else's asset by id
    7. return db.materials.find(id);
    8. }
    9. // Correct approach: also check ownership or role permission
    10. function getMaterial(id, user) {
    11. if (!user.isLoggedIn) {
    12. throw new Error("Unauthorized");
    13. }
    14. const material = db.materials.find(id);
    15. if (!canAccess(user, material)) { // Verify ownership / role
    16. throw new Error("Access denied");
    17. }
    18. return material;
    19. }

    This is a useful reminder: permissions, security, and final acceptance are blind spots for automated testing, and people have to guard them. A model can make the feature work and get the tests green, but there is still a gap between “green” and “secure” that needs human judgment.

    From Code Generation to Security Review: Know the Boundary

    One clear advance in current coding tools is that the capability boundary has extended from writing code to reviewing code — performing white-box code review and vulnerability reasoning, and spotting common logic defects and security risks. On basic security tasks this is already fairly usable. But in scenarios that require full attack-chain reasoning or judgment against complex business context, there are still clear shortfalls.

    So the sensible way to use it is not as a replacement for security auditing, but as the first filter in front of it: let the tool scan for obvious issues and produce a list of suspicious points, then have security engineers do targeted follow-up review. That improves efficiency without giving up the critical judgment calls.

    Engineering Recommendations: Drawing a Clear Human-AI Boundary

    Based on the above practice, here are a few actionable recommendations for putting these tools in the right place:

    1. Allocate by task type, not by level of trust. Long-horizon, cross-module, repetitive coding work is a good fit for continuous model-driven execution. Design decisions involving permissions, security, and core business rules must keep human review.
    2. Write security constraints explicitly into the task context. The clearer the constraints around uploads, authentication, and unauthorized access, the less likely the model is to miss them. Constraints left implicit in “common sense” are where problems start.
    3. Test cases should cover unauthorized paths, not just normal ones. Automated tests should actively construct illegal access, boundary inputs, and abnormal states, bringing issues like permission isolation into the verification scope early.
    4. Manual acceptance cannot be skipped. No matter how green the tests are, schedule human acceptance — especially on security and permissions. Treat it as the final gate of the quality loop.
    5. Prefer options you can self-host and verify. Models with open weights that can be deployed locally make it easier for a team to validate and improve on their own real data, and the evaluation conclusions become more trustworthy.

    Conclusion

    Overall, tools like these have pushed the practical ceiling of long-horizon coding forward: they can read complex context, keep cross-module changes moving forward, and bring tasks to a runnable, testable state. That is already enough to put them into real project workflows, and they are especially well suited to long-horizon, production-grade coding work.

    But “worth testing” is not “ready to hand off”. Permissions, security, and final acceptance still need a human backstop. The rational stance is to treat it as a capable collaborator whose output needs review — use it to compress coding time and filter out obvious problems, while keeping the critical security judgments and acceptance responsibility firmly on the human side.