From Leaderboards to Real Projects: How to Scientifically Evaluate LLM Coding Ability

    When teams pick an AI coding tool for real development, they often start with benchmark leaderboards. But rigorous hands-on testing frequently tells a different story, because leaderboards measure static, controlled correctness while real software engineering demands progress through ambiguous, interdependent, security-constrained codebases. This article breaks down a reusable evaluation method built around long-horizon coding, examines where post-training gains come from, and proposes two complementary test tasks: modernizing a legacy system and building from scratch. It concludes with a clear boundary: models can take on implementation work in real projects, but permissions, security, and final acceptance must stay with humans. Readers will come away with practical guidance on designing cross-module tests, distinguishing "it runs" from "it ships," and building layered safety nets.

    Introduction: Why Benchmark Scores Are Not Enough

    When choosing an AI coding tool for real development, many teams look at benchmark leaderboards first: if a model leads on a coding benchmark, it seems ready to handle any project. But a rigorous hands-on test often leads to a different conclusion. Leaderboards measure the static ability to “produce the correct answer to a well-defined problem in a controlled environment,” while real software engineering requires the ability to steadily push a task toward a runnable, testable state under ambiguous context, cross-module dependencies, legacy baggage, and security constraints. The gap between the two is the core of this article.

    My basic judgment is this: once a large model’s coding ceiling is pushed high enough, it is worth validating in a real project, especially in long-horizon coding scenarios; but as soon as permissions, security, and final acceptance are involved, humans cannot fully step back. Below I break this conclusion into a reusable evaluation method.

    Long-Horizon Coding: What Really Deserves Our Attention

    “Long-horizon coding” means a model must maintain goal consistency over an extended span, steadily advancing a task that crosses multiple modules, rather than producing isolated snippets in one shot. It differs from traditional “function-level completion” along three dimensions:

    • Depth of context understanding: Can it read the existing directory structure, data models, interface contracts, and established implementation conventions, instead of reinventing the wheel from scratch?
    • Coherence of changes: In a single monorepo, can it touch the contract layer, database, task state machine, interfaces, background workers, and the frontend and rendering modules at once, while keeping the changes mutually consistent?
    • State convergence: Can it actually bring the task to a “compiles, tests run, boundaries verifiable” state, rather than stopping at a half-finished product that merely “looks done”?

    An evaluation of coding ability should prioritize designing tasks that expose these three points, rather than looking only at the accuracy of isolated algorithm puzzles.

    Where the Capability Gains Come From: The Value of Post-Training

    One noteworthy technical phenomenon is that, without changing the base model, scaling post-training alone can significantly improve a model’s long-horizon task performance. The key usually lies in three points:

    1. Expansion of the training environment scale: Placing the model in far more real task environments of richer variety, having it repeatedly rehearse the full loop of “detect a problem, locate it, make the change, verify it.”
    2. Longer training duration: More thorough exploration makes the model more stable under complex dependencies and reduces the chance of “going off track” or “abandoning the task halfway.”
    3. Shift in capability focus: Extending from “generating code” to “reading code and reasoning about its correctness,” including basic code review and vulnerability identification.

    This line of reasoning shows that when evaluating a new version, we should not only ask “did the parameters change?” but also “did the training paradigm change, and did the task distribution expand?” Those are the real sources of change in long-horizon ability.

    Evaluation Methodology: Designing Two Complementary Tasks

    A single evaluation task easily leads to a one-sided conclusion. A more reliable approach is to design two tasks with complementary capability profiles in parallel, one testing “renovating the old world” and the other testing “building a new world from scratch”:

    • Task A (Legacy system modernization): In a project with a large body of historical code and built-in security constraints, add a new capability that involves user data processing and external presentation. It tests the model’s ability to understand the old system, respect existing conventions, and work within security boundaries.
    • Task B (From zero to one): Give the model only a requirements document and a set of reference materials, and require it to implement from scratch an application that can be fully used end to end. It tests completeness across the whole path, from analyzing the materials and breaking down the product to engineering delivery.

    Passing both tasks only shows that the model is “basically usable”; the real dividing line appears in the manual acceptance stage.

    Practice One: Handling Security Constraints in a Legacy System

    In Task A, it is already no small feat for the model to locate the module that needs changes, follow the project’s existing layering, and generate runnable code. But manual acceptance exposed a typical problem: functionally correct, but permission isolation broken.

    For example, a newly added “asset import and reference” capability passed the entire business flow, yet it did not strictly validate resource ownership, so a user could construct an identifier to gain unauthorized access to someone else’s data. Below is an illustrative comparison of the flawed and corrected versions:

    1. # Bad example: only checks login state, not resource ownership, creating a privilege escalation risk
    2. def get_asset(request, asset_id):
    3. user = require_login(request)
    4. asset = asset_repo.find(asset_id)
    5. return asset # Any logged-in user can read any resource
    6. # Good example: validate ownership at the access layer, deny by default
    7. def get_asset(request, asset_id):
    8. user = require_login(request)
    9. asset = asset_repo.find(asset_id)
    10. if asset is None or asset.owner_id != user.id:
    11. raise PermissionDenied() # Always reject unauthorized access
    12. return asset

    The root of this kind of problem is that models are good at reproducing the “happy path” that works, but tend to overlook security defaults such as “untrusted input + deny by default.” These issues do not make tests fail, so “all tests green” alone can never surface them. This is exactly why security-related capability must be backed up by manual review or dedicated security auditing.

    Practice Two: From Requirements Materials to a Deliverable Product

    Task B is closer to a real product going from 0 to 1. Ideally, the model should be able to: parse the requirements document, break it down into the core mechanics and data structures, design module boundaries, generate runnable code, and get the core paths fully working end to end.

    When evaluating, I suggest focusing on three points: first, requirements coverage, whether any key rules from the PRD were missed; second, runnability, whether it actually runs rather than stopping at pseudocode; third, consistency, whether the data contracts between modules are self-consistent. If the model can bring these three points up to a passing standard, it already has the potential to take on small and medium-sized greenfield projects.

    A Weakness You Can’t Ignore: Security Isolation and Final Acceptance

    Synthesizing the two tasks yields a clear boundary: a model’s coding ability can be tested in a real project, but responsibility for permissions, security, and final acceptance cannot be transferred to the model.

    There are three reasons:

    • Security flaws often do not affect functional tests, so automated pipelines naturally struggle to cover them;
    • Problems like privilege escalation, injection, and sensitive data leakage carry severe consequences and a high cost of trial and error;
    • Acceptance involves business semantics and compliance judgment, which falls outside the model’s deterministic capability.

    A pragmatic approach is to treat the model as an efficient “implementer” and “refactorer,” while building multiple layers of defense with security scans, permission test cases, code review checklists, and manual acceptance.

    Conclusion and Practical Recommendations

    Here are a few actionable suggestions for teams preparing to adopt AI coding tools:

    1. Use long-horizon tasks as the evaluation baseline, designing two kinds of tasks, cross-module modernization and from-scratch implementation, rather than relying on leaderboards.
    2. Distinguish “it runs” from “it’s deliverable”: passing tests is only the starting point, so be sure to add negative test cases targeting permissions and inputs.
    3. Establish a security safety net: mandate ownership validation and least-privilege review for changes that involve data access.
    4. Keep acceptance in human hands: the model provides productivity, while humans own the boundary of responsibility, especially in security and compliance.

    In the end, evaluating a large model’s coding ability is essentially assessing the “division-of-labor ratio between human and machine.” The reasonable posture at this stage is: let the model take on more repetitive, cross-module implementation work, while humans focus on security, architectural decisions, and final acceptance. This both unlocks productivity and holds the line on quality.