AI Coding Tools in Long-Horizon Development: Test Results and Engineering Delivery Lessons

    This article explains why short-horizon public benchmarks fall short when judging real coding ability, and breaks down what long-horizon coding actually tests: context understanding and retrieval, cross-module coordinated changes, advancement under constraints, and convergence to a usable state. It reports two desensitized hands-on tests—adding a feature to a mature legacy project and building a deliverable product from scratch—along with the testing criteria, observed defects (including an object-level authorization flaw), a side-by-side comparison, four common mistakes, and a human-acceptance checklist. The takeaway: AI coding tools are productive collaborators for heavy, cross-module work, but security reasoning and final acceptance still require a human gate.

    1. Why Benchmark Scores Don’t Tell the Whole Story

    Most short-horizon public coding benchmarks measure one thing: the ability to answer a single-point question. Given a function, an algorithm description, or a local defect, can the model produce correct code? Some repository-level benchmarks go further and involve multi-file changes. This article, however, focuses on the gap between short-horizon, single-point benchmarks and real delivery.

    Real engineering delivery tests something entirely different: whether a model can continuously advance a long task that spans multiple rounds and multiple modules, until the system actually runs, passes tests, and stays maintainable. The two capabilities differ structurally:

    • Short-horizon benchmarks are typically short, isolated, and complete in context—the problem statement already contains everything needed to solve it.
    • Real development is long-horizon, continuous, and sparse in context. Requirements are scattered across documentation, historical code, interface contracts, permission models, and implicit conventions. The model must fill in the gaps itself and maintain global consistency.

    So the question of whether an AI coding tool is good cannot be answered by its score on a single algorithm problem. It depends on endurance and stability across long tasks: can it understand complex context, keep advancing cross-module changes, and carry the work to a “runnable, testable, acceptable” state?

    2. What Long-Horizon Coding Actually Tests

    Long-horizon coding ability breaks down into four progressive requirements:

    1. Context understanding and retrieval: In an older project full of historical code, accurately identify the relevant contracts, data structures, call chains, and existing conventions—rather than inventing a new style from scratch.
    2. Cross-module coordinated changes: In a monorepo or a combined frontend-backend project, a single feature often touches the contract layer, the data persistence layer, the task state machine, the interface layer, background jobs, the frontend pages, and even rendering. Miss any one link and the feature cannot run end to end.
    3. Advancement under constraints: The model must work within established security boundaries, compatibility requirements, and the chosen tech stack—not ignore constraints and “rewrite everything.”
    4. Convergence to a usable state: It’s not enough to “run.” The work must be verifiable—leaving behind executable tests, reproducible builds, observable runtime paths, and traceable defect records.

    3. Test Design: Two Differentiated Tasks and the Testing Criteria

    To cover different facets of these abilities, the test used two tasks that are deliberately far apart. Where a tool or environment shouldn’t be disclosed publicly, it can be presented as a desensitized case study—but the testing criteria must stay reviewable.

    Desensitized Testing Criteria

    Item Details
    Tool or model A mainstream AI coding assistant or code model (desensitized)
    Version Internal trial version from March 2026
    Test dates March 10–20, 2026
    Project scale Task A: ~30 modules in a legacy project; Task B: built from scratch, including homepage, question-bank page, checkout page, and state machine
    Tech stack Task A: Python backend, React frontend, object storage; Task B: Python/TypeScript, frontend pages, local database or object storage
    Runtime environment Local containerized environment plus a CI pipeline, supporting reproducible builds and log observation
    Prompt summary Task A: Add user asset import and animation-scene references within the existing permission system without breaking existing logic; Task B: Implement a complete, winnable knowledge-learning game based on learning materials and a PRD
    Acceptance cases Main functional flows, object-level permissions, exception branches, rollback, regression, build, log monitoring
    Pass criteria Zero P0 and P1 defects, P2 defects clearly recorded, basic tests pass, builds are reproducible

    Task A: Adding a capability to a mature legacy project. In a project with a large body of historical code, add a “secure user asset import and animation-scene reference” feature. This task tests two things: understanding the structure of an old system, and advancing carefully under security constraints without breaking existing logic.

    Task B: Building a deliverable product from scratch. Given only a set of learning materials and a product requirements document (PRD), implement from scratch a knowledge-learning game that can be played to completion. This task tests completeness across the whole path—from analyzing materials and breaking down requirements to shipping the product—and is closer to the “here’s a vague goal, deliver independently” scenario.

    One task is about “modifying someone else’s world safely,” the other about “building a city on empty ground.” Combined, they give a reasonably three-dimensional test of long-horizon coding ability.

    4. Key Observations and Problem Analysis

    Observation 1: Long-Horizon Endurance Is the Biggest Strength

    Under the acceptance cases and manual review defined here, both tasks ran and passed basic tests. The model was able to keep working within a monorepo, coordinate frontend, backend, and background jobs, and advance the feature to a testable state. This is a clear improvement in long-horizon capability over single-point generation.

    Observation 2: Runnable Is Not the Same as Deliverable

    Task A passed its tests and was functional, but manual acceptance still surfaced high-priority defects. Task A had 1 P1-level permission isolation issue, plus 2 P2 issues covering an incomplete error message and a missing rollback path. Illustrative code:

    1. # Broken authorization example: checking "is logged in" alone is far from enough
    2. def import_asset(current_user, asset):
    3. # Wrong approach: only checking whether the user is authenticated
    4. if not current_user.is_authenticated:
    5. raise Unauthorized()
    6. # Risk: no ownership check, may operate on another user's asset
    7. return storage.put(asset.id, asset.file)
    8. # Correct approach: must verify resource ownership for row-level/object-level isolation
    9. def import_asset(current_user, asset):
    10. if not current_user.is_authenticated:
    11. raise Unauthorized()
    12. if current_user.id != asset.owner_id: # Object-level authorization check
    13. raise PermissionDenied("Unauthorized access: no rights to operate on another user's resource")
    14. return storage.put(asset.id, asset.file)

    Object-level authorization should be combined with tenant, role, and data-ownership policies, so the example isn’t reused mechanically. The root of these problems is that the model tends to implement the feature’s happy path while paying too little attention to boundary authorization, object-level permissions, and exception branches. It can light up the feature, but it doesn’t guarantee the feature is lit up securely.

    Task B’s win flow was verified by manually checking each level against the PRD, combined with a script to verify answering, scoring, settlement, and the completion state. Manual review found no P1 security defects in Task B, but one P2 issue: when the question bank is empty, there is no friendly empty-state prompt.

    Task A vs. Task B

    Dimension Task A: Legacy project modification Task B: Build product from scratch
    Context understanding Can locate the main call chains, but implicit conventions still depend on human input Can quickly break down requirements and generate structure from the materials
    Cross-module changes Frontend, backend, storage, and task state basically connected Pages, question bank, checkout, and state machine form a closed loop
    Security constraints Insufficient permission isolation; 1 P1 defect No high-risk permission issues, but input validation needs to be added
    Delivery acceptance Basic tests pass; manual acceptance found defects Win flow passes; needs empty-state and prompt text

    Observation 3: Security Capability Needs to Be Read in Two Layers

    One is the code-generation layer—writing functional code to meet requirements. The other is the white-box review layer—actively reading code, reasoning about vulnerabilities, and spotting permission bypass or injection risks. The former is fairly mature; the latter, especially reasoning through a complete attack path, still lags behind top-tier closed-source models. In other words, expecting a tool to “discover its own security holes” is not realistic today.

    5. Common Mistakes

    In practice, four types of mistakes recur with long-horizon coding:

    1. Misjudging by the leaderboard: Looking only at single-point benchmark scores while ignoring context retrieval, cross-module coordination, and acceptance requirements in real projects.
    2. Shipping on all-green tests: Passing basic tests doesn’t mean security, permissions, exception handling, and rollback are reliable.
    3. Missing cross-module links: Changing an interface but forgetting the frontend, or changing a state machine but forgetting background jobs—so the feature can’t run end to end.
    4. Broken authorization: Checking only login state without checking resource ownership, creating object-level privilege escalation risk.

    6. Where to Draw the Line Between AI Tools and Human Acceptance

    Long-horizon coding is well suited to the heavy lifting of “from requirements to a runnable prototype,” especially long, cross-module changes. But anything involving permissions, security, compliance, and final acceptance must be gated by a human—no “all tests green, ship it.”

    A good practice is to define acceptance criteria upfront as executable tests, and to confirm the following checklist before assigning a task:

    • Object-level permissions: Are resource ownership, tenants, roles, and access boundaries checked?
    • Least privilege: Is access denied by default, granting only what’s needed for the current task?
    • Exception branches: Are nulls, unauthorized access, timeouts, duplicate submissions, and concurrency conflicts handled?
    • Rollback: Can failed operations be rolled back, and is a recoverable state preserved?
    • Regression: Are there automated regression tests for the core paths?
    • Log monitoring: Are key paths observable, and can errors be pinpointed?
    • Asset isolation and access control: Are user-uploaded assets isolated per user or per tenant, and are access-control policies configured?

    On the process side, beyond general cloud capabilities, local containerization, CI pipelines, and code-review platforms can serve as alternatives. Containers isolate and reproduce, CI automates build and test, and the code-review platform retains review records. For any code touching authentication, authorization, or data access, no matter how well it “runs,” it should go through manual code review or an independent security scan, with a focus on object-level authorization and the principle of least privilege.

    7. Conclusion: From Single-Point Accuracy to Long-Horizon Delivery Quality

    The standard for evaluating AI coding tools is shifting from “single-point answer accuracy” to “endurance and delivery quality on long-horizon tasks.” This test shows that, given clear requirements, executable acceptance criteria, and human security oversight, such tools can handle cross-module, long-cycle changes in real projects and have the potential to build from zero.

    Still, there is a clear gap between “runs” and “safe to deliver.” Permission isolation, security reasoning, and final acceptance remain areas where humans are irreplaceable. The sensible stance is to treat these tools as high-output collaborators capable of carrying heavy loads, while keeping the human as the final gate on security and acceptance.