When Coding Models Take On Long Tasks: What Real-World Evaluation Reveals Beyond Benchmarks

    The latest generation of general-purpose large language models keeps pushing the ceiling on coding ability. This article looks beyond benchmark scores and evaluates how these models perform on long-horizon tasks — whether they can read complex context, make consistent changes across modules, and bring a task to a runnable, testable state. Through two hands-on evaluations (extending an existing security-sensitive project, and building a completable learning game from scratch), the author finds clear progress: models are now worth using on real projects, especially for long coding tasks. But green tests do not equal safe delivery — one test surfaced a high-priority (P1) permission-isolation defect that all of the model's own tests passed. The article closes with reusable acceptance checklists, an authorization matrix, a lightweight scoring sheet, and principles for dividing work between humans and AI.

    Conclusion First

    Start with the bottom line: the latest generation of general-purpose large language models keeps pushing the ceiling on coding ability. More interesting than benchmark scores is how they perform on long tasks — whether they can read complex context, keep making changes across modules, and bring a task to a “runnable, testable” state. On these dimensions, this round of hands-on testing shows clear progress.

    But there is another half to the conclusion: current models are not yet at the level where “all tests green means you can ship with confidence.” The author ran a round of testing on a general-purpose model that closely mirrors real development scenarios, split into two task types:

    • Reworking a project that has accumulated a fair amount of legacy code and carries explicit security constraints;
    • Implementing a learning game from scratch based only on a public AI-agent learning resource and a product requirements document.

    Both projects ultimately ran end to end. However, the first one exposed a top-priority (P1) permission-isolation defect during manual acceptance. The applicable boundary this suggests: models are now worth investing in real projects, especially for long-horizon coding tasks; but on permissions, security, and final acceptance, humans cannot fully let go.

    The two tasks prepared for this evaluation are very different, chosen to observe model ability from different angles:

    • Task A: Add the capability for “secure import of user image assets and their referencing in HTML animation scenes” to an existing system, emphasizing understanding legacy systems, cross-module consistency, and security constraints;
    • Task B: Provide only an AI-agent learning resource and a product requirements document, then implement from scratch a learning game that can be fully completed, emphasizing the full chain from resource analysis and structural design to product delivery.

    Below, the article first discusses how to evaluate coding ability beyond benchmarks, then presents the evaluation method, then details the observed performance on each of the two tasks, and finally offers a reusable acceptance checklist and principles for human-AI division of labor.

    1. Why Benchmarks Alone Are Not Enough: Long Tasks Are the Real Test

    1.1 Benchmarks Test “Points,” Development Cares About “Lines”

    Public evaluation benchmarks are usually framed as single-turn Q&A, isolated function completion, or a small number of patches. Their strengths are consistent methodology, reproducibility, and ease of horizontal comparison, but they have two limitations:

    1. Limited context. Evaluation problems rarely include a full directory structure, design docs, past decisions, and implicit conventions — which are precisely the main sources of information in real projects.
    2. The result is the endpoint. A benchmark only checks whether a single output is correct; real development goes through implementation, self-testing, integration, fixing, regression, and acceptance — and a problem at any stage affects final delivery.

    More importantly, knowing a problem and doing the engineering are two different abilities. A model knowing that “file uploads should be type- and size-checked” does not mean it can find the correct place for that check within a project of hundreds of files and guarantee that nothing is missed along the check chain. Long tasks are exactly what expose this gap.

    1.2 Long-Horizon Coding Ability Breaks Down Into Five Observable Dimensions

    To avoid vague judgments like “it feels stronger,” long-horizon coding ability can be broken down into five observable dimensions, each corresponding to a clear acceptance action:

    Dimension Meaning Typical failure mode
    Context understanding Reads existing architecture, historical conventions, and implicit constraints Introduces a new style that conflicts with the project
    Planning and decomposition Breaks a vague requirement into executable, verifiable subtasks Generates a large amount at once, hard to localize problems
    Cross-module consistency Aligns contracts, storage, interfaces, async tasks, and frontend changes Interface fields don’t match frontend values
    Progress and self-correction Locates the cause and fixes it when hitting errors or failing tests, instead of repeatedly patching The same class of error keeps recurring
    Delivery completeness Output is runnable and testable, with key paths documented Runs but lacks error handling, and requires manual rework after delivery

    2. Evaluation Method and Scoring Criteria

    2.1 Design Logic Behind the Two Task Types

    A single task cannot cover all five dimensions, so this evaluation deliberately picked two complementary types:

    • Task A (incremental development under constraints): Extend an existing system with a new feature. The model must understand the old code, respect security constraints, and keep the new capability compatible with existing modules;
    • Task B (a complete from-scratch implementation): Provide only resources and a requirements document, testing the full chain from information extraction and structural design to implementation and delivery.

    Together, the two tasks cover all five dimensions. Task A mainly checks context understanding, cross-module consistency, and delivery completeness; Task B mainly checks planning, decomposition, and progress.

    2.2 A Unified “Three-Round Delivery” Process

    To control variables in a single experiment and stay close to a real iteration rhythm, both tasks followed the same process:

    1. First round — implementation: The model independently breaks down the task and completes the core functionality.
    2. Second round — verification and fixing: Using browser automation, command-line execution, and API testing, the model verifies its own work and fixes issues based on the results.
    3. Third round — manual acceptance: Humans check each item against an acceptance checklist, focusing on negative cases for security and permission design.

    The reason for this arrangement is that the first two rounds reflect the model’s own execution and self-correction ability, while the third round reflects the complementary value of the human perspective. Only by recording all three separately can we tell where the “ability boundary” actually lies.

    2.3 Scoring Is Based on Acceptance Results

    Scoring is based not on “whether a solution was proposed” but on “whether it passes acceptance.” Each requirement maps to a clear pass condition, for example:

    • Functional correctness: A specified operation returns the expected result;
    • Testability: Key paths have automated tests that run with a single local command;
    • Security: Unauthorized access is explicitly rejected, rather than “returning empty data” or “silently degrading”;
    • Maintainability: New logic follows the existing layering, with no duplicate implementations.

    3. Task A: Incremental Development in an Existing Project With Security Constraints

    3.1 Task Background and Challenges

    Task A took place in an existing modular monorepo project that already had animation-related functionality. The new capability chain to add: a user uploads image assets, which pass security checks and are then stored, after which HTML animation scenes reference these assets.

    The difficulty is not writing new code, but handling three things at once:

    • Understanding legacy code: The existing project already includes a domain-contract layer, database access, an API layer, an async task executor, and an HTML rendering stage. The new feature had to be consistent with the existing layering rather than invent a new structure.
    • Keeping the chain consistent: An asset travels from upload to being referenced by an animation, passing through multiple modules. A field or state mismatch at any layer breaks the whole chain.
    • Meeting security constraints: The word “secure” is a hard requirement, including file type and size checks, storage paths that cannot be traversed, and protection against script injection during rendering. These requirements are scattered across the chain and easy to miss.

    3.2 Observed Performance: Long Context and Cross-Module Progress

    From the execution process, the model’s behavior can be summarized as “moving forward along the established structure”:

    • It first located where the existing animation chain sits in the contract layer, API layer, and rendering stage, then extended it in the same manner rather than creating a new structure;
    • When facing errors and failing tests, it could trace back to the relevant module to find the cause, rather than repeatedly patching at the error site;
    • Implementation and verification were done in phases: after the first round, the second round of verification and fixing used browser automation and API tests. Each of the two rounds took more than an hour. The time was driven mainly by the large scope of changes and the many verification steps, not by the model frequently “getting stuck.”

    This shows that in a well-structured project, the model can now reliably deliver a feature spanning both frontend and backend. This relates directly to the “engineering endurance” that benchmarks cannot reflect: what is truly scarce in long tasks is the ability to keep producing usable changes.

    3.3 The P1 Defect Exposed During Acceptance: Permission Isolation

    After functional verification passed, manual acceptance checked each item against the checklist and found a P1-level defect in the permission model. The defect is unauthorized access (commonly recorded in the industry as IDOR): as long as someone knows an asset’s identifier, they can bypass ownership checks and read or manipulate assets that do not belong to them.

    Looking at the shape of the defect, the problem is not “authentication was skipped,” but that the granularity of authorization was wrong. Most endpoints already checked “whether the user is logged in” and “whether the requester’s role matches,” but did not check “whether this resource belongs to the requester.” In single-user or demo scenarios, this style typically does not surface a problem. Once it enters a real environment with multiple users or project isolation, it becomes a typical horizontal privilege-escalation entry point.

    More notably, all of the model’s tests passed. The reason is that the test cases and the implementation were “in step” on the same defect:

    • The test cases followed the tested endpoint’s assumptions, defaulting to “a logged-in user can access all assets under their own project”;
    • When the test cases are generated by the model itself, it tends to verify that “the requirement is implemented” rather than challenge “whether the requirement is secure”;
    • The model produced positive cases — uploading a valid image, an animation scene correctly referencing it, an invalid file being rejected — but did not produce negative cases like “user A trying to access user B’s assets.”

    This is exactly the gap between “all tests green” and “safe to deliver”: green only proves the code satisfies the current cases, not that the cases cover the real risks. The earlier such problems are found, the lower the fixing cost.

    3.4 Fix Approach: Centralized Authorization Checks

    The root cause of the defect is that authorization judgments were scattered across individual endpoints, making it easy to forget one when adding a new endpoint. The first step in fixing it is to converge authorization into a single entry point: any asset operation first passes an ownership check, and on failure returns a uniform rejection result, rather than different endpoints each returning “empty results” of different shapes. Example code:

    1. # Single entry point for all asset operations: authenticate first, then authorize by ownership
    2. def access_material(user_id: str, material_id: str, action: str) -> Material:
    3. material = material_repo.get_or_none(material_id)
    4. if material is None or material.owner_id != user_id:
    5. # Return the same response for "not found" and "no permission" to avoid leaking whether the resource exists
    6. raise ForbiddenError("forbidden")
    7. material_repo.assert_action_allowed(material, action)
    8. return material
    9. # Authorization case matrix covering "role x ownership x action"
    10. authz_cases = [
    11. ("user_a", "mat_a_1", "read", True), # owner reads own asset
    12. ("user_b", "mat_a_1", "read", False), # unauthorized read by another user
    13. ("user_b", "mat_a_1", "delete", False), # unauthorized delete by another user
    14. ("user_a", "mat_b_1", "update", False), # cross-project operation
    15. ]
    16. for actor, material_id, action, allowed in authz_cases:
    17. result = try_access(actor, material_id, action)
    18. assert result.succeeded is allowed, (actor, material_id, action)

    The second step in fixing it is to lock the authorization matrix into automated tests and include them in continuous integration, preventing the same class of problem from being reintroduced in later iterations. The value of manual acceptance is precisely switching the model’s “implementation perspective” to an “attacker’s perspective.”

    4. Task B: Building a Completable Learning Game From Scratch

    4.1 From Resources to Levels: Testing Information Structuring

    Task B provided only two inputs: a detailed AI-agent learning resource and a product requirements document, asking for a learning game that can be fully completed.

    In processing this, the model had to make several key transformations:

    1. Extract knowledge units from natural-language material, such as concepts, categories, processes, and common mistakes;
    2. Order the knowledge units into a level sequence that goes from shallow to deep;
    3. Turn abstract knowledge into interactive, judgeable level goals.

    This transformation from unstructured text to structured level design heavily tests long-context understanding and planning. It requires grasping the whole picture of the material while not over-pruning its knowledge density.

    4.2 Gameplay, Feedback, and Save State

    In terms of game mechanics, the model implemented a fairly complete closed loop, including:

    • Level progression and completion judgment: Completing one unlocks the next; finishing all levels shows a summary;
    • Instant feedback: Correct answers or operations get an explanation, while wrong ones get a hint rather than the answer directly;
    • Progress persistence: Completed levels and scores are recorded, so play can resume when re-entering.

    On the engineering side, it decoupled learning content from game logic: knowledge data is stored separately, making it easy to expand the question set later without touching the core logic. This matters especially in a from-scratch project.

    4.3 Issues Worth Noting in the From-Scratch Phase

    Getting features running is only the starting point. Manual review still found a few things worth attention:

    • Whether the acceptance standard is clear: “Completing” the game in the requirements document needs a clear definition. Is it counted by levels, by score, or by completing all knowledge units? When the standard is unclear, the model tends to pick the easiest one to implement — “completable” on the surface, but possibly diverging from the product’s intent.
    • Whether judgment is robust: Game-type requirements have many edge cases, such as repeated submissions, mid-way exits, out-of-order operations, and extreme inputs. Whether these are handled correctly directly determines how trustworthy “completion” is.
    • Whether the content is accurate: The knowledge content the model generates may deviate from the reference material. Game logic can be self-tested, but content accuracy needs manual verification against the original material — you can’t assume that a model “understood it” means it “explained it correctly.”

    5. Turning Evaluation Conclusions Into a Reusable Acceptance Method

    5.1 Checklist One: Prompting and Decomposition for Long-Horizon Tasks

    • Give the full picture: At the start of a session, have the model first map out the project structure, existing conventions, and a list of related modules before implementing.
    • Contracts first: For cross-module changes, settle the interface contracts and data structures first, then implement against them to reduce frontend-backend mismatches.
    • Acceptance in phases: Break the large task into several verifiable milestones, each runnable and testable, avoiding delivering a large amount of code all at once.
    • State security constraints explicitly: Write requirements like authentication, input validation, path restrictions, and output escaping as hard items, and provide the negative cases that must pass. This is the most direct way to compensate for “implementation and cases failing in sync.”

    5.2 Checklist Two: Manual Acceptance Before Delivery

    Regardless of task complexity, the acceptance stage should cover three types of checks:

    • Structural consistency: Does new code follow the existing layering and naming conventions, without introducing duplicate implementations or “shortcuts” that bypass the layering?
    • Permissions and authorization: Check authentication and authorization endpoint by endpoint; build a case matrix of “role x ownership x action”; confirm there is no path where “knowing the identifier is enough to access.”
    • Security and input boundaries: Upload features must validate file type, size, and content, and generate filenames rather than using user input directly; storage paths must be normalized to prevent directory traversal; user-controllable content must be escaped or isolated when rendered.

    The authorization matrix can be maintained with a simple structure for item-by-item review:

    1. Endpoint Auth Owner access Same-project others Cross-project
    2. Asset list query Required Allow Deny Deny
    3. Asset detail view Required Allow Deny Deny
    4. Asset delete Required Allow Deny Deny
    5. Asset metadata update Required Allow Deny Deny

    5.3 Human-AI Division of Labor: Three Things Not to Let Go

    Stage More reliance on the model Must be owned by humans
    Requirements and architecture Organizing resources, drafting proposals Defining requirement boundaries, architecture, and key trade-off decisions
    Coding and testing Cross-module implementation, writing unit and API tests Reviewing test effectiveness, designing negative cases
    Acceptance and release Running regression tests, initial defect triage Permission and security review, final sign-off

    The fundamental principle in practice is just one: the model is responsible for “whether the code got written,” and humans are responsible for “whether the code should be written this way.” Permissions and security involve accountability and should not be left for auto-generated test cases to prove themselves.

    5.4 A Simple Scoring Sheet

    To compare the long-horizon coding performance of different models within a team, you can use a lightweight scoring sheet. Score each task independently out of 100; below 80 is not recommended for direct entry into real projects:

    Dimension Points Scoring focus
    Context understanding 20 Whether it follows existing structure and conventions
    Planning and decomposition 15 Whether it works in phases with verifiable milestones
    Cross-module consistency 20 Whether contracts, data, and frontend align
    Self-correction and progress 20 Whether it can independently locate and fix problems
    Delivery completeness 15 Runnable, testable, with key documentation
    Security and permissions 10 Whether it passes manual authorization and security review

    6. Conclusion

    The judgment from this evaluation can be summed up in three points:

    1. The ceiling of long-horizon coding is rising: models can now understand context in complex projects, make progress across modules, and bring a task to a runnable, testable state — worth trying on real projects.
    2. Green tests do not equal safe delivery: a model’s implementation and its test cases can miss permission boundaries at the same time. Human acceptance must independently design negative cases and cover security scenarios.
    3. The accountability boundary must be drawn up front: on permissions, security, and final acceptance, humans cannot let go.

    To use this capability in production, the recommendation is to embed it in a controllable engineering pipeline:

    • Input side: requirement documents and code repositories integrated with version control;
    • Process side: reproducible build and test environments provided in a container platform, with the model’s changes automatically triggering continuous integration;
    • Artifact side: build outputs and runtime logs stored separately in object storage and a logging service;
    • Observability side: monitoring and alerting used to track actual performance after delivery.

    On Baidu AI Cloud, these foundational capabilities can be combined as needed to build an evaluation and regression environment for measuring model coding performance.

    For a team, the more practical path is to first try it on small-scope, low-risk tasks and accumulate data with a unified scoring sheet, while locking acceptance items such as the authorization matrix and security checks into the pipeline as mandatory gates. Rising capability will always push boundaries outward, and the job of engineering practice is to make sure every step beyond the boundary has a human watching it.