Long-Horizon Coding Capability: Evaluating AI from "Runnable" to "Deliverable"
This article argues that evaluating AI coding systems should go beyond whether the generated code runs. Using two representative tasks—adding security-sensitive features to a legacy system and building a complete interactive learning game from scratch—it shows what long-horizon coding capability actually demands. It then distills eight evaluation dimensions, discusses the strengths and boundaries of long-horizon coding, and outlines a practical five-step assessment process. The core conclusion: a mature standard moves from "runnable" to "testable," to "secure and controlled," and finally to "acceptable through human review."
1. Why “Does the Code Run?” Is Not Enough
In real engineering settings, the value of code generation or AI-assisted programming does not come from producing a function that looks correct. It comes from whether the system can keep advancing a complete task inside a complex context. Long-horizon coding capability usually covers the following layers:
- Context understanding: recognizing an existing system’s architecture, module boundaries, dependencies, naming conventions, and hidden constraints.
- Task decomposition: breaking a vague requirement into executable steps such as interfaces, data models, task state, asynchronous processing, page interactions, and test verification.
- Cross-module changes: staying consistent across frontend, backend, the data layer, the task layer, and the rendering layer, instead of editing a single local file.
- Verifiability: proactively adding tests, exception branches, state rollback, and observability, rather than only aiming for something that “seems to run.”
- Delivery closure: producing something that not only runs but also passes tests and human review, and carries the security boundaries and acceptance evidence needed for release.
So when assessing long-horizon coding, it is not enough to check whether a single piece of code is correct, or whether all tests are green. A more sound approach is to construct tasks that resemble real R&D work and put the system through the full chain of “requirement understanding → solution design → implementation → test and fix → security review → final acceptance.”
This article uses two typical tasks for the assessment. One adds a security-sensitive feature to a legacy system with accumulated technical debt; the other builds a fully usable interactive learning application from scratch, starting from limited material and a requirements brief. The former tests understanding of old-system constraints and security boundaries; the latter tests the overall capability from organizing information to shipping a product.
2. Task 1: Adding Secure Asset Import and Web Animation References to a Legacy System
2.1 Task Background
The first task is to add two capabilities to a project that already carries a fair amount of legacy code: “user image asset import” and “secure referencing of web animation scenes.”
The value of this kind of task is that it is not simply adding a page or an endpoint. It is inserting a new feature chain into an existing system while trying not to break existing behavior. Typical constraints include:
- The existing user and permission model must not be disrupted.
- The existing resource storage approach must not be replaced arbitrarily.
- New files must handle type, size, naming, metadata, and access permissions.
- Referencing assets from pages must avoid introducing script injection, cross-user access, or resource privilege escalation.
- Changes must cover the API contract, data model, asynchronous tasks, state transitions, and frontend rendering.
- The result must not only run but also support rollback, troubleshooting, and future extension.
This kind of task heavily tests long-horizon coding, because it combines legacy-system understanding, security constraints, cross-module coordination, and engineering delivery requirements at once.
2.2 Key Evaluation Points in a Long-Horizon Task
2.2.1 Understanding the Legacy System Correctly
In legacy systems, the most common failure is not “not knowing how to write code” but “starting to make changes without understanding the old logic.” A mature implementer first confirms:
- Where the user identity comes from.
- Where the file upload entry point is.
- Where resources are saved.
- Whether resource access needs to pass through authorization.
- How animation scenes are referenced by pages.
- Whether existing endpoints already have versioning, compatibility, or fallback requirements.
If the system does not first locate these key points and instead directly generates a new upload endpoint and page logic, it easily causes permission bypass, duplicate implementations, inconsistent state, or regressions in old features.
So the first layer of long-horizon coding is the ability to interpret context. It must distinguish places that “look editable” from places that are “actually allowed to be edited.”
2.2.2 Secure File Import Awareness
Image asset import looks like a simple file upload feature, but it actually involves multiple security points. The assessment should not look only at “whether the upload succeeds” but also at whether the following are handled:
- Whether the file extension matches the real file type.
- Whether the file header signature matches the expected format.
- Whether the file size exceeds the limit.
- Whether image parsing risks exceptions or resource exhaustion.
- Whether abusable metadata has been stripped.
- Whether unpredictable access identifiers are used.
- Whether resources are scoped to the range visible to the user.
- Whether internal storage paths are exposed directly.
- Whether a failure feedback and cleanup mechanism exists for failed uploads.
The illustrative validation flow below shows the difference between a “secure import” and a “simple upload”:
validate_image(file, owner):assert allowed_type(file)data = read_limited(file, max_size)assert file_header_matches(data, expected_types)safe_data = normalize_image(data)safe_data = remove_metadata(safe_data)object_key = generate_unpredictable_key()store_private(object_key, safe_data, owner=owner)return create_temporary_access_url(object_key=object_key,scope=f"owner:{owner}",expires_in=reasonable_ttl)
The point of this pseudocode is not the specific language. It is that it moves security requirements forward into the task design: type validation, size limits, content normalization, metadata stripping, private storage, temporary access, and ownership binding are all part of a deliverable implementation.
2.2.3 Designing a Secure Web Animation Reference Mechanism
If pages need to reference user assets or animation content, the question is not just “can it display” but “can it display without expanding the attack surface.”
A sound implementation needs to define:
- Whether pages are allowed to embed user-provided content directly.
- Whether only trusted, structured scene configurations are allowed.
- Whether unverified scripts, event handlers, or external redirects are prohibited.
- Whether an isolated container carries uncertain content.
- Whether a content security policy restricts scripts, styles, and external resource loading.
- Whether asset access is scoped, so one user cannot read another user’s resource by guessing identifiers.
The core here is “capability boundaries” rather than “feature availability.” A feature that only covers the happy path but does not define the rejection path cannot count as a completed security design.
2.2.4 Closing the Loop Across Modules
This task may involve at least the following modules:
- API layer: receiving upload requests, validating parameters, returning resource reference information.
- Data layer: storing asset metadata, ownership information, and status information.
- Storage layer: isolating the saved original or normalized files.
- Task layer: handling asynchronous conversion, failure retries, and state updates.
- Rendering layer: safely loading and displaying animation scenes.
- Permission layer: deciding whether a user may access, reference, or delete an asset.
- Observability: recording failure reasons, access audits, and exception traces.
The difference in long-horizon coding often shows up right here: whether the system can expand a single change into a complete chain, instead of producing an isolated endpoint or page.
2.3 Typical Issues Surfaced in Manual Acceptance
A common risk in this kind of task is that all automated tests pass, yet manual acceptance still reveals high-priority issues. For example, a system may pass functional tests, upload tests, and display tests, but still expose a permission-isolation flaw:
- Users can access others’ assets by enumerating resource identifiers.
- The frontend hides the button, but the backend does not enforce ownership checks.
- Temporary access links have an unclear expiry policy, leaving resources exposed long-term.
- Animation configurations can reference unauthorized resources.
- The delete endpoint only checks the login state, not resource ownership.
- Failed asynchronous tasks still leave accessible intermediate files.
These issues are easy to miss because tests often assume “the user only follows the normal path.” But real engineering acceptance must account for privilege escalation, guessing, replay, enumeration, and abnormal states.
So for this kind of task, whether long-horizon coding is qualified cannot be judged only by whether the feature is implemented. It must be judged by whether permission boundaries, access control, and failure paths are built into the design.
3. Task 2: Building a Complete Learning Game from Scratch
3.1 Task Background
The second task is very different: only a body of agent-learning material and a requirements brief are provided, and a fully playable learning game must be implemented from scratch.
The core of this task is not “how much code to write” but whether the system can turn unstructured material into a runnable product. It requires the full capability from content analysis, task modeling, and interaction design to engineering implementation.
A qualified agent-learning game usually includes at least:
- Learning content structure: chapters, knowledge points, difficulty, and dependencies.
- Level mechanics: goals, pass conditions, feedback, and rewards.
- State management: progress, score, unlock status, and replay logic.
- Data persistence: user progress that can be saved and restored.
- Interaction closure: start, continue, fail, retry, and complete.
- Testability: key flows that can be automatically verified.
- Maintainability: content, logic, and presentation kept as separate as possible.
3.2 How Long-Horizon Coding Shows Up in This Task
3.2.1 Extracting Knowledge Structure from Source Material
The difficulty of a learning product is not in the pages but in content organization. The system needs to decide:
- Which concepts are essential for beginners.
- Which knowledge points have a required order.
- Which questions work as basic practice.
- Which challenges work as stage assessments.
- How to break the material into consumable levels.
If the system simply lists the material, the product feels like a document reader rather than a learning game. Truly valuable long-horizon capability turns content into a learning path that can be progressed through.
3.2.2 Designing a Complete Game Loop
“Being able to complete the game” is not a vague goal; it needs to be broken down into a verifiable state machine. For example:
Start -> Chapter 1 -> Chapter 1 completion check -> Unlock Chapter 2-> On failure, allow retry -> Record error points-> All chapters complete -> Generate completion result-> Support restart or review of weak areas
The key is that state transitions must be clear. Whether long-horizon coding is qualified can be judged by whether it can reliably implement these boundaries:
- Locked chapters cannot be entered.
- Completed chapters can be replayed.
- Failure does not cause progress loss.
- Scoring rules stay consistent throughout.
- State is not lost after a page refresh.
- Completion conditions are not triggered repeatedly.
- Empty data and abnormal data have clear prompts.
3.2.3 Building Engineering Structure from Zero
When building from scratch, a common risk is that the code structure quickly spins out of control as features grow. A good implementation establishes clear boundaries early:
- Content data separated from runtime state.
- Level logic separated from page presentation.
- User progress separated from learning material.
- Validation logic separated from rendering logic.
- Reusable feedback mechanisms implemented in one place.
- Key interfaces and models given explicit contracts.
This reflects long-term maintainability, not short-term demo capability.
3.3 The “Looks Done” Trap to Watch Out For
Build-from-scratch tasks most easily produce an illusion: the flow works, the pages are clickable, so it is judged complete. But manual acceptance still needs to check carefully:
- Whether the learning content is accurate, or over-simplified or misclassified.
- Whether levels can be completed repeatedly by different users, rather than relying on a specific path.
- Whether failure feedback is valuable, or just says “wrong.”
- Whether progress is truly saved, or only lives in memory.
- Whether interruption, exit, retry, and recovery are handled.
- Whether basic abnormal states and empty states exist.
- Whether hardcoded demo data replaces real logic.
In other words, being able to complete the game is only the minimum requirement. A truly deliverable learning game also needs trustworthy content, reliable state, continuous experience, and clear boundaries.
4. Evaluation Dimensions Distilled from Both Tasks
Combining these two task types, long-horizon coding capability can be broken into the following dimensions.
| Evaluation Dimension | Focus | Failing Signal | Passing Signal |
|---|---|---|---|
| Requirement understanding | Whether it identifies the real goal and constraints | Implements only surface features | Defines boundaries, dependencies, and acceptance criteria clearly |
| Context use | Whether it understands the existing system and material | Reinvents wheels or breaks old logic | Reuses sensibly and changes sparingly |
| Task decomposition | Whether it forms an executable plan | Fragmented implementation | Has module boundaries and an order of progress |
| Code consistency | Whether changes coordinate across modules | Chaotic interfaces, state, and naming | Clear contracts and a complete chain |
| Runnability | Whether it starts and demos correctly | Missing dependencies or broken flows | Starts, interacts, and rolls back |
| Testability | Whether it covers key paths and exceptions | Only the happy path | Has validation, exceptions, and regression awareness |
| Security | Whether it handles permissions, input, and access boundaries | Checks login but not ownership | Has clear isolation and minimal exposure |
| Deliverability | Whether it meets manual acceptance and release requirements | Suitable only for demos | Maintainable, auditable, and fixable |
The point of this table is to turn “can the model write code” into “is the engineering result acceptable.” That is also the essence of assessing long-horizon coding: not searching for a magical answer, but establishing a repeatable set of judgment criteria.
5. Strengths and Boundaries of Long-Horizon Coding
5.1 Strengths: Closer to the Real R&D Rhythm
From these tasks, the real value of long-horizon coding is not generating isolated code snippets but being able to keep advancing toward a goal. It usually shows the following strengths:
- It expands vague requirements into a task list: for example, growing “build a learning game” into content structure, level state, persistence, and acceptance criteria.
- It maintains change direction in a multi-file project: it is less likely to fall into the state of changing one point while ignoring the whole chain.
- It implements and fixes at the same time: when it hits errors, inconsistent state, or page anomalies, it can return to context and keep adjusting.
- It outputs a runnable skeleton: even if human polishing is ultimately needed, it significantly reduces the time to set up an environment from zero.
- It carries longer context: it suits tasks where documents, code, constraints, and tests all exist together.
These capabilities give it real value in R&D assistance, internal tooling, prototype validation, and complex refactoring scenarios.
5.2 Boundaries: Security and Final Acceptance Cannot Be Fully Automated
At the same time, the boundaries must be seen clearly. Long-horizon coding can greatly improve efficiency in many scenarios, but it should not be understood as fully replacing human responsibility.
In particular, humans are still needed to judge:
- Whether the permission model is truly isolated.
- Whether data ownership is rigorous.
- Whether sensitive operations need approval.
- Whether the security policy covers the threat model.
- Whether the business semantics are correct.
- Whether acceptance criteria are truly met.
- Whether unacceptable compatibility risks have been introduced.
This is not a capability defect but an inherent requirement of engineering delivery. No automated system can independently bear final responsibility, because that responsibility comes not only from code but also from business rules, organizational processes, and security boundaries.
6. Designing a Practical Long-Horizon Coding Assessment Process
If a team wants to continuously assess code generation or AI-assisted programming capability, it can borrow the following process.
6.1 Step 1: Make the Task About “Acceptance,” Not “Demo”
Assessment tasks should avoid giving only a vague goal. A better approach is to define acceptance criteria in advance, for example:
- Which user operations must be supported.
- Which exception branches must be covered.
- Which permission requirements must be met.
- Which state must be saved.
- Which tests or verification methods must be provided.
- Which behaviors count as forbidden items.
The clearer the acceptance criteria, the more reliable the assessment results.
6.2 Step 2: Prepare Real Context
Long-horizon tasks cannot do without context. The assessment should provide as much as possible:
- Existing system documentation.
- Key module boundaries.
- Permission model description.
- Data structure examples.
- Historical constraints.
- Known pitfalls.
- Material input or a requirements brief.
The more complete the context, the better one can tell whether the system “truly understands the task” or merely generates a surface-level answer.
6.3 Step 3: Check in Stages Rather Than Only Looking at the Final Result
It is advisable to split a long-horizon task into several observation points:
- Understanding stage: whether it restates the goal, constraints, and boundaries.
- Design stage: whether it provides module breakdown, data flow, and state transitions.
- Implementation stage: whether it advances module by module rather than piling up code in jumps.
- Verification stage: whether it proactively adds tests, exceptions, and boundaries.
- Delivery stage: whether it explains limitations, risks, open items, and acceptance results.
This staged check identifies long-horizon capability more accurately, rather than looking only at the final code block.
6.4 Step 4: Automated Tests Must Pair with Manual Security Review
For tasks involving permissions, files, payments, audits, privacy, and sensitive data, one cannot rely only on automated tests. At minimum, manual review should cover:
- Whether unauthorized access exists.
- Whether input is left unvalidated.
- Whether sensitive information is exposed.
- Whether state becomes inconsistent.
- Whether irreversible errors exist.
- Whether rollback is difficult.
- Whether audits are missing.
This is also why many projects cannot be delivered directly even when “all tests are green.”
6.5 Step 5: Establish Issue Classification and Regression Mechanisms
An assessment should not record only “there is a bug” but should record the severity level of the issue:
- Blocking: the core flow is unusable.
- High priority: security, permission, or data errors.
- Medium priority: experience defects or missed boundaries.
- Low priority: naming, structure, or maintainability suggestions.
At the same time, discovered issues should be captured as regression cases to avoid the same problems recurring. Long-horizon coding is not a one-time performance but whether quality can be maintained through continuous iteration.
7. Conclusion: The Standard for Long-Horizon Coding Should Upgrade from “Runnable” to “Acceptable”
Overall, evaluating long-horizon coding should not stop at “can it generate code” or “can it run a flow.” A more sound standard is:
- Runnable is only the starting point.
- Testable is the foundation.
- Secure and controlled is the key.
- Acceptable through human review is the finish line.
In the legacy-system enhancement task, it must face old code, the permission model, and security constraints. In the build-from-scratch learning game task, it must complete content organization, state design, interaction closure, and full implementation. Together, both tasks show that long-horizon coding can significantly improve development efficiency, especially in complex contexts and cross-module tasks; but it does not mean final responsibility can be automated, and permissions, security, business correctness, and delivery acceptance must still be overseen by humans.
Therefore, a more mature way to use it is not as an “automatic delivery machine” but as a “long-horizon development collaborator”: let humans define boundaries, review risks, and confirm acceptance, and let the system expand context handling, accelerate implementation, and shorten iteration paths. Only under this division of labor can the value of long-horizon coding be released consistently.
