How to Evaluate Long-Horizon Coding Assistants: From Working Code to Safe Delivery
A practical framework for evaluating coding assistants on cross-module software work, covering constraint preservation, authorization, end-to-end validation, learning-product acceptance, and the evidence required for safe handoff.
1. The Core Challenge of Long-Horizon Coding: Preserving Constraints as Requirements Evolve
Completing a function, fixing a localized defect, and delivering a cross-module feature are three different types of work. Local tasks usually have clear inputs and outputs, and their results are relatively easy to inspect. Long-horizon tasks may involve API contracts, database schemas, background jobs, frontend interactions, and rendering pipelines at the same time.
These changes rarely fit into a single implementation pass. A data structure chosen early may affect how it is read later, while a problem found during testing may require the original design to be revisited. The key question when evaluating a coding assistant is therefore not how much code it can generate continuously. It is whether the assistant can preserve consistency across modules after repeated changes while retaining the constraints agreed at the beginning.
In practice, completion can be evaluated at three levels:
- Runnable: The application starts and the main flow works.
- Verifiable: Key requirements have corresponding tests or acceptance records, and failures can be attributed to the implementation, data, or environment.
- Deliverable: Permissions, error handling, operational impact, and recovery after changes have been reviewed.
A browser demonstration usually proves only part of the first level. Even a fully passing automated test suite shows only that the covered behavior matches the test expectations; it does not replace a delivery decision.
The following two cases examine different types of work. One modifies an existing system under established rules. The other turns incomplete requirements into a usable product. Both can produce a working demo, but they require different forms of acceptance evidence.
2. Modifying an Existing System: Image Preview Does Not Prove Authorization
The first case adds user image imports to an existing motion-effects system and allows users to reference those images in web-based animation scenes. On the surface, this appears to require only an upload entry point and an asset picker. In reality, the change spans uploading, storage, scene references, preview, and backend rendering.
If any of these stages interprets asset ownership or access permissions differently, the feature may work during a demonstration while failing under real usage conditions.
Map the Data Flow Before Deciding What to Change
Before implementation begins, establish three facts:
- Where does the system identify the current actor?
- How are scenes associated with assets?
- How does the rendering layer retrieve those assets?
It is especially important to distinguish between an API accepting an asset identifier and the current actor being authorized to use that asset. The former is a data-format concern. The latter is an authorization decision. It cannot be replaced by whether the frontend displays an option.
Web preview and backend rendering also require separate review. Preview usually occurs during an interactive user session, while backend rendering may happen long after the original request. The design must define:
- Which identity a background job uses to read an asset
- What authorization rule applies at execution time
- What happens if an asset is deleted or its sharing permission is revoked
- Whether a job should continue using already confirmed input or stop and report an error
These behaviors should not be improvised by the coding assistant. The project owner must determine them according to the product and business rules.
Turn Permission Rules into Observable Acceptance Criteria
In this case, the main flow eventually worked, but manual acceptance uncovered a high-priority authorization-isolation issue. This illustrates that “a user can use their own assets” and “a user cannot use another user’s unauthorized assets” are two independent requirements. Satisfying the first does not imply that the second is satisfied.
A more practical approach is to express the rules as a concise permission matrix and mark each case as allowed, denied, or handled according to an explicit rule:
| Acceptance area | Questions to answer |
|---|---|
| Asset ownership | Can assets belonging to the same actor be imported, associated, and rendered? Are assets belonging to other actors isolated according to the authorization rules? |
| Reference entry points | Do scene creation, existing-scene editing, and scene retrieval follow the same permission rules? |
| Execution stage | Do preview and asynchronous rendering use consistent authorization semantics? |
| State changes | What happens to existing references when an asset is deleted, invalidated, or unshared? Can users understand why the operation failed? |
File validation should be accepted independently. During import, the system should check allowed file types, actual parsing results, and resource-consumption limits rather than relying only on file extensions. Unsupported files should produce clear feedback. A failed import should not leave behind an asset record that appears usable but cannot actually be rendered.
The goal is not to maximize the number of checks. It is to ensure that every rule maps to an observable user outcome and to a state that maintainers can locate and diagnose.
3. Building from Scratch: A Learning Game Must Teach, Play, and Support Completion
The second case provides an AI agent with learning materials and a product requirements document, then asks it to build a knowledge-learning game that players can complete from beginning to end.
This type of project has no legacy constraints, but it is more exposed to misinterpretation. A coding assistant may quickly create pages and answer interactions, yet the number of pages does not prove course coverage, and working answer buttons do not prove that players can complete the learning journey.
Derive Levels and Feedback from Learning Objectives
First organize the source material into concepts, applicability conditions, common misconceptions, and observable learning outcomes. Then decide what each level should test.
For example, consider the question of how an AI agent decides whether to call an external capability. Asking for a definition tests memorization only. Presenting an underspecified task and asking the player to choose between answering directly, requesting missing information, or calling an external capability provides a stronger test. Explaining the rationale for each choice also helps determine whether the player understands the relevant conditions.
Product requirements should likewise be separated into required items, optional items, and items awaiting confirmation:
- Required items should enter the acceptance checklist.
- Optional items should not displace core-flow stabilization.
- Open decisions about completion criteria, retry rules, or progress persistence should be resolved early, because they can change the overall experience.
These decisions should not be handled independently with arbitrary defaults on different pages.
Validate the Player Journey Instead of Checking Pages One by One
A reasonable acceptance process starts with a new state that contains no previous progress. The reviewer then follows the full journey:
- Enter the game.
- Read the instructions.
- Answer questions.
- Review feedback.
- Unlock subsequent levels.
- Complete the final level and reach the completion state.
Failure paths should then be tested:
- Can the player continue learning after an incorrect answer?
- Does refreshing or leaving midway follow the defined progress rules?
- Can repeated submissions create an invalid state?
- Does the final level consistently lead to the completion screen?
This case successfully produced a playable application with a complete primary path. However, “can be completed” does not mean that the knowledge is accurate or that the product is educationally effective.
The first conclusion can be confirmed through flow acceptance. The other two still require manual review: checking whether questions match the original materials, whether incorrect options have reasonable explanations, and whether the concepts and difficulty levels fit the corresponding stages.
Keeping these conclusions separate prevents a successful demonstration from being treated as a substitute for content-quality evaluation.
4. Turning a One-Time Trial into a Reusable Engineering Evaluation
1. Fix the Inputs and Acceptance Criteria Before Work Begins
Provide the coding assistant with the repository entry point, task context, behaviors that must not be broken, and explicit non-goals. Ask it to first describe the existing flow, the proposed modification scope, and any assumptions that require confirmation.
For an existing system, prioritize authorization and compatibility boundaries. For a greenfield project, prioritize the core user journey and completion criteria.
Any assumption that could change the core data model or user-visible behavior should be confirmed before large-scale implementation. The purpose is not to create more documentation. It is to prevent a mistaken interpretation from spreading across multiple modules and causing extensive rework later.
2. Leave Inspectable Results at Every Stage
Cross-module changes can be divided into stages such as:
- Contract and data changes
- Server-side logic
- Interface integration
- End-to-end validation
At the end of each stage, record what changed, how it was verified, and what remains unresolved.
The stages do not need to follow a rigid process. Their value is that reviewers can answer the question, “At which layer is this behavior guaranteed?”
For example, the interface can improve an error message, but it cannot take responsibility for server-side authorization. A unit test can verify progress calculations, but it cannot independently prove that a player can move from the entry point to the completion screen.
3. Track Implementation Cost Separately from Acceptance Cost
Initial implementation time is only one metric. More useful indicators for a team include:
- How much manual clarification was required
- Which gaps were discovered during the first acceptance pass
- How many modules were affected by rework
- How many key requirements ultimately had reviewable evidence
When comparing different working methods, keep the task scope, repository starting point, and acceptance cases as consistent as possible. Time spent on tasks of different complexity cannot be compared directly. Lines of code and the number of added tests should not be used as substitutes for actual completion.
4. Focus Human Sign-Off on What Automation Cannot Answer
Before delivery, clearly separate what has been verified from what remains unverified.
Automation is well suited to repeatedly checking known behavior. Human review should focus on whether the tests omitted important rules and what the system and the user will each see when something fails.
For changes involving access control, data migration, or irreversible operations, reviewers must also confirm the impact scope and the response plan.
If a critical constraint has not been verified, a runnable main flow and a fully green test suite should not lower the delivery bar.
The value of a long-horizon coding assistant lies in moving a cross-module task into a state that people can run, inspect, and take over safely. The most important evaluation question is not “Did it finish writing the code?” but rather:
Which constraints have been proven to hold, and which are still only assumptions made during implementation?
Once a team can answer both questions, it can make informed decisions about which work to delegate to an assistant and where human judgment and final acceptance must remain in control.
