How to Evaluate Long-Horizon Code Agents for Real Production Capability
This article explains how to evaluate long-horizon code agents for real production capability rather than treating them as simple code-completion tools. It covers evaluation tasks that reflect real engineering delivery, core capability dimensions, security and permission checks, the limits of automated testing, quantitative metrics, sandbox execution, staged permissions, and practical guidance for teams.
How to Evaluate Long-Horizon Code Agents for Real Production Capability
In real software engineering, the value of code agents is shifting from “can it generate a code snippet?” to “can it advance a complete engineering task?” A model that only produces example snippets in a single-turn conversation may appear capable. However, once it enters a complex project, it must understand existing code, identify module boundaries, follow security constraints, modify multiple files over time, and ultimately move a task to a state that is runnable, testable, and acceptable for release.
This capability is often described as long-horizon coding capability. It is also the key factor in evaluating whether a coding tool has real production value.
Therefore, evaluating long-horizon code agents cannot be reduced to code-completion quality alone. It should not focus only on static exercises or a single benchmark score. A more reasonable approach is to build an evaluation system close to real engineering delivery. Such a system should examine whether the model can understand complex context, complete cross-module changes under permission, security, and data-consistency constraints, make the project runnable, and pass high-risk review during human acceptance.
1. From Single-Point Generation to Long-Horizon Delivery
Traditional code-generation evaluations often focus on whether a function, class, or short script is correct. These tasks can measure basic syntax ability and local logic ability, but they rarely reflect the engineering difficulty of real projects.
Real projects usually have the following characteristics:
- Complex existing code: Existing modules may have implicit dependencies. Naming conventions, error-handling patterns, and data models may not be consistent.
- Multiple possible modification paths: The same requirement may have several implementations. The agent must make decisions based on architecture, performance, security, and maintainability.
- Frequent cross-module collaboration: A feature often requires changes across interface layers, business logic, data storage, task scheduling, frontend presentation, and logging or monitoring.
- Strong security and permission constraints: When user data, file uploads, resource access, or tenant isolation are involved, functional correctness does not equal security compliance.
- Delivery standards beyond “it runs”: A project must be buildable, testable, deployable, and rollback-capable. It also needs observability and auditability.
For this reason, evaluating a long-horizon code agent starts with redefining “completion.” Completion is not generating code that looks reasonable. It means producing a result that an engineering team can accept: clear task objectives, controlled change scope, reproducible test results, identifiable risks, and executable human acceptance.
2. Evaluation Tasks Should Cover Evolving and Greenfield Scenarios
A complete evaluation set should not contain only one type of task. It should cover two typical modes in real development: evolving an existing system and building a system from zero. These tasks test different capabilities.
1. Evolving Tasks: Testing Understanding of Legacy Systems
Evolving tasks usually require a code agent to add capabilities to an existing project. For example, in a system that already has user data, file storage, access control, and a display interface, the agent may be asked to add secure user resource import and display-scenario reference capabilities.
This looks like a feature addition, but the evaluation points are dense.
First, the agent must understand the existing directory structure, module responsibilities, and data flow instead of blindly adding files.
Second, it must understand existing security constraints, such as how resource ownership is determined, how file types are validated, whether sensitive paths can be accessed, and whether access behavior is logged.
Third, it must ensure that the new capability does not break existing functions and does not introduce new privilege-escalation risks.
This type of task can be expressed with an illustrative acceptance configuration:
task:name: Secure resource import and display referencesconstraints:- Access only authorized resources- Imported files must pass type and size validation- Access events must record subject, resource, and timeacceptance:- Unauthorized access is denied- Legitimate access completes successfully- Malformed input cannot produce executable content- Logs remain traceable
The advantage of evolving tasks is that they reveal whether a code agent truly understands an existing system. If the agent only generates an isolated feature without integrating it with the existing user system, permission system, or logging system, the local code may run, but the overall delivery is still unacceptable.
2. Greenfield Tasks: Testing Completeness of Product Delivery
Greenfield tasks usually provide only requirement documents, learning materials, and boundary descriptions. The code agent is asked to build a fully runnable system from zero.
For example, based on a set of learning materials and product requirements, the agent may need to build a knowledge-learning application that supports a complete progression flow.
This type of task does not focus on complex legacy dependencies. Instead, it emphasizes complete delivery.
The agent needs to decompose requirements such as:
- Who are the target users?
- What is the core learning path?
- How is knowledge presented at each level?
- How is answer state saved?
- How is incorrect-answer feedback designed?
- How are completion conditions determined?
- How is data persisted?
- How are malformed inputs handled?
It must also turn product requirements into a code structure, including page routing, state management, data storage, interaction logic, and test plans.
Greenfield tasks are useful for observing the product-oriented thinking of a code agent. An agent that does not merely produce code snippets should be able to fill in reasonable structure when technical details are incomplete and form a runnable loop.
However, if the agent only generates static pages without state management, data validation, and complete progression logic, it should not be considered a real completion.
3. Supplementary Tasks: Repair, Refactoring, and Test Backfill
In addition to new features and greenfield construction, an evaluation set should include supplementary tasks, such as:
- Fixing an intermittent defect
- Refactoring a tightly coupled module
- Adding tests for a critical path
These tasks further examine the engineering judgment of a code agent.
Repair tasks test whether the agent can locate the root cause from error logs, call chains, and code context.
Refactoring tasks test whether the agent can improve structure without changing external behavior.
Test backfill tasks test whether the agent can identify key branches, boundary conditions, and security paths.
Combining these task types makes evaluation results closer to real development work.
3. Core Capability Dimensions of Long-Horizon Code Agents
When evaluating long-horizon code agents, the final code being runnable is not enough. The evaluation should be broken down into multiple capability dimensions. Each dimension should be assessed through task design, evidence collection, and human review.
1. Complex Context Understanding
The first step in a long-horizon task is not writing code. It is understanding the current state.
A code agent needs to read project descriptions, historical code, interface definitions, data models, configuration files, and test cases, then extract key constraints.
Strong context understanding should show that the agent can:
- Identify which modules must be modified for the current task
- Identify which modules are only dependencies
- Recognize existing naming conventions, error-handling patterns, and logging formats
- Understand implicit security requirements
For example, a requirement such as “users can only access their own resources” usually means resource-ownership validation, not simply returning a successful response.
If a code agent cannot understand context, typical problems appear:
- Reimplementing existing capabilities
- Calling interfaces that do not exist
- Breaking existing data formats
- Ignoring permission constraints
- Creating inconsistent states across multiple modules
2. Cross-Module Progression
Real features are often not single-point changes. They usually progress across layers.
For example, a resource-import capability may involve:
- Receiving a request at the interface layer
- Validating parameters
- Writing to object storage
- Recording metadata in a database
- Updating task status
- Checking permissions
- Referencing the imported resource in the frontend
- Recording logs
Cross-module progression requires the code agent to have a global view. It needs to know where a request enters, how data flows, where state is stored, how failures are rolled back, and how logs can be traced.
An implementation that changes only the interface layer may fail to persist data without a corresponding database change. An implementation that changes only the frontend may create a security risk without backend permission validation.
When evaluating cross-module capability, pay attention to whether the code agent:
- Produces a complete change plan
- Explains the impact scope
- Adds related tests
- Moves the task to a runnable and testable state
3. Runnable and Testable Capability
Whether a code agent can bring a task to a runnable and testable state is an important signal of long-horizon capability.
Runnable means the code can be built, started, and used to complete the main flow.
Testable means critical paths have automated validation, abnormal inputs are covered, and boundary conditions have assertions.
However, testable does not mean that all tests passing is sufficient.
Tests cover known paths. Human acceptance focuses on boundaries that may not be fully covered.
For example, normal user access may pass completely, but the following scenarios may not be covered by automated tests:
- Unauthorized access attempts
- Invalid credentials
- Forged resource identifiers
- Duplicate submissions
Therefore, testability is the foundation, not the endpoint.
4. Security and Permission Constraint Compliance
In systems involving user data, resource files, tenant isolation, and sensitive operations, security is more important than functional correctness.
One of the most common mistakes a code agent makes is ignoring permission boundaries in order to make a feature “run.”
Examples include:
- Querying resources only by identifier without checking resource ownership
- Validating only file extensions without checking actual file content or execution risk
- Returning too much internal information in error messages, leading to sensitive information disclosure
Security evaluation should use a threat-modeling approach, not just functional acceptance.
Key checks include:
- Can a user access resources that do not belong to them?
- Can unauthenticated or low-privilege users call high-privilege interfaces?
- Can resource identifiers be tampered with?
- Can file content contain dangerous content?
- Do logs record key access behavior?
- Do exception paths leak sensitive information?
An illustrative permission-check logic can be expressed as:
can_read(user, resource):if resource.owner == user:return trueif policy.allow(user, resource, "read"):return truereturn false
This pseudocode emphasizes the principle of least privilege: deny by default unless resource ownership or an explicit policy allows access.
In real systems, identity authentication, resource encryption, access auditing, and anomaly monitoring are also needed. However, the core idea remains the same.
4. Human Acceptance Cannot Be Replaced by Automated Tests
Long-horizon code agents can significantly improve development efficiency, but they cannot fully replace human acceptance. This is especially true when permissions, security, data consistency, and final delivery standards are involved. Humans must still retain critical judgment.
The reason is that automated tests usually verify whether known behavior matches expectations. Human acceptance focuses more on whether there are unexpected risks.
For example, a permission-isolation issue may not appear at all in routine functional tests:
- Legitimate users can access resources normally
- Illegal requests are not included in the test dataset
- Test cases do not simulate forged identities or tampered resource identifiers
In this case, all tests may pass, but that only shows that functional paths work. It does not prove that security boundaries are reliable.
Human acceptance should focus on the following areas:
- Permission-model consistency: Resource ownership, role permissions, and tenant isolation should be expressed in one consistent model.
- Minimal exposure of sensitive data: Error messages, logs, and API responses should avoid exposing secrets, internal paths, and user privacy.
- Controllable exception paths: Invalid inputs, duplicate requests, timeouts, and rollbacks should have clear handling.
- Auditable changes: It should be clear who modified which modules, why the changes were made, which dependencies were affected, and whether regression is needed.
- Rollback feasibility: When a new feature causes an incident, it should be possible to restore the previous behavior quickly.
Therefore, the evaluation conclusion should not be “tests passed, so it can be delivered.” It should be: “tests passed, so it can enter human acceptance; after human acceptance, it can be released in stages.”
5. Establish a Quantifiable Evaluation Metric System
To make evaluation results more stable, a metric system is needed. Metrics should not focus only on whether a task is completed. They should also cover completion quality, engineering cost, and security risk.
| Dimension | Observation points | Typical evidence |
|---|---|---|
| Task completion | Whether the core objective is implemented | Feature demonstration, task results, requirement mapping |
| Code correctness | Whether the code passes build and tests | Build logs, test results, regression reports |
| Cross-module consistency | Whether related modules are covered | Change list, interface descriptions, data-change records |
| Security compliance | Whether there are privilege-escalation or data risks | Security test cases, human review records |
| Maintainability | Whether the code follows project structure | Code organization, naming conventions, comments |
| Observability | Whether problems are easy to investigate | Logs, monitoring metrics, alert information |
| Human intervention cost | Whether extensive rework is needed | Revision rounds, manual repair time, rework scope |
In practice, weights can be assigned to each dimension. However, the final conclusion should not depend on a single score.
This is especially true for security. If a high-priority permission-isolation problem exists, the implementation should be considered not directly deliverable, even if all functional tests pass.
6. Sandbox Execution and Staged Permission Granting
Long-horizon code agents have the ability to modify code. Therefore, the evaluation process itself needs engineering governance.
Running high-risk tasks directly in a real production repository is not appropriate. A safer approach is to build an isolated environment and grant permissions in stages.
1. Sandbox Environment
All evaluation tasks should be executed in an isolated environment.
- Code repositories can use controlled copies
- Data can use desensitized samples
- External dependencies can use controllable substitutes
This allows evaluators to observe real task performance while avoiding accidental operations against production data.
Common cloud capabilities such as object storage, logging services, and monitoring/alerting can also be used to collect runtime evidence. For example, test artifacts, build logs, change records, and abnormal events can be archived together for later review and audit.
2. Staged Permission Granting
The permissions of a code agent should be opened gradually from low risk to high risk:
Stage 1: Read-only analysis
Allow the agent to read code, explain call chains, and propose solutions, but do not allow file modification.Stage 2: Local changes
Allow the agent to modify a single file or module, such as adding tests or correcting function logic.Stage 3: Cross-module changes
Allow the agent to modify interfaces, business logic, data storage, and tests together.Stage 4: High-risk changes
For changes involving permissions, security, data migration, or architectural adjustments, human confirmation must be obtained before execution.
This staged approach preserves efficiency while keeping risk within an acceptable range.
7. Key Findings from Practical Evaluation
Based on common performance of long-horizon code agents, several relatively stable conclusions can be drawn.
First, long-horizon code agents are already suitable for trial use in real projects, especially for cross-file modification, complex context understanding, and continuous task progression. Their value is clearly higher than traditional single-turn code generation. They can automate repetitive and mechanical engineering work, allowing developers to focus more on architecture, security, and acceptance.
Second, long-horizon code agents have not yet reached a fully unattended level. They can complete many paths from requirements to implementation. However, in areas such as permission isolation, security boundaries, abnormal inputs, and final acceptance, human engineering judgment is still needed as a safeguard.
Third, evaluation should not rely only on static leaderboard scores. Long-task performance in real projects better reflects the engineering maturity of a code agent. More important indicators include whether the agent can progress across modules, maintain context consistency, produce testable evidence, and continue fixing after failure.
Fourth, security capability must be evaluated separately. A feature-complete implementation with privilege-escalation risks should not be considered a high-quality delivery. For systems involving user data, resource access, and multi-role collaboration, security testing should be part of the evaluation loop, not an optional add-on.
8. Practical Recommendations for Teams
For teams that want to introduce long-horizon code agents into their development process, the following directions are recommended.
1. Build a High-Quality Task Set
The task set should come from real projects, not fictional exercises.
Teams can start with tasks that have clear boundaries, explicit acceptance criteria, and controllable impact, such as:
- Adding tests
- Fixing low-risk defects
- Refactoring local modules
- Implementing internal tools
As experience accumulates, the task set can be expanded to cross-module features and security-related tasks.
2. Define Human Review Gates
For tasks involving permissions, data deletion, bulk changes, or exposure of external interfaces, human review gates should be established.
The review should not focus only on code style. It should cover:
- Permission model
- Data flow
- Exception handling
- Log auditing
- Rollback plan
3. Preserve a Complete Evidence Chain
Every task execution should preserve an evidence chain, including:
- Requirement description
- Change plan
- Code diff
- Test results
- Human acceptance records
- Rollback plan
The evidence chain improves audit efficiency and helps teams identify where code agents are most reliable and where human reinforcement is still needed.
4. Continuously Improve Security Evaluation
Security evaluation should be updated continuously rather than treated as a one-time activity.
As business changes, permission models, resource types, exposed API surfaces, and threat scenarios will also change. Teams should regularly add security test cases and turn typical risks into evaluation baselines.
Conclusion
Evaluating long-horizon code agents is essentially evaluating whether they can take on reliable delivery in complex engineering environments.
A capable agent must not only write code. It must also understand context, follow constraints, advance cross-module changes, produce testable results, and remain restrained on security and permission issues.
Current long-horizon code agents already have high practical value, especially for long-running development tasks in real projects. However, when permissions, security boundaries, and final acceptance are involved, human engineers must still participate deeply.
The more reasonable approach is neither to mythologize nor to reject these agents. Instead, teams should build a rigorous evaluation and adoption mechanism:
- Use real tasks to test capability
- Use security evaluation to constrain risk
- Use human acceptance to protect boundaries
- Use staged permission granting to improve efficiency
Only in this way can long-horizon code agents become reliable productivity in the engineering system, rather than something that looks efficient but is difficult to deliver.
