Evaluating Long-Horizon Coding Agents on Real Projects: A Practical Methodology
This article examines why evaluating coding agents on real projects matters more than relying on benchmark scores, and presents a practical methodology for testing and accepting long-horizon coding agents. It covers two representative task designs—a legacy-system security enhancement and a from-scratch product build—along with five key observation dimensions, a five-step evaluation and acceptance process, and a pseudocode example for permission validation. The article also highlights a common boundary revealed by testing: automated tests may pass while high-priority security isolation issues remain, recommending that humans retain responsibility for security review and final acceptance.
Introduction: Why Evaluation Should Shift from Benchmarks to Long-Horizon Engineering Performance
Early evaluations of coding agents often relied on isolated tasks: patching a function, fixing a syntax error, or generating a short script. These tasks are easy to score, but they do not answer the questions that engineering teams actually care about. Can the agent understand complex context? Can it push changes forward across multiple modules? Can it bring a task to a runnable, testable, and reviewable state?
Long-horizon coding tasks are more representative of real development workflows. A single requirement often spans interface contracts, data models, background jobs, frontend interaction, permission checks, audit logging, and regression testing. The agent must maintain long-range state memory, infer implicit constraints in legacy code, and validate the impact of every change. Public benchmarks can reflect some capability, but they cannot replace the security constraints, historical baggage, and acceptance criteria found in real projects.
A more reliable evaluation approach is to place agents in two distinctly different tasks: one that tests understanding of a legacy system and security-aware modification, and another that tests the full path from requirement analysis to a deployable product. It is also necessary to separate “automated tests pass” from “safe to hand over.” The former is a necessary condition; the latter involves permissions, security, observability, and human acceptance.
Designing Two Evaluation Tasks
The first task is set in an existing project with substantial legacy code: add a capability for secure user image asset import and page animation scene references. The task appears to be feature development, but it actually involves multiple engineering challenges: asset upload type and size limits, tenant and user permission isolation, storage path normalization, content security validation, resource referencing strategies during page rendering, sandbox isolation for animation scenes, and failure rollback with audit logging. Legacy systems often contain historical interfaces, compatibility logic, and implicit conventions. The agent must complete cross-module changes without breaking existing behavior.
The second task provides the agent with study materials and a product requirements document, asking it to build from scratch a knowledge-learning game about AI agents that players can fully complete. This tests a complete chain: requirements interpretation, knowledge modeling, level design, state persistence, completion criteria, frontend interaction, and automated testing. Compared with legacy system modification, this task has fewer historical constraints, but it more easily exposes issues such as “looks runnable, but is not actually playable”—for example, level state cannot be restored, completion conditions are inconsistent, or exception paths are unhandled.
Together, these two tasks reveal whether an agent can both “make safe incremental changes within existing constraints” and “build an acceptable product from scratch.” This combination is closer to enterprise R&D scenarios than a single benchmark score.
Key Observation Dimensions for Long-Horizon Coding Agents
Evaluating a long-horizon coding agent should consider at least the following dimensions.
1. Context understanding and constraint adherence. Does the agent read the relevant modules, configuration files, tests, and documentation? Does it understand the permission model and data boundaries of the legacy system? Long context is not simply “stuffing in more files”—the agent must extract constraints and keep honoring them throughout subsequent changes.
2. Cross-module progress. Is the requirement decomposed into interface contracts, data models, backend executors, frontend presentation, and test cases? Do the changes form a closed loop? The biggest risk in cross-module changes is that each part is locally correct but the whole is broken. Watch whether the agent proactively adds contract tests and regression tests.
3. Runnable and testable state. At task completion, can the project start? Can critical paths execute? Can tests run repeatedly? Note that “all tests pass” only confirms that existing cases pass; it does not mean that permission isolation, abnormal inputs, and final acceptance have been covered.
4. Security and permission awareness. For user asset import tasks, does the agent consider least privilege, tenant isolation, content-type validation, path traversal protection, rendering sandbox, and audit logging? For page animation references, does it restrict script execution, resource domains, and template escaping?
5. Failure recovery and observability. Are there clear error messages on failure? Are key logs recorded? Is rollback or retry supported? In long-horizon tasks, the ability to localize a problem and continue after a failure matters more than generating the correct result in one attempt.
Boundaries Revealed by the Evaluation
In both tasks, the agent was able to bring the project to a runnable and testable state. This shows that long-horizon coding capability is already worth validating on real projects. In particular, cross-module changes and from-scratch product delivery are no longer at a simple demo level.
However, the first legacy project exposed a high-priority permission isolation issue during human acceptance. This is a typical problem: automated tests may cover the normal upload and display path, but not cross-tenant access, unauthorized references, abnormal content types, or rendering isolation. The agent completed the “functional chain,” but the “security boundary” still required human review. This is a reminder: automated tests turning green is only a precondition for delivery, not a substitute for security review and final acceptance.
A clear conclusion follows: long-horizon coding agents are suitable for exploration-oriented implementation, cross-module refactoring, and test completion in real projects—especially long-horizon coding tasks. But when it comes to permissions, security, data isolation, and final delivery acceptance, humans should not step away. The human role should shift from writing code line by line to defining constraints, designing acceptance criteria, and reviewing risks.
A Reusable Evaluation and Acceptance Process
For repeatability, the evaluation and acceptance process can be organized into five steps.
Step 1: Define task boundaries and security levels. Specify which modules can be modified, which interfaces must not be broken, and which data is sensitive. Set human review checkpoints for critical paths related to permissions, security, and billing or other sensitive operations.
Step 2: Require the agent to produce a change description and a risk list. This should include affected modules, new contracts, tests not yet covered, and security points that require human confirmation. This makes hidden risks explicit.
Step 3: Use automated tests to cover both normal and abnormal paths. Beyond unit tests, add cross-module contract tests, permission matrix tests, abnormal input tests, and rendering isolation tests.
Step 4: Conduct human security acceptance. Focus on tenant isolation, least privilege, path normalization, content-type validation, audit logging, and rollback mechanisms. The following pseudocode illustrates the permission validation logic:
function AllowAssetImport(currentUser, currentTenant, file):if currentUser.tenant != currentTenant: rejectif currentUser.role not in [Admin, AssetEditor]: rejectif file.size > threshold: rejectif file.declaredType not in whitelist: rejectactualType = SniffContent(file)if actualType != file.declaredType: rejectnormalizedPath = Normalize(currentTenant, file.name)if normalizedPath escapes tenantDirectory: rejectWriteAuditLog(currentUser, currentTenant, file, action=Import)return allow
Step 5: Build a cloud acceptance loop. In a generic cloud environment, object storage can hold assets, a container platform can run backend executors, a logging service with monitoring and alerting can record audit information, and a secret manager can store sensitive credentials. The focus is not on specific products, but on forming a closed loop of “upload → validate → isolate → render → audit → alert.”
Conclusion: Capable, but with Boundaries
Overall, long-horizon coding agents have demonstrated enough value to be tested in real projects, especially for long-horizon coding, cross-module refactoring, and from-scratch product prototyping. When evaluating them, teams should reduce reliance on public benchmark scores and increase the weight of legacy system modification, security constraints, abnormal paths, and human acceptance.
A more practical collaboration model is to let agents handle verifiable implementation and test completion, while humans retain responsibility for permission models, security boundaries, final acceptance, and delivery decisions. Automated tests passing is an important milestone, but it is not the endpoint. Only by incorporating security review, permission isolation, and human acceptance into the process can long-horizon coding agents become a controllable productivity tool for engineering teams.
