Capability Boundaries and Engineering Acceptance for Long-Horizon Coding Agents
Long-horizon coding agents are increasingly able to keep multi-step software work runnable and testable, but “all tests green” is not the same as safe delivery. This article explains how to evaluate them on real projects, what post-training scaling changes, and how permission, security, and human acceptance still set the boundary.
Capability Boundaries and Engineering Acceptance for Long-Horizon Coding Agents
1. Shift the Evaluation Focus from Leaderboard Scores to a Deliverable State
Evaluating a long-horizon coding agent means more than checking a single benchmark score or whether it can emit a snippet that runs. The useful question is whether it can keep a task moving under real engineering constraints: read a complex context, infer implicit rules in existing code, make cross-module changes, and bring the work to a state that is runnable, testable, and reviewable.
Three levels should be kept distinct:
- Runnable: the code starts and the main path works.
- Testable: critical paths have coverage and exceptional branches can be verified.
- Deliverable: engineering constraints such as permissions, security, data isolation, rollback, and audit meet the acceptance criteria.
Current long-horizon coding agents are getting stronger on the first two levels. The third still cannot be left entirely to automation. Humans should not step away when the work involves permission models, security boundaries, or final acceptance. A feature with all tests green is not automatically safe to ship, because the tests themselves may miss privilege escalation, tenant isolation, resource ownership, and failure recovery.
A more realistic evaluation puts the agent into a real project and watches whether it stays consistent across a long task chain, whether it actively recognizes constraints, whether it splits changes into verifiable milestones, and whether it leaves artifacts that can be reviewed and rolled back.
2. How Capability Improves: Post-Training Scale and Long-Horizon Environments
Technically, some upgrades to long-horizon coding agents do not require replacing the base model. They come from scaling the post-training stage: more environments, more environment types, and longer training, so the model learns to advance tasks in long chains that look more like real engineering work.
That improvement usually shows up in three directions:
- Software engineering: moving from single-file generation to long-horizon edits across files, services, and state, with more emphasis on production-grade engineering.
- Cybersecurity: extending beyond code generation into white-box review, vulnerability reasoning, and understanding of security constraints.
- Weight release: opening weights after safety evaluation and model hardening so developers can deploy, verify, and improve the model themselves.
The boundary still matters. Better performance on basic security tasks does not mean the model can complete full exploit development or attack-chain reasoning. Stronger code generation does not replace a security expert’s final risk judgment. Capability gains and responsibility boundaries should be evaluated separately.
3. Two Contrasting Tasks: Legacy Change and Greenfield Productization
To test a long-horizon coding agent, the tasks should contrast. One should stress understanding of an existing system and its security constraints. The other should stress completeness from source material and requirements through a working product.
3.1 Incremental change on a legacy system: safe asset import and HTML motion-scene references
The first task adds “safe user-image asset import and HTML motion-scene references” to a project that already has substantial historical code. It looks like an ordinary feature, but it crosses several constraints:
- The permission model and tenant boundaries already in the codebase
- Format, size, content safety, and storage-path checks for uploaded assets
- Asset ownership, reference relationships, and scene isolation
- Injection risk and resource access control when HTML motion effects reference assets
- Audit logs, failure rollback, and recovery from exceptions
This kind of task exposes real capability. The agent cannot only write the happy path. It must understand default rules in the old system, tell implicit constraints from checks that must live on the server, and cover privilege-escalation cases in tests.
3.2 Greenfield productization: from source material to a completable learning game
The second task provides only a set of agent learning materials and a product requirements document, and asks for a knowledge-learning game that can be completed end to end. It tests full product delivery: extracting material, level design, state machines, scoring rules, save persistence, completion checks, and failure recovery.
The game must not only be playable; it must be completable. That means splitting requirements into verifiable modules: the main flow, retry after failure, progress persistence, scoring consistency, edge conditions, and content safety. For a long-horizon coding agent, the task shows continuity from documents to implementation, but humans still need to accept playability, requirement completeness, and security boundaries.
4. A P1 Permission-Isolation Failure Exposed by the Legacy Change
The most important warning from the original practice is that both projects ran, yet human acceptance of the legacy change found a P1 permission-isolation issue. The functional path worked and tests could pass, but asset or scene references between users still allowed unauthorized access.
The usual root cause is not that the agent “cannot write code.” It is weak upfront understanding of security constraints:
- The permission matrix was never modeled explicitly.
- Tenant boundaries and resource-ownership checks were scattered through business code.
- Tests mostly covered the normal flow and lacked unauthorized-access cases.
- Permission context was lost as state moved across modules.
- The model tended to implement the feature first and treat security as a later patch.
The safer pattern is to check permissions first and deny by default. On asset import and scene binding, the server must enforce user identity, tenant membership, resource ownership, and reference rights. It must not rely on the client or the caller:
function importAsset(user, file):assert user.hasPermission("asset:import")validateMimeAndSize(file)asset = store(file, owner=user.id, tenant=user.tenantId)audit(user, "asset.import", asset.id)return asset.idfunction bindScene(user, sceneId, assetId):scene = load(sceneId)asset = load(assetId)assert scene.tenantId == user.tenantIdassert asset.tenantId == user.tenantIdassert asset.owner == user.id or user.canUse(asset)assert scene.canReference(asset)saveBinding(scene, asset)audit(user, "scene.bind", sceneId, assetId)
The principle is simple: permissions, tenant isolation, and resource ownership must be enforced together on the server and must appear on the test list. Otherwise “it runs” often means only the normal path runs.
5. Value and Risk of the Greenfield Task
Building a completable learning game from scratch shows another strength of long-horizon coding agents. They can turn materials and a requirements document into a modular implementation, generate levels, state machines, scoring, and completion checks, and keep fixing until the main flow runs.
The risks are equally clear:
- Missing requirements: implicit rules in the documents may never be implemented.
- Weak playability: a complete feature set is not the same as an acceptable experience.
- State consistency: saves, retries, and failure recovery can break.
- Scoring and completion checks: edge cases are easy to leave untested.
- Content safety and permissions: learning materials and user progress may also need isolation.
Greenfield productization is a good capability probe. It is not evidence that delivery can be unattended. Human acceptance should focus on whether the main path can be completed, failures can be retried, progress can be restored, scores can be reproduced, and exceptions can be handled.
6. An Acceptance Framework: Before, During, and After the Task
For long-horizon coding agents, a three-stage acceptance framework is a practical baseline.
Before the task: define the task boundary, historical constraints, security requirements, permission model, and acceptance criteria. Unacceptable risks should be written as explicit rules, not left for the model to guess.
During the task: require a change list, a test plan, and a risk note. Advance by milestones instead of one large rewrite. Keep a context summary for cross-module edits. Add static checks, dependency auditing, and observable logs.
After the task: run layered verification: unit tests, integration tests, end-to-end tests, privilege-escalation tests, security regression, and human acceptance. The question is not whether every test is green, but whether the tests cover the critical risks.
7. Where These Agents Fit, and Where They Do Not
Long-horizon coding agents fit incremental work on existing systems, cross-module features, prototypes through MVP, added tests, a first implementation from documents to code, and long tasks that need continuous repair.
They should not be left fully in charge of core permission systems, security-critical paths, final compliance judgments, unattended production changes, or full exploit-chain reasoning. The steadier pattern is agent progress plus human acceptance plus a security backstop.
8. Conclusion: Worth Testing on Real Projects, Not a Reason to Drop Human Acceptance
Taken together, this class of long-horizon coding agent is already worth testing on real projects, especially long coding tasks. It can read complex context, push cross-module changes, and bring work to a runnable and testable state. It is not yet at the point where all-green tests mean the change can be shipped with confidence.
Evaluation should move from “can it write code” to “can it advance work to an acceptable state under constraints.” Where permissions, security, and final acceptance are involved, people still cannot let go. For a team, the sound approach is to prove capability on real tasks, control risk with security-first design and layered acceptance, and treat the agent as an accelerator rather than the final owner of responsibility.
