From Working Demo to Safe Delivery: An Acceptance Guide for Long-Horizon Coding Models
Long-horizon coding models can now carry real projects to a runnable, testable, and reviewable state, but a green test suite is not the same as safe delivery. This guide explains how to evaluate them on legacy security changes and greenfield products, and which release gates for permissions, isolation, migration, and final sign-off must stay with people.
The conclusion first: a new generation of long-horizon coding models can push a real project to a stage that is runnable, testable, and reviewable. Their value shows up more clearly in complex context, cross-module changes, continued repair, and end-to-end closure than in isolated code generation. There is still a wide gap between “the feature runs” and “it is safe to ship.” Permissions, data isolation, security policy, migration, and final acceptance still require human judgment.
This article does not recap leaderboard scores for any specific product. It treats the problem as a reusable engineering question: how do you judge whether a long-horizon coding model is actually fit for a real project? A representative evaluation can be split into two tasks. One adds a security-sensitive capability to an older system with substantial existing code. The other builds a complete, finishable product from source material and a product requirements document alone. The first tests whether the model can understand a legacy system and respect security constraints. The second tests completeness from source analysis and architecture through product delivery.
1. A Leaderboard Is a Floor, Not a Delivery Line
Static evaluations usually share a few traits: clear problem boundaries, few dependencies, explicit verification criteria, and tests written in advance. They can quickly compare models on algorithms, function completion, and local fixes. They cannot answer several more important questions:
- Can the model understand the structure of a project with years of history?
- Can it follow implicit conventions between modules instead of editing only the current file?
- Can it keep moving across databases, APIs, background jobs, frontend pages, and rendering paths?
- After tests turn green, can it still inspect permissions, failures, rollback, observability, and security boundaries?
- When human acceptance is required, can it provide verifiable evidence rather than only a claim that the work is done?
In real projects, requirements are often incomplete, older code may use unusual patterns, tests may cover only the happy path, and security checks span multiple entry points. Evaluation should therefore shift from “can it write code?” to “can it carry a long task to a deliverable state?”
Recent progress in long-horizon models points the same way. Many upgrades do not replace the base model. They enlarge the long-horizon task environment, add environment types, and extend post-training so the model is better at keeping state, reading context, and verifying repeatedly across multi-step work. That is useful for coding, but it also raises the bar for evaluation. It is not enough to see whether the first pass generates code. You also need to see whether the model can locate failures, add tests, review boundary conditions, and recognize decisions it must not make alone.
Capabilities fall into three layers:
- Coding: Can it implement behavior across files and modules while preserving style and project constraints?
- Security: Can it perform white-box review, recognize permission bypasses, treat untrusted input and dangerous output seriously, and propose fixes that can actually be applied?
- Delivery: Can it leave the task runnable, testable, reversible, and auditable, with evidence for human acceptance?
Differences in the first two layers already show up on real tasks. The third is still the part automatic coding most often underestimates. On basic security work, a model may already offer useful review comments. A complete exploit chain, cross-tenant attack path, business permission combination, or final security sign-off still needs a specialist.
2. Two Tasks: Changing a Legacy System and Building a Product from Scratch
To see whether long-horizon ability transfers, design two very different projects.
Task A: Add Safe Asset Import and Scene References to a Legacy Project
The first project already has a large amount of historical code. The new capability is: users can import image assets and reference them in web motion-effect scenes. On the surface this looks like an ordinary upload feature. In practice it contains several risk paths:
- Whether an uploaded image can be trusted.
- Whether file type, size, dimensions, and metadata are validated on the server.
- Whether storage paths are isolated and hard to guess or access without authorization.
- Whether ownership and visibility are checked again when an asset is referenced by a scene.
- Whether a motion-effect scene can inject scripts, external links, or other dangerous content.
- Whether delete, replace, unshare, and expired access share consistent state semantics.
- Whether background jobs, the database, APIs, the frontend, and the renderer follow the same permission rules.
This is a strong evaluation task. The model is not generating a demo from an empty directory. It must find integration points in existing code and change the database schema, APIs, job state, and frontend interaction under constraints that already exist. It has to understand how the old system works before deciding where the new capability belongs.
Task B: Build a Learning Game from Source Material and a Product Requirements Document
The second project provides only learning material on a technical topic and a product requirements document. The model must implement a knowledge-learning game that can be completed from start to finish. The difficulty is not one API. It is the full product loop:
- Extract a knowledge structure from the material instead of producing simple Q&A.
- Map that structure to levels, goals, questions, feedback, and completion conditions.
- Design a state machine and save data so a user can play from the beginning to the end.
- Handle answers, retries, unlocks, scoring, progress recovery, and completion checks.
- Keep content configuration separate from the game engine so questions can be added later.
- Provide reproducible tests and seed data. Completion must not depend on manually editing the database.
- Balance usability, maintainability, and testability.
One task asks the model to change an old system while wearing constraints. The other asks it to go from requirements to a product in a blank project. Together they are closer to real engineering than a single demo page.
3. Legacy Changes: Cross-Module Progress Is Not Enough If Tests Only Turn Green
On an older project, the most valuable behavior is usually not the volume of code. It is the ability to follow one complete path: read the existing layout and conventions, find data models and APIs, understand background job state, change frontend calls, and make the page and renderer work. For a system that includes APIs, a task queue, file storage, and a frontend, that cross-layer progress matters more than single-file completion.
At the functional level, a model can often:
- Recognize module boundaries and existing naming.
- Add data models for assets, scene references, and state transitions.
- Add server APIs and connect them to the existing authentication framework.
- Handle asynchronous states such as pending, available, failed, and rejected.
- Change frontend pages for referencing and preview.
- Add some integration tests so the main flow can run.
The serious risk sits in non-functional constraints.
In a real acceptance review, the warning sign is not a page that fails to open. It is a main path that already runs while human review still finds a high-priority isolation bug. Typical causes include:
- The frontend hides an entry point, but the server does not recheck ownership on every resource access path.
- The scene-reference API checks whether the current user can access the scene, but not whether the asset belongs to the same user or tenant.
- File access depends on an unguessable URL, yet anyone who obtains that URL can still open it directly.
- Tests cover only “the current user references their own asset,” not cross-user, cross-tenant, unauthorized, or deleted objects.
- A background job or rendering path bypasses the shared permission check.
These bugs are dangerous because automated tests create false confidence. Unit tests may be green, and end-to-end tests may be green, while they only prove that the feature works under normal conditions. They do not prove that permissions hold under attack. For uploads, assets, and scene references, at least these checks belong in the engineering bar:
- Enforce authorization on the server. Every resource access must check the tuple of current user, tenant, resource, and action. Do not treat a hidden button or a random URL as a security boundary.
- Do not let the client decide ownership. Asset owner, scene owner, and tenant ID must be derived on the server from a trusted context. Do not trust an owner field in the request body.
- Treat files as untrusted. Validate size, pixel dimensions, content type, and the actual file signature. Re-encode images when needed and strip unnecessary metadata. Strictly allowlist vector or rich content that might carry scripts. Do not emit it unchanged.
- Store only identifiers in a scene. Save an asset ID or a controlled reference. Do not store executable HTML, arbitrary external links, or user-controlled scripts. Resolve the reference on the server at render time and check permissions again.
- Encode and isolate output. Motion-effect scenes should use structured configuration, allowlisted styles, and a safe sandbox. Do not concatenate user input directly into the page. Apply a content security policy to rendered content and restrict scripts, external links, and network requests.
- Isolate storage and keep access short-lived. Use random identifiers and a tenant prefix in object-storage paths. Use short-lived signed URLs, and invalidate them immediately after delete or revoke.
- Audit and revoke. Record create, reference, share, revoke, and delete. If an authorization failure appears, the trail should support investigation and fast revocation.
- Build a negative test matrix. Cover roles, tenants, resource states, and actions. At minimum include signed-out users, ordinary users, owners, non-owners, administrators, deleted resources, pending resources, and revoked shares.
A short pseudocode sketch states the minimum rule:
def authorize_scene_asset(user, scene_id, asset_id):scene = scene_repo.get(scene_id)asset = asset_repo.get(asset_id)if scene is None or asset is None:raise NotFound()if scene.owner_id != user.id or asset.owner_id != user.id:raise Forbidden()if asset.state != "ready":raise Conflict()if not policy.can_reference(asset, scene):raise Forbidden()
This is not a complete design. It makes one point: permission checks must run on the server’s real read and write paths, and they must cover the relationship between a scene and an asset. Checking only that the user is signed in is not enough. Checking only that the scene belongs to the current user is not enough. Writing tests only for the happy path is not enough either.
4. A Product Built from Scratch: A Working Flow Is Not a Complete Product
The second task shows the move from source material to a product. The model must read the learning material and the requirements, then decide the product structure. A sound implementation usually has these properties:
- Knowledge points have a clear structure, and question content is separate from product logic.
- Levels have prerequisites, completion conditions, and a retry path after failure.
- Learning progress is persisted and survives a refresh.
- Completion is explicit and does not depend on hidden actions or hand-edited data.
- Critical flows have automated tests that can go from an empty state to completion.
- Documentation includes how to start the app, run tests, and initialize data.
- The code structure allows later question additions, level changes, and content replacement.
The content layer can look like this:
{"levelId": "agent-basics-01","goal": "Understand the basic method of breaking an agent task into parts","prerequisites": [],"checks": [{"type": "single_choice","question": "Which approach better follows the principle of task decomposition?","options": ["Split goals and dependencies first", "Write the final answer directly", "Ignore failure paths"],"answer": "Split goals and dependencies first"}],"completion": {"minScore": 0.8}}
The structure is illustrative, not a prescription for one implementation. The principle is: knowledge content should be configurable, and the game state machine should be testable. If content is hardcoded into page logic, a question change can ripple through the product. If completion depends on manual steps, the result is not truly deliverable.
On this project, a long-horizon coding model can often produce pages, APIs, data models, and a basic completion flow quickly. What tends to fall short is product completeness:
- Only the first level is a demo, with no full path to the end.
- Source material is extracted thinly, so questions are weakly tied to it.
- Progress recovery, repeated submission, failed retries, and concurrent actions are incomplete.
- Error messages and exceptional states are missing, so users can get stuck.
- Automated tests cover APIs but not the full user path.
- Documentation is missing, so someone else cannot reproduce the environment.
Acceptance should require verifiable evidence of completion: start from the initial state, finish several levels, fail and retry, restore progress, and reach the end. Tests must run automatically. They must not depend on manual clicks or hand edits to the database.
5. Release Gates from “Tests Are Green” to “This Can Ship”
A green test suite only means the known tests passed. It does not mean the system is secure, reliable, or operable. When a model participates in a long chain of development, the tests themselves may omit risk. Before release, use a broader gate:
release_gates:functional:- Main end-to-end flow passes- Critical failures and retries pass- Data migration can be rolled backsecurity:- Role, resource, and action permission matrix is covered- Cross-user and cross-tenant negative cases pass- Uploaded content is treated as untrusted input- Scene output has no script-injection path- Secrets and access credentials are not in the repositoryreliability:- Idempotency, retry, timeout, and rate limits have explicit policies- Logs, monitoring, and audit records can be correlated- Delete, revoke, and expired access behave consistentlyhuman_review:- Product acceptance passes- A security owner reviews high-risk paths- A release owner confirms the rollback plan
The point worth stressing is that high-risk paths must not pass automatically. For ordinary page styling and low-risk utility functions, a model can edit, verify, and open a review on its own. Changes that touch permissions, payments, deletion, data migration, external sharing, tenant isolation, or secret management need a hard human approval gate. A model can assist review, suggest tests, and analyze the attack surface. A person still owns the final signature.
6. Working Practices That Make Long-Horizon Coding Models More Reliable
The stronger the model, the more deliberately it needs to be used. In practice, risk comes down in a few ways:
- Write tasks that can be accepted. Do not say only “add an asset feature.” State the background, data boundaries, API constraints, security requirements, what is out of scope, and how acceptance works.
- Give the model a runnable context. Include a directory guide, start commands, test commands, code conventions, existing module boundaries, and key data models. More accurate context makes cross-module edits more stable.
- Split long work into observable stages. Ask for an implementation plan first, then proceed stage by stage: data model, server, frontend, rendering path, tests, and security review. Each stage should report verification results.
- Limit the runtime environment. Run commands in an isolated workspace, provide only the minimum credentials, block direct access to production data and real secrets, restrict network egress, and keep an operation log.
- Review diffs in small batches. A single review of a huge final change is an easy place to miss a security issue. Review by theme: APIs, permissions, data migration, and frontend rendering.
- Require evidence. That includes test output, reproduction steps, key decisions, known risks, and uncovered items. “Done” is not evidence.
- Move security review earlier. Before implementation, do a short threat model that names untrusted input, permission boundaries, and sensitive data flows. After the feature works, do a white-box check. Do not leave both until just before launch.
- Measure something other than generated lines. Examples include the share of tasks that run on the first attempt, how often a person has to take over, regression defect count, coverage of negative security cases, and release rollbacks.
A task card can look like this:
task:goal: Add asset import and scene references to a legacy systemcontext:- Existing authentication framework and tenant model- Existing object storage and background job pathconstraints:- Every asset access must be authorized on the server- Scenes must not insert user-controlled scripts directly- Access URLs become invalid immediately after delete or revokeacceptance:- The normal same-tenant flow passes- Cross-tenant, unauthorized, and deleted resources are rejected- The full completion path or business flow can be reproduced automaticallyevidence:- Automated test results- Permission matrix and negative-case notes- Known risks and points that need human review
7. Conclusion: Use Them on Real Projects, but Do Not Give Up Delivery Control
Across legacy modification and greenfield product work, a steadier judgment is this: long-horizon coding models are already worth trying on real projects, especially cross-module, long-running coding tasks that need repeated verification. They are not yet fit to be left unattended on permissions, security, and final acceptance.
They fit best when the work is:
- A medium-sized feature under existing tests and clear APIs.
- A path across frontend, backend, data models, and asynchronous jobs.
- Draft tests, extra boundary cases, and assistance with white-box review.
- A fast move from source material to a prototype.
- Local refactoring and consistency fixes in existing code.
People should keep final control when the work involves:
- Permission models, tenant isolation, and access to sensitive data.
- Security boundaries around upload, rendering, rich text, external links, and script execution.
- High-risk operations such as payment, billing, deletion, and migration.
- External release, production deployment, and security sign-off.
- A change that looks fine because tests passed, but has not been threat-modeled.
The practical test is not whether the model can write it in one pass. It is whether it can bring the task to a state that is runnable, testable, reviewable, and reversible, while making the remaining risk explicit. If it can, it is already a productivity tool. If a team treats “tests are green” as “safe to ship,” a more capable model only enlarges the gap in the process.
A sound engineering stance is to treat a long-horizon coding model as a high-throughput pair programmer and review assistant: use it to speed up implementation, verification, and documentation. Keep release gates, permission review, security sign-off, and final acceptance with people. The ceiling of these models is still moving forward, but delivery quality is still decided by whether the team has a complete engineering acceptance system.
