Engineering Acceptance for Long-Horizon Coding Tasks
Evaluating AI coding tools on long-horizon engineering work requires more than benchmark scores or single-file completions. This article presents an acceptance approach based on two distinct task types: modifying an existing service with security constraints and building a small product from domain materials. It explains how to compare against a fixed baseline, which process metrics to record, how to test permission and injection risks in legacy changes, how to verify closure and state recovery in from-scratch builds, why security review needs a separate track, and which human gates and cloud capability checks should remain in place before delivery.
Why Long-Horizon Coding Is Harder to Evaluate Than Single-Point Completion
AI coding tools are relatively stable at short function completion and single-file edits, but real engineering rarely stops at “write a function that runs.” More often, the repository already contains historical conventions, cross-module contracts, state machines, permission models, and regression tests. Requirements may need to be advanced continuously over several hours, including reading documentation, changing interfaces, adding migrations, running tests, and fixing regressions. To judge whether an upgrade truly raises the coding ceiling, the question is whether it can keep the goal from drifting in long context and bring changes to a runnable, testable state—not just what average score it achieves on a public leaderboard.
Leaderboards still have reference value: they measure solve rate on a fixed problem set. Engineering acceptance measures something else: whether constraints are followed, whether failures can be localized, and whether deliverables withstand manual spot checks. The two cannot replace each other. If post-training expands the scale and variety of long-task environments, it will usually improve the stability of continuous progress, but it will not automatically eliminate permission, injection, and privilege-escalation defects. Evaluation design must therefore cover both the capability ceiling and the security floor.
Use Two Task Types to Expose the Capability Surface
A single demo project can easily overestimate a model. A more reliable approach is to prepare two clearly different tasks that test “understanding an old system” and “turning materials into a product.”
The first type is existing-system modification. Choose a service with historical code, clear module boundaries, and security constraints, and add only one narrow capability—for example, “user-uploaded image assets enter controlled storage and are referenced by page animation scenarios according to an allowlist.” The task must touch several parts of contracts, persistence, asynchronous task state, APIs, backend processing, and frontend rendering, forcing the model to understand existing naming, error codes, and permission models instead of building a parallel implementation.
The second type is from-scratch delivery. Provide only domain learning materials and a product requirements document, and require a small product that can complete its main path end to end—for example, a completable knowledge-learning mini-game with levels, grading, progress, and failure feedback that all run. This tests requirement decomposition, information architecture, and the minimal closed loop, not memory of old code.
Both task types require the final state to be reproducible locally or in a container environment: dependencies installable, configuration explainable, and key paths covered by tests or manual scripts. Delivering only a directory structure that “looks complete” does not count as passing.
Compare Against a Baseline, Not Isolated Scores
When evaluating a new model or AI coding tool, use the same task set and the same acceptance checklist, and compare against the previous baseline. If the baseline can already work continuously across frontend and backend in a monolith or multi-module repository, the new round’s value should appear as less goal drift, less repeated rework, and clearer tradeoffs when constraints conflict—not merely a shorter first-round generation time.
In practice, record four kinds of process metrics rather than inventing performance numbers that cannot be reviewed:
- Time to first runnable state, plus subsequent validation and repair time;
- Number of human interventions, and whether each intervention adds requirements, corrects architecture, or plugs a security hole;
- Whether cross-module changes keep contracts consistent, and whether there are parallel implementations, dead code, or unwired states;
- Whether tests and browser or API smoke checks cover the main path and at least one failure path.
Time itself is not quality. Generating a large amount of code within an hour with the wrong permission model will return multiplied costs during acceptance and after launch.
In Legacy Changes, Watch Constraints, Not Just “The Feature Exists”
Take “secure user image import and reference in animation scenarios” as an example. The feature list is easy to complete superficially: the upload API returns success, the page displays the image, and the animation timeline can reference the URL. Acceptance should go deeper, to whether constraints hold.
For permission isolation, check at least these points: whether uploader, resource owner, and scenario referencer are separated; whether references are allowed only to reviewed objects or objects owned by the current tenant; whether object identifiers can be enumerated or guessed; whether temporary URLs are bound to expiry and origin; and whether references become invalid after an asset is deleted rather than leaving dangling addresses. A common hazard in legacy code is the old assumption that “logged-in users can read the public bucket” being spread unchanged into the new path.
Input and output must also be examined separately. File type, size, pixel dimensions, and content detection belong at ingress; URL concatenation in HTML or style fragments belongs at egress. Checking only suffixes at ingress while egress still concatenates user-controlled strings into markup turns “asset import” into an injection surface. If animation scenarios allow custom fragments, use allowlisted attributes and escaping rather than trusting model-generated markup wholesale.
Illustrative acceptance assertions can be written as test intent, without binding to a specific vendor API:
given user A uploads an image authorized only to tenant T1when a scenario in tenant T2 tries to reference that object identifierthen the reference is rejected, and the response does not leak the storage pathgiven an asset is deleted or rejected in reviewwhen a published scenario renders againthen it does not fetch a guessable address and degrades to a placeholder
A first-round run only proves the main path is connected. Permission issues often surface as high-priority defects during manual acceptance: if automated tests assert only “upload succeeds, page has image,” they will systematically miss privilege escalation. The pass criterion for a legacy task should therefore be “all security constraint cases pass,” not “the feature demo can be clicked through.”
From-Scratch Builds: Look for the Closed Loop, Not Page Count
Materials for the second task type are usually scattered concept notes plus a requirements document. A qualified implementation should first map concepts to decidable learning objectives, then build a completable loop: reading or hints, answering, judging, feedback, unlocking the next level, and recording progress. Completion is not pasting document paragraphs into the interface; each level must have auto- or semi-auto-judged criteria, and failure must point out which rule is missing.
Review should focus on four things. First, whether requirements coverage is traceable, with each main requirement pointing to a level or module. Second, whether state is recoverable, with progress not lost after interruption and rewards not issued twice. Third, whether content and code are separated, so the question bank can change without changing the grader. Fourth, whether error handling is honest, with missing materials, invalid answers, and timeouts having explicit branches instead of placeholder copy hiding unimplemented behavior.
From-scratch projects are more likely to “look complete”: the directory is full, the copy is professional, and there may even be test files, but the grader always returns pass, or levels have no real dependency. Acceptance must personally walk through one pass path and one failure path, and spot-check progress storage.
Security Capability Needs Its Own Track, Not Folded Into “Can Write Code”
Progress in code capability does not mean vulnerability reasoning is inspection-free. A more reasonable positioning is this: the model can participate in white-box reading and help list trust boundaries, dangerous concatenation, and missing validation, but full exploit-chain validation, fix confirmation, and release sign-off still belong to humans and existing security processes. Basic checks can approach the depth of daily code review—for example, identifying hardcoded secrets, overly broad authorization, insecure deserialization, and queries missing tenant conditions. Once the work involves combined exploitation, bypassing multiple validation layers, or judgments tied to production configuration, model conclusions should not be treated as evidence.
Open weights, private deployment, or continued training only change the control surface for data and environments; they do not cancel acceptance. After self-deployment, teams should still keep dependency and license review, output filtering, least privilege for tools, secrets out of context, and isolation between production and experimental environments. Cybersecurity experiments must be conducted within an authorized scope, with the goal of hardening and regression, not extending generation capability into attack steps.
Human Gates That Cannot Be Released
The collaboration process can be reduced to three gates.
The first is the task brief. Write down scope, non-goals, inviolable invariants, and acceptance cases. Invariants must be concrete—for example, “cross-tenant object references are prohibited” and “user input must not enter executable markup.” Vague instructions such as “pay attention to security” will rarely be followed consistently.
The second is the runtime gate. Each round should provide only the directories, documentation, and read-only or controlled write permissions needed for the current subgoal; test and staging credentials should be separated; generation commands should first run in an isolated environment. In long tasks, the model will actively expand the change surface. The gate’s role is to turn that expansion into a reviewable diff, not to accept it by default.
The third is human acceptance. Even when all tests are green, sample by risk: permissions, injection, migration reversibility, failure retry, and idempotency. High-priority findings should be written back as permanent cases so the next long task does not drop them again. All-green tests only mean the written assertions hold; they do not mean the assertions cover the threat model.
Choosing General Cloud Capabilities for Delivery
Long-horizon coding output often needs object storage for assets, message or task queues to decouple processing, a container platform to run rendering or grading, and logs and monitoring to replay failures. These are general capability categories. Selection should look at whether the permission model, lifecycle, audit, and network isolation fit existing systems, not whether one model demo architecture can be copied directly into production. If a team uses Baidu AI Cloud for experiments or staging, bucket policies, queue permissions, and runtime identities should also be written into acceptance items, avoiding the situation where “the code is correct but the cloud policy is too broad.”
Do not carry temporary keys, permissive CORS, and public-read configurations from the experimental environment into formal delivery notes. The delivery checklist should include at least: how to build, how to configure non-production dependencies, which cases must be manually rerun, and what the known limitations are.
Conclusion: Worth Testing with Real Projects, but Not Acceptance-Free
Whether long-horizon coding has improved should be answered with two task types: cross-module legacy modification and from-scratch closed-loop delivery. The signs of progress are the ability to read complex context, keep changing the system until it is runnable and testable, and make fewer hidden assumptions when constraints conflict. The signs that it is not yet established are equally clear: permission isolation, injection surface, and final business acceptance still expose high-priority issues, and all-green tests cannot replace human sign-off.
Teams can turn these two task types into an internal baseline and compare only process records and defect types each round, rather than chasing unreviewable score narratives. This evaluates engineering capability, not the completeness of a single demo.
