Long-Horizon Coding Model Evaluation: From Task Progress to Security Acceptance

    This article examines how to evaluate long-horizon coding models beyond public benchmark scores. Using two anonymized projects—a security-constrained legacy-system modernization and a from-scratch agent knowledge-learning game—it reviews where these models now sustain cross-module work and implementation-validation-fix loops, and where they still fail. The legacy project passed functional tests but exposed a high-priority tenant/project authorization isolation issue during manual acceptance. The article provides six evaluation dimensions, a reusable engineering workflow, server-side authorization pseudo-code, and acceptance cases, concluding that such models are ready for real long-horizon engineering tests but not for hands-off delivery where permissions, security, data isolation, and final sign-off are involved.

    Evaluating a long-horizon coding model by public leaderboard scores alone is often the least reliable approach. Leaderboards can reflect single-point code generation, tool invocation, or short-task repair, but they rarely cover long-cycle collaboration in real engineering: understanding unfamiliar code, maintaining a plan, making cross-module changes, reading logs, locating regressions, adding tests, and taking responsibility for permission and security constraints. This article does not target any specific product or vendor. It discusses a class of coding models with longer context, code editing, command execution, and tool-calling capabilities. The cases have been anonymized and abstracted; only reusable evaluation methods and engineering conclusions are retained.

    The conclusion comes first: the ceiling of coding capability for this class of models is rising. Especially on long tasks, they can already read fairly complex context, push cross-module changes forward, and bring a task to a runnable and testable state. But they are still far from the point where “all tests pass” means it is safe to deliver. The evaluation used two very different projects: one was a legacy-system modernization with historical code and security constraints; the other was a from-scratch implementation of a fully completable agent knowledge-learning game based only on learning materials and a product requirements document. Both projects could run, but the first exposed a high-priority permission isolation issue during manual acceptance. This shows that such models are now worth testing with real projects, especially for long-horizon coding. But when permissions, security, data isolation, and final acceptance are involved, humans cannot step away.

    1. Why Evaluate Coding Models with Long Tasks

    Short-task evaluation usually focuses on “can it write a function” or “can it fix a clearly defined error.” Real development is closer to another shape:

    • It requires understanding business constraints, historical code, data models, API contracts, frontend interaction, and asynchronous tasks at the same time.
    • Changes are distributed across multiple modules; local correctness does not mean overall correctness.
    • Failures are often not compilation errors, but state inconsistency, missing permission boundaries, and uncovered exception paths.
    • The delivery standard is not “the code looks reasonable,” but “it can run, be tested, be rolled back, and be audited.”

    From a technical evolution perspective, a new generation of models does not necessarily replace the base model. Improvements may also come from post-training-stage scaling: training the model in more types of environments, with longer steps, and closer to real engineering, so that it learns to maintain a plan, handle failures, read validation results, and continue fixing. Public capability descriptions often focus on three areas: complex software engineering, code security review and vulnerability reasoning, and open weights. The first two directly affect engineering adoption; the third affects whether developers can deploy and verify the model themselves. However, open weights do not mean direct production launch. Security evaluation, model hardening, and engineering-side safeguards are still required.

    Therefore, the evaluation focus should shift from “can it write code” to the following six dimensions:

    1. Runnability: After changes, can the system build, start, and execute the main flow?
    2. Testability: Are effective tests added, and do they cover exception and boundary cases?
    3. Change completeness: Do APIs, data, task state, frontend, and background jobs work together?
    4. Constraint compliance: Does it follow existing architecture standards, naming conventions, and compatibility requirements?
    5. Maintainability: Is the code readable, extensible, and rollbackable?
    6. Security: Do permissions, data isolation, input validation, and resource access form a closed loop?

    2. Designing Two Real-World Tasks

    Task 1: Secure Asset Import and Web Animation References in a Legacy System

    The task background is a business system that has run for many years. It already has a large amount of historical code, old APIs, data migration logic, and permission constraints. On the surface, the requirement is to add “image asset import and web animation scene reference” capabilities. In practice, it involves API contracts, storage, task state, permissions, rendering, and auditing.

    The key to this task is not the upload control, but the security boundary. The constraints given to the model included:

    • Allowlists for file type, size, and count; the server performs content sniffing rather than trusting only the extension.
    • Strip executable metadata; reject disguised files and abnormal archive structures.
    • Assets must be bound to a tenant, project, creator, and business purpose.
    • Read, reference, replace, and delete operations must all be authorized on the server side.
    • Resource URLs must be short-lived and bound to the current session and tenant.
    • Paths must be normalized to prevent directory traversal and unauthorized overwrite.
    • The web rendering side must restrict external links, inline scripts, event attributes, and untrusted styles.
    • Critical operations must be written to audit logs and traceable on failure.

    The acceptance criteria were not “the page can upload.” They were: legacy data compatibility, rollbackable migration, testable normal and exception flows, a verifiable permission matrix, and no way for the rendering side to bypass access control.

    Task 2: From Materials and Requirements to a Completable Learning Game

    The second task provided only learning materials and a product requirements document. It required building an agent knowledge-learning game from scratch. The acceptance focus was not whether the page looked beautiful, but whether the complete loop held:

    • Can the learning materials be structured into knowledge points, levels, and questions?
    • Is there a clear learning map, level state, and completion condition?
    • Do Q&A, hints, scoring, and mistake review form a closed loop?
    • Are progress saving, restart, and invalid input handling complete?
    • Is there obvious fabrication or hallucination in the content, and do questions map to knowledge points?
    • Are run instructions, seed content, basic tests, and a maintainable code structure provided?

    This task tests completeness from unstructured materials to product delivery: information extraction, data modeling, state-machine design, frontend interaction, backend APIs, tests, and documentation are all required.

    3. Real Progress Observed

    In the legacy-system modernization task, the model could first read the directory and key code, establish module relationships, and then list the change surface. It could generate local functions and also push changes forward around API contracts, data models, API services, async task executors, the web client, and the renderer. When it encountered compilation errors or state inconsistency, it could read logs, locate related files, and continue fixing. The final artifact could start, execute the main flow, and had some test coverage.

    In the from-scratch learning game task, the model could split the requirements document into entities, states, page flows, and an API list, fill in seed content, implement a completable main path, and provide run instructions. It could adjust the implementation based on errors midway rather than generating once and stopping.

    The value of this progress lies in reducing the cost of cross-module collaboration. In the past, long tasks required frequent human handoff of context. Now the model can maintain a plan over a longer period and complete scaffolding, API wiring, migration scripts, test drafts, and documentation drafts. It does not necessarily get everything right in one pass, but it has an implementation-validation-fix loop. For teams, this means clear improvements in prototype speed, modernization efficiency, and compression of repetitive work.

    But speed is not quality. Especially in legacy systems, the model tends to prioritize explicit requirements and the main flow, while underestimating implicit permission boundaries, exception states, and adversarial inputs.

    4. The Problem: Passing Tests Is Not Passing Security Acceptance

    The first project exposed a high-priority permission isolation issue during manual acceptance. Functional tests could pass, but cross-tenant or cross-project asset reference boundaries were not fully tightened. Typical risks include: resource URLs not bound to a session; read APIs only checking login, not ownership; references not invalidated after deletion; the rendering side being able to reuse another user’s resource identifier; or error messages leaking resource existence.

    These issues are not mysterious, but they are very easy for automated tests to miss. There are usually three reasons:

    1. Permission rules are often implicit in business semantics, and the requirements document may not fully specify them.
    2. Model-generated tests are often same-source as the implementation, covering the main flow but lacking adversarial cases.
    3. Security acceptance requires threat modeling, data-flow tracing, and a permission matrix, not merely “the API returns success.”

    A more robust server-side authorization pseudocode example is as follows:

    1. function open_asset(session, asset_id):
    2. require session.tenant_id is not empty
    3. record = asset_store.find_by_id(asset_id)
    4. if record is null or record.tenant_id != session.tenant_id:
    5. return not_found
    6. require authorize(session, "asset_read", record)
    7. return generate_short_lived_url(record, session, 300 seconds)

    The corresponding manual acceptance cases should at least include:

    1. Case: A tenant A user reads a tenant B asset
    2. Action: Request a tenant B asset identifier using a tenant A session
    3. Expected: Return not found or access denied
    4. Check: Response body, error code, and logs must not leak the tenant B identifier, filename, or storage path
    5. Case: Access an old URL after the asset is deleted
    6. Expected: The short-lived URL is invalidated, and the server denies access

    The conclusion is clear: models can accelerate implementation, but they cannot replace permission design, threat modeling, and final security review.

    5. A Reusable Engineering Workflow

    To use this class of models in production development, the following process is recommended.

    First, turn tasks into acceptance-ready task cards. In addition to feature descriptions, specify non-functional constraints, acceptance commands, rollback methods, untouchable boundaries, and manual review points. Vague requirements amplify the model’s room for free interpretation.

    Second, set stage checkpoints. Do not let the model change all modules at once. Proceed by “design and impact surface—interfaces and data model—security review—minimum vertical slice—integration—regression.” Each stage should require verifiable evidence.

    Third, turn security policy into executable checks. Permission checks must be completed at a unified server-side entry point and cannot rely on hidden frontend buttons. Resource access must be bound to the tenant and session. Logs must not output sensitive paths or credentials.

    Fourth, build an independent acceptance suite. It should cover at least normal flows, exception flows, the permission matrix, concurrency conflicts, migration rollback, legacy data compatibility, and invalidation after deletion. Humans should define the key assertions to avoid full self-certification by the implementation.

    Fifth, preserve observability and rollback capability. Long tasks have a large change surface, so problems must be locatable through logs, metrics, and audit records, and rollback must be possible in stages.

    Sixth, set human gates. Scenarios involving permissions, data isolation, compliance audit, irreversible data changes, and final delivery must have human final acceptance.

    6. Final Assessment

    This class of long-horizon coding models is now worth testing with real projects, especially for cross-module modernization, long-cycle refactoring, requirement-to-runnable prototypes, test draft generation, and documentation organization. It can maintain context over a longer period and move a task from “can write code” to “can run and can be tested.” That is significant progress.

    But it has not reached the point of hands-off delivery. Permissions, security, data isolation, and final acceptance remain human responsibilities. A more reasonable collaboration model is: humans define boundaries, acceptance criteria, and high-risk review; models accelerate implementation, continuous repair, and completion of engineering materials. Evaluating a coding model should not only ask whether it can write code, but whether it can remain continuously correct, verifiable, and maintainable over long tasks—and whether it understands its own capability boundaries.