Long-Horizon Coding Model Evaluation: Security Constraints, Full Product Delivery, and Human Review

    This article evaluates a new-generation open-source coding model on two long-horizon tasks: extending a legacy system under security constraints and building a complete agent-learning game from reference material and a product requirements document. Both projects ran end to end, but a P1 permission-isolation flaw in the legacy task showed that human security review remains essential. The article also examines post-training scaling, coding and security capabilities, practical failure modes, and a methodology for assessing long-horizon coding models.

    1. Evaluation Verdict: A Capability Leap With Persistent Risk

    The new generation of open-source coding models—referred to below as “the model”—raises the upper bound of what can be expected from open-source coding systems in long-horizon code generation and complex engineering tasks. Compared with single-task benchmark scores, its long-task performance in real projects is more instructive: it can read complex context relationships, sustain cross-module changes, and carry a task to a runnable and testable completion state.

    However, it is not yet mature enough for a “all tests pass, so ship it” workflow. In this evaluation, we assigned two very different tasks: modernizing a legacy project carrying historical constraints and security requirements, and building from scratch a fully playable agent-learning game. Both projects eventually ran successfully, but the former exposed a high-severity permission-isolation defect during human acceptance review.

    The overall assessment is therefore balanced: the model is already capable of supporting real project development, especially in long-horizon coding scenarios. But human review remains indispensable for permission control, security compliance, and final delivery acceptance.

    2. Evaluation Task Design: Testing Two Dimensions

    To make the evaluation both comprehensive and discriminative, we designed two distinct tasks that examine the model’s performance in “inheritance and constraints” and in “creation and delivery.”

    Task 1: Security-Oriented Extension of a Legacy System

    In a project with a large amount of existing code, add a feature for “secure user image asset import and HTML motion-effect scene references.”

    The core difficulties were:

    • Understand the existing system’s architecture style and coding conventions, and keep new code consistent with them.
    • Identify and follow the existing security constraint mechanisms, especially in high-risk areas such as file upload, path validation, and resource references.
    • Implement the feature within a limited change scope without damaging existing modules.

    This task was designed to test the model’s ability to interpret legacy system logic and its rigor in extending functionality within security red lines.

    Task 2: End-to-End Construction From Material Analysis to Product Delivery

    The model received only agent-learning material based on a mainstream development guide and a product requirements document, and was asked to build from scratch an agent knowledge-learning game that could actually be completed.

    This task involved no existing code, but posed broader challenges:

    • Extract, organize, and restructure a knowledge system from unstructured material.
    • Design reasonable game mechanics and level architecture based on the requirements document.
    • Independently implement the full stack, from data model and business logic to interactive interface.

    With no existing code constraints, this task tested the model’s analytical and synthesis abilities as well as its engineering ability to build a complete product from zero.

    3. Core Change: Continued Evolution From the Training Paradigm

    Previously, we used the prior-generation model for a high-intensity development validation. The task was to add a motion-video generation pipeline to a complex repository containing interface definitions, a database, task states, a server side, background tasks, a frontend interface, and a modular renderer. The first implementation took about 71 minutes, followed by about 64 minutes of browser-side verification and issue fixing. That confirmed the series could work continuously in a complex repository and handle cross-frontend/backend features. The new-generation model builds on that foundation.

    The new model does not change the base architecture; it still uses the same parameter-level base model as the previous generation. The main driver of this upgrade is scaled post-training: the long-horizon task training environment was expanded dozens of times, environment types became more diverse, and the training cycle was extended accordingly. The optimization direction can be summarized at three levels:

    1. Coding capability: Positioned as the strongest open-source version in the series for code capability, it maintains a leading position among open-source models in multiple evaluations targeting terminal operations and software engineering tasks, with stronger emphasis on long-horizon tasks and production-grade software engineering.
    2. Security capability: Its scope extends from code generation to white-box code review and vulnerability root-cause reasoning. It is close to frontier closed-source models on basic security tasks, but still has a gap in complete exploit and penetration scenarios.
    3. Open strategy: Full weights will be released after security evaluation and model hardening, allowing developers to deploy, validate, and build on the model themselves.

    4. Practical Records and In-Depth Analysis

    Scenario 1: Legacy System Modernization—Cautious Progress Under Security Constraints

    Task Execution

    We gave the model a feature request for “secure image import and HTML motion-effect references.” The task involved coordinated changes across nearly four layers: the data layer (a new asset table), the interface layer (upload and authentication APIs), the service layer (file processing and path mapping), and the frontend presentation layer (rich-text motion-effect references).

    The model performed better than expected:

    • Context understanding: Facing an old project with complex structure and sparse comments, the model did not propose a rewrite. Instead, it accurately located the module boundaries where the new capability could be attached and identified the existing authentication middleware and resource routing rules.
    • Convention consistency: The generated code closely matched the project’s original layered structure, introduced clear DTOs (Data Transfer Objects) for parameter validation, and used the project’s customary concatenation approach in the data access layer. Although this was not best practice, it strictly followed the project’s current state.
    • State completeness: The full chain worked on the first pass, from file upload and format validation (only whitelisted extensions allowed), a virus-scan placeholder, to reference path generation and permission checks. The first run successfully generated an accessible URL, and the HTML motion effect displayed correctly.

    Risks Exposed During Acceptance

    Although the feature ran smoothly, human code review found a serious permission-isolation issue:

    • Defect description: When generating the HTML motion-effect reference, the model only performed HTML entity escaping on the variable user-input portions inside a template string, without context-aware JavaScript escaping. Because the motion-effect engine relies on JavaScript to parse user data, a maliciously crafted image name or description could theoretically form an XSS (cross-site scripting) vector.
    • Severity rating: The issue was marked P1—core business affected, with data security and compliance risk.
    • Root-cause analysis: The model focused too heavily on the completeness of the “functional facade” and did not model the “data-flow trust boundary” clearly enough. When data flowed from user-controlled database fields into a high-privilege HTML rendering engine, the model did not build a security policy based on data lineage; instead, it applied a one-size-fits-all escaping approach. In a long task, this appeared as follows: the model remembered the action of “escaping” but forgot where to escape and why the security context required it.

    Scenario 2: Building an Agent Learning Game From Scratch—A Full Test of Completeness

    Task Execution

    This task provided agent-learning material and a product requirements document. The model had to extract knowledge points, plan a course path, and turn them into an interactive game.

    The result went well beyond a technical demo:

    • Knowledge reconstruction: The model organized a core concept chain for agents—perception, decision, execution, and feedback—and designed multiple progressive levels that turned abstract concepts into concrete code-fill exercises, process-correction tasks, and strategy multiple-choice questions.
    • Closed-loop game mechanics: The game included a complete task system, inventory system, score leaderboard, and save mechanism, all driven by local storage without external services.
    • Playability validation: We attempted to complete the game from a beginner’s perspective. The model-generated guidance hints were logically clear and dynamically adjusted the level of feedback detail based on the number of errors. Following the hints, the game could be completed and an achievement badge obtained.
    • Engineering quality: The code used a simple global singleton for state management and introduced no complex framework, yet fully met the game’s needs while maintaining readability. Key modules had concise comments.

    Highlight Analysis

    The success of this task centered on the model’s strong chain-of-thought capability for requirements analysis, task decomposition, and stepwise verification. It could transform a vague requirements description into a project matrix containing user stories, technical choices, and acceptance criteria, and it showed strong self-correction: when a logical conflict appeared in one level’s implementation, it could identify the cause and adjust other modules that depended on that logic.

    5. Methodology: How to Systematically Evaluate Long-Horizon Capability

    Based on the practical evaluation above, we summarized a methodology for long-horizon coding tasks that can help developers verify a new model’s capability boundaries before adoption.

    1. Design adversarial tasks: Do not only test “write a feature from scratch”; test “modify a feature under constraints.” Real-world pain points often occur at the conflict between understanding legacy code intent and following new constraints.
    2. Watch context stickiness: The key metric is not whether the model can generate a given code block, but whether—after generating line 1,000—it still remembers variable naming styles or function boundary conditions from line 10. Use automated scripts to check consistency in its changes and conventions.
    3. Introduce security probes: Plant inherent anti-patterns in the task, such as unsafe file path concatenation, and observe whether the model copies insecure code due to path dependence rather than making security adaptations for the specific context.
    4. Define acceptance-level standards: Do not stop at “it runs.” Build a multi-dimensional delivery checklist covering coding conventions, security scanning, and performance baselines, and require developers—or the model itself—to confirm each item against the checklist.

    Final Thoughts: The Far Side of Long-Horizon Capability Is Still Human-Machine Collaboration

    The long-horizon task capability demonstrated in this evaluation has moved the model from a “code snippet generator” toward a “project-level development collaborator.” It can break large requirements into ordered engineering steps and keep tracking execution. However, it still lacks a kind of adversarial thinking: while pursuing “correct feature implementation,” it can easily overlook “multi-path security defense.” When core assets and user data are involved, experienced engineers must conduct strict security audits.

    The paradigm shift in AI coding has already begun: we no longer ask only “who can write this code,” but also “who can take responsibility for this code.” In the foreseeable future, the most stable production environment will be one where the model handles high-density logic production and cross-module progress, while humans focus on architectural boundaries and red-team testing of security baselines.