Long-Horizon Coding Evaluation: Open-Source LLMs Reach a New Step Forward

    This article evaluates long-horizon coding capability in next-generation open-source large language models through two complementary engineering tasks: extending an existing system with security-sensitive features and building a complete, playable learning game from scratch. It compares the results with the previous generation’s baseline, explains why the main improvement is expected to come from expanded post-training environments, and summarizes what can be delegated to the model and what still requires human review.

    1. Why single-point coding benchmarks are no longer enough

    In the past, coding ability was often evaluated with single-point questions: an algorithm problem, a function-completion snippet, or the fix for a single error. These metrics can show whether a model can write code, but they do not show whether it can deliver a functional feature. The hard part of real engineering is not syntax at a single point, but consistency along a long chain: whether requirements can be fully decomposed, whether cross-layer changes keep contracts consistent, whether failures can be localized and resolved, and whether the final artifact is runnable and testable.

    For next-generation open-source large language models, a more meaningful dimension is long-horizon task completion. It includes four observable subquestions:

    1. Context understanding: Can the model read the existing code’s layering, naming conventions, and implicit constraints, rather than introducing a new style?
    2. Cross-module progression: When a change involves data contracts, persistence, task state, APIs, and frontend rendering, can it remain end-to-end consistent?
    3. Failure convergence: When compilation errors, test failures, or runtime exceptions appear, does it blindly rewrite, or make evidence-based minimal fixes?
    4. Verifiable delivery: Does it proactively add tests, logs, error handling, and migration scripts, moving the result to a state where others can verify it?

    The evaluation below is designed around these four points.

    2. Evaluation method: two complementary tasks

    To cover two very different engineering forms—legacy-system modification and product-building from scratch—I prepared two task groups.

    Task A: Feature extension in a codebase with legacy baggage. In an existing project with a substantial amount of historical code, I asked the model to add two capabilities: secure import of user-uploaded image assets, and HTML motion scenarios that can reference those assets. The key constraint was “do not break existing behavior”: coding style, layering, and error-handling conventions had to be preserved. At the same time, a security requirement was added: user-uploaded content is untrusted, and rendered content may carry scripts.

    Task B: A complete product built from zero to a playable level. I provided only learning materials about Agent-related topics from a public technical knowledge base, plus a short product requirement note, and asked the model to implement a knowledge-learning game that could actually be completed. The task did not prescribe a technology stack. It assessed the completeness of material analysis, knowledge structuring, level design, state progression, progress persistence, and the playability loop.

    The two tasks are complementary. Task A tests “incremental work under existing constraints”; Task B tests “system building under blank constraints.” The former tends to expose misreadings of context and security blind spots; the latter tends to expose insufficient product completeness and lack of self-verification.

    3. Baseline: where the previous generation stood

    To judge the magnitude of the improvement, there must be a clear baseline. Earlier, I used the previous generation of the same model family for a non-trivial development task: adding an HTML motion-rendering path to a cross-stack video generation toolchain. The change spanned data contracts, a relational database, a task state machine, an API layer, a Worker, and a web frontend.

    The result was: the first implementation took about 1 hour and 11 minutes, and browser-side verification plus defect fixes took another about 1 hour and 4 minutes. The data itself is informative: the implementation phase was already efficient, but the real cost was in “verification and repair.” In other words, the previous generation was already capable of working continuously in large multi-repository projects and handling frontend-backend linked features. The bottleneck had shifted from “can it write the code?” to “can it get it right the first time, and fix it when wrong?”

    The improvement of the new generation should be viewed against this baseline, not simply compared with much earlier models.

    4. Source of improvement: expanded post-training environments under the same base

    According to official disclosure, this upgrade did not replace the base model: it still uses a base with the same parameter scale as the previous generation. The improvement mainly comes from scaling the environment scale in post-training—the number of long-horizon task environments used for training expanded to dozens of times the original number, the environment types became more diverse, and the training duration increased significantly.

    This has a clear engineering implication: when the base model’s language and reasoning capability remain unchanged while long-horizon task environments scale substantially, the benefit should concentrate on multi-step task stability. Concretely, this means less context drift, fewer cross-file contract breakages, and more accurate mid-state self-checks. This directly corresponds to the shortcomings exposed by the previous generation.

    The official summary of the changes has three directions, each worth understanding separately:

    • Coding capability: positioned as the strongest tier of code ability among current open-source models. In several third-party long-horizon coding and real software engineering evaluations, it maintains an open-source lead, with the focus shifting from “solving problems” to “production-grade software engineering.”
    • Cybersecurity: capability extends from “generating code” to “white-box code review and vulnerability reasoning.” It is close to the frontier closed-source level on basic security tasks, but still lags in complete vulnerability exploitation chains.
    • Open-source openness: full weights will be released after security evaluation and model hardening, making it easier for teams to deploy privately, verify independently, and make secondary improvements.

    For users, the third point is often underestimated. Weight availability means teams can continue training with domain data, run the model in an isolated network, and keep sensitive code from leaving the domain. These are difficult to replace with a closed-source API-only model.

    5. Test 1: security-focused modification in an existing system

    In Task A, the model’s performance fell into three layers.

    The first layer is context adaptation. It did not introduce a new framework or directory convention. Instead, it first located the existing “upload—storage—reference” skeleton within the current layering, then inserted new logic using the original error-return structure and dependency-injection style. This matters more than code correctness alone: in legacy projects, stylistic consistency directly affects future maintenance cost.

    The second layer is the quality of security constraints. Facing the combination of “untrusted user upload + a rendering context that can be executed by the browser,” it decomposed the protection into multiple steps instead of performing only a single extension check:

    1. Flow: receive file
    2. -> Enforce size and total pixel limits; reject decompression-bomb-like abnormal inputs
    3. -> Read file headers to detect the actual type; do not trust the client-declared type
    4. -> Fully decode with a restricted decoder; verify that the declared format matches the actual format
    5. -> Normalize and re-encode; strip metadata and embedded payloads
    6. -> Store with a randomized key; prevent user-controlled paths from participating in disk path concatenation
    1. Rendering-side constraints (policy points):
    2. · Keep motion-scenario templates separate from user assets; templates do not participate in user-content concatenation
    3. · Asset references use a fixed allowlist of protocols and domains; block loading of external resources
    4. · Responses carry a randomized script allowlist policy; disable inline script evaluation
    5. · Use safe attachment response headers for download/preview to avoid type-sniffing parsing ambiguity

    This implementation shows that the model can translate security requirements into executable control points, rather than stopping at prompt-style slogans.

    The third layer is behavioral compatibility. It kept backward compatibility for existing callers: new fields have default values, old interfaces are not forced to change, and the database evolves incrementally rather than being rebuilt. This keeps regression costs controllable.

    However—this implementation passed automated tests but was still judged to have a P1 privilege-isolation defect during manual acceptance: access to some resources only checked “whether it exists” and hid visibility in the frontend, without a server-side mandatory check on resource ownership. In other words, if the resource identifier was replaced with another person’s identifier, the server still returned the content. This kind of problem is not covered by ordinary functional tests because it is completely correct on the “normal path.”

    1. Missing critical layer (resource-level authorization):
    2. Read resource -> Verify resource exists -> Verify ownership/shared relationship -> Authorize and return
    3. ^ The missing step in this test

    6. Test 2: building a completable learning game from scratch

    Task B better demonstrates “product completeness.” The model did not simply pile up features. It first abstracted the learning material into a knowledge structure, then mapped that structure into levels and judgment rules, forming a closed loop of “concept → practice → feedback → completion”:

    • Content structuring: It divided the material into three entity types—knowledge points, prerequisites, and common pitfalls—to provide a basis for level ordering.
    • Levels and checks: Each level had a clear success condition and automatically checkable items, avoiding “looks completed but cannot be verified.”
    • State progression: It used explicit state to describe player progress, supporting retries, tiered hints, and progress persistence, ensuring that the game could be resumed at any time.
    • Completability verification: It implemented a full walkthrough path from entry to ending, rather than stopping at several independent pages.
    1. Example level state:
    2. level_id Current level
    3. attempts Number of attempts, used to dynamically adjust hints
    4. hint_level 0 no hint / 1 directional hint / 2 key answer points
    5. checks List of completion conditions; all must be satisfied to unlock the next level
    6. completed_at Completion timestamp, used for statistics and review

    It is worth noting that the biggest risk in a “build from scratch” task is not the inability to write features, but the model’s tendency to self-verify success: once it runs through the main path it designed, it may consider the task deliverable. In actual acceptance, humans still need to add three things: the correctness of the knowledge content, the reasonableness of the difficulty curve, and the robustness of non-main paths such as mid-game abandonment, repeated attempts, and cross-session continuation.

    7. Conclusion: what can be trusted and what cannot

    Combining the two projects, my conclusion is layered.

    What can be handed over to the model:

    • Skeleton implementation and first-version integration of cross-layer features.
    • Incremental changes that follow existing conventions, including database evolution, API extension, and frontend linkage.
    • Completion of supporting engineering such as tests, logs, and error handling.
    • End-to-end construction of medium-complexity applications from requirements and source materials.

    What cannot be handed over:

    • Permissions and multi-tenant isolation: functional tests cannot cover unauthorized paths. Verification must be done on the server side using a permission matrix and reverse use cases.
    • Confirmation of security boundaries: entry points such as uploads, rendering, external links, and deserialization need independent security review. They cannot rely on the implementer’s self-judgment.
    • Final acceptance: knowledge correctness, product reasonableness, and robustness of non-main paths all belong to human judgment.

    Based on this, my engineering recommendation is to establish three gates:

    1. Task-slicing gate: Split long tasks into independently verifiable units. Each unit should include acceptance criteria, avoiding one-time generation of large changes that are hard to attribute.
    2. Evidence gate: Require every change to include a reproducible verification method, such as tests, scripted walkthroughs, or log excerpts. “Should be fine” is not an acceptable conclusion.
    3. Human red-line gate: Any change involving authentication, data isolation, key handling, or destructive operations must enter human review, with corresponding unauthorized-access and boundary test cases.

    Finally, one methodological suggestion: when evaluating this class of models, focus less on aggregate scores and more on three observable indicators—the usable ratio of the first implementation, the number of rounds needed for the model to converge after failure, and the severity distribution of defects found during manual acceptance. These three are closer to answering “can this model enter my delivery process?” than any single leaderboard.