Long-Horizon Coding Benchmarks: How Far Are LLMs from “It Runs” to “It Ships”?
Leaderboard scores only measure standardized coding tasks. This article evaluates long-horizon coding on two practical jobs—adding a feature to a legacy codebase and building a complete learning game from a brief—and shows why “it runs” is not the same as “it is safe to ship.”
Why Coding Benchmarks Should Not Stop at Leaderboards
Every new model release brings a wave of leaderboard screenshots and score comparisons. From day-to-day engineering practice, those scores mainly show how a model performs on standardized exam-style tasks. Real software work is harder because the context is complex, the task is long, and the constraints are numerous.
A more realistic evaluation should look at three dimensions:
- Long-context understanding: Can the model read an existing project that contains a large amount of historical code, implicit conventions, and accumulated technical debt?
- Cross-module progress: Can it keep working in one session across the interface layer, data layer, job scheduling, and frontend rendering, instead of fixing one place and breaking another?
- Delivery completeness: Does the output reach a runnable, testable state, or does it only look finished?
Using this framework, I designed two tasks with a clear difficulty gap to measure how today’s open-source code models actually perform on long-horizon coding.
What This Iteration Really Changed: Same Base Model, Deeper Post-Training
The underlying logic of this kind of upgrade is worth understanding. This iteration did not replace the base model. It still uses the same previous-generation ultra-large base model, on the order of hundreds of billions of parameters. The gains come mainly from post-training scaling, specifically:
- The training environment for long-horizon tasks was expanded by tens of times, covering a much wider range of scenarios.
- Training time was extended, improving stability in multi-step reasoning and state retention.
- The focus moved from “writing correct code” to “writing secure code,” including white-box code review and vulnerability reasoning.
This path points to an industry trend: once base-model capability plateaus, environment diversity and task length in post-training become the variables that separate models on real engineering work. For users, two generations built on the same base may look similar at single-point coding, but the experience gap is obvious when the model must stay on track for tens of minutes and finish a complex task.
Task Design: One Tests Legacy Systems, One Tests Completeness
To make the results distinguishable, the two tasks were aimed at different goals.
Task 1: Add a new capability to a project with historical baggage.
The requirement was to add “secure import of user image assets and their use in HTML animation scenes” to a project that already contains a substantial amount of existing code. The test is whether the model can understand the old architecture, follow existing security constraints, and make an incremental change without breaking current behavior.
Task 2: Build a fully playable agent knowledge-learning game from scratch.
The model received only a set of learning materials and a product requirements document (PRD), then had to complete the full path from material analysis and solution design to implementation. This tests the end-to-end path from information to product: understanding the materials, designing game mechanics, managing state, and judging whether the player has cleared the game.
Task 1: It Ran, but It Did Not Run Safely
On Task 1, the result was functionally working. The model understood how the existing modules were divided, correctly added a processing node to the asset-import path, and made the frontend animation references work. Its ability to keep moving across modules was impressive: there was no need to keep pasting context by hand, and task state stayed coherent across multiple iterations.
The problem appeared during human review: a P1 authorization-isolation defect. The model implemented the main functional path but failed at the boundary. A material-read API did not strictly check resource ownership, creating a risk of unauthorized access. The typical shape of this bug is:
// Defect: checks only that the user is logged in, not resource ownership@GetMapping("/materials/{id}")public Material getMaterial(@PathVariable Long id) {// Missing: ownership check between currentUserId and material.ownerIdreturn materialService.findById(id);}// Correct approach: explicit ownership check and least privilege@GetMapping("/materials/{id}")public Material getMaterial(@PathVariable Long id,@AuthenticationPrincipal User user) {Material material = materialService.findById(id);if (!material.getOwnerId().equals(user.getId())) {throw new AccessDeniedException("Access to this asset is denied");}return material;}
This case is representative: there is a real gap between “functionally correct” and “ready to deliver.” A green main path and passing-looking tests do not mean security boundaries, permission models, and error branches were handled properly. Wherever permissions, data isolation, or injection protection are involved, human review is still an irreplaceable backstop.
Task 2: Completeness from Materials to Product Exceeded Expectations
Task 2 showed a different strength. With only learning materials and a PRD, the model completed:
- Structuring the materials: breaking unstructured learning content into a set of knowledge points that could be assessed.
- Designing the mechanics: a full game loop for generating questions, judging answers, advancing levels, and determining final completion.
- Closing the state loop: a consistent state machine linking learning progress, level state, and completion conditions.
- End-to-end playability: a game that could be played from the first level through to completion, with no broken step.
This suggests that current models are already practical for “from zero to one” completeness. When the requirements are clear and the acceptance criteria are measurable—for example, “can the game be completed from start to finish”—a model can independently deliver a closed-loop product. The implication for teams is that delivery quality is significantly higher on new projects with explicit acceptance criteria and clear boundaries than on ambiguous changes to existing systems.
A Reusable Evaluation Method
From the two tests, I summarize model coding evaluation in four steps:
- Step 1: Test long tasks, not single points. Use cross-module, multi-hour tasks as the main exam and watch state retention and forward progress.
- Step 2: Separate “it runs” from “it can be delivered.” A working demo is only the passing line. Security constraints, boundary conditions, and error branches need their own acceptance checks.
- Step 3: Define measurable completion criteria. As with “can the game be cleared,” turn acceptance into objective judgments and reduce subjective noise.
- Step 4: Focus human review on high-risk surfaces. Permissions, security, data isolation, and money-related logic still need a final human gate.
Conclusion
The coding ability of the current generation is already worth testing on real projects. In long-horizon coding, the ability to keep working does raise development efficiency in a concrete way. On permissions, security, and final acceptance, people still cannot step away. A fully green test suite and something you can confidently ship are still separated by a serious security review.
The practical model is “model output plus tiered acceptance.” Teams building AI-assisted development workflows in the cloud can also use a complete cloud stack for development, testing, and monitoring alerts to make human review a mandatory pipeline checkpoint, so the efficiency gain stays inside a controlled process.
