How to Evaluate Long-Horizon Coding in Code LLMs: Two Hands-On Tasks
When a new code model lands, the easiest mistake is treating a leaderboard ranking as an engineering verdict. Leaderboards answer whether a model can solve a single problem; real delivery asks whether it can finish an entire piece of work. This article examines how to evaluate long-horizon coding through two deliberately contrasting tasks—extending a legacy system under security constraints, and building a game from scratch—and shows what each reveals. It breaks long-horizon capability into three observable layers, explains why post-training-driven upgrades demand long-horizon evaluation, and documents a P1 permission-isolation flaw that fully green automated tests still missed. It closes with a reproducible six-step evaluation process and one takeaway: green tests are the starting point of engineering, not a delivery permit.
1. Conclusion First: Judge a Code Model by Three Things, Not by Scores
When people discuss a new generation of code models, the most common mistake is treating a leaderboard ranking as an engineering conclusion. Leaderboards answer “can it get one problem right,” while real delivery cares about “can it finish one thing.” The gap between the two is exactly what long-horizon coding is meant to fill.
My conclusion can be compressed into three sentences:
- It is worth bringing into real projects. Especially for long-horizon changes that cross modules and span frontend and backend, the output quality has cleared the “good enough to look at” bar and entered the range of “can modify, can run, can keep iterating.”
- It is not a one-pass delivery tool. In an old project I refactored that carried legacy code and security constraints, the model’s implementation passed all automated tests, yet manual acceptance exposed a P1-level permission-isolation defect.
- Capability boundaries must be managed explicitly. Functional correctness and cross-module consistency can be handed to the model; the permission model, security boundaries, compliance constraints, and final acceptance cannot be let go of.
To judge whether those three statements hold, first be clear about what “long-horizon capability” is actually made of. I break it into three observable layers:
- Context understanding: Can it read the implicit conventions of an existing system—how the data model is organized, which state-machine transitions are legal, how errors propagate upward—rather than only looking at the few files being changed?
- Persistence of progress: Can it keep the goal from drifting across dozens of files and multiple process roles, and complete the downstream adaptations that one change triggers, instead of leaving half-finished interfaces behind?
- A verifiable end state: Can it bring the task to a “runnable, testable” state—that after the change, the system starts, key paths are reproducible, and failures can be localized?
The third layer is the most easily underestimated. Many models can write code that looks reasonable but cannot construct a verification loop on their own, so a gap opens between “implementation complete” and “feature usable” that a human has to fill. The real dividing line of long-horizon capability is how long that gap is.
2. Before You Evaluate: Understand What a Model Upgrade Actually Changed
The scenario where evaluation most easily fails is measuring an upgrade with the wrong expectations. So before starting, I first answer a prerequisite question: relative to the previous generation, where does this generation’s capability gain come from?
Take the kind of code-model iteration I have observed as an example. One typical and noteworthy route is: the base model is not replaced; the capability gain comes mainly from post-training. Concretely, the scale of the training environment for long-horizon tasks was significantly expanded (an order-of-magnitude increase), the environment types became more diverse, and the training duration increased accordingly. The implication of this route is direct—the model’s foundational knowledge and language ability did not jump; what changed is its behavioral strategy in multi-step tasks for “staying on goal, self-correcting, and calling tools.”
Such upgrades are usually summarized by the official sources into three directions, each of which directly affects evaluation design:
- Code and software engineering: moving from “generating snippets” to “long-horizon and production-grade engineering,” emphasizing sustained work within real repository structures.
- Security and code auditing: extending capability from writing code to white-box review and vulnerability reasoning—approaching frontier closed-source levels on basic security tasks, but still showing a clear gap on complete exploit chains.
- Openness: full weights are usually released after security and hardening evaluations are complete, making self-hosting, verification, and secondary improvement easier.
There is a key inference here: if the gain comes mainly from post-training and long-horizon environments, then the evaluation must be long-horizon too. Testing it with single-file function problems is like judging a long-distance runner by a 100-meter sprint—not that it can’t, but that you measured the wrong dimension. Meanwhile, the “close to frontier on basics, gap on complex chains” characteristic of security capability is itself a hint: security-related acceptance cannot be reduced to “asking the model whether there is a security problem.”
3. Task Design: Why Two Opposing Projects Are Necessary
A single task cannot separate “capability” from “luck.” My approach is to pick two deliberately opposing tasks as a contrast:
Task A: Extending an existing system. The goal is to add the ability to “securely import a user’s image assets and reference them in an animation scenario” to a project that already carries a fair amount of legacy code. The value of this task is that it applies three kinds of pressure at once: the existing conventions of legacy code, a complete cross-module chain, and explicit security constraints.
Task B: Building a deliverable app from scratch. Given only a piece of learning material (teaching content on agent concepts, tool calling, and task orchestration) and a requirements document, implement a knowledge-learning game that can actually be played through to the end. This task tests the full arc from digesting material and designing mechanics to shipping a product—there is no legacy code to lean on and no prior conventions to copy.
The contrast between the two can be summarized like this:
| Dimension | Task A (existing system) | Task B (from scratch) |
|---|---|---|
| Main challenge | Understanding implicit conventions, avoiding breaking existing behavior | Resolving requirement ambiguity, inventing a sound architecture |
| Security pressure | High (permissions, input, storage) | Medium (its own data boundaries) |
| Verifiability | Depends on existing tests and manual acceptance | Depends on “beatable to the end” as the end condition |
| Failure mode | Function correct, boundary broken | Features complete, experience fragmented |
There is also a practical reason for choosing these two: their time costs differ greatly, but their failure modes are complementary. If both tasks pass, it means the model holds up in both directions—reading what others wrote, and organizing from zero.
4. Task A in Practice: Passing Tests Is Not Passing Acceptance
In this legacy project, the chain to complete is fairly full: data contract definition, asset storage, database fields and indexes, status transitions for the import task, backend interfaces, a background worker process, frontend upload and selection interactions, and finally the reference from the animation renderer. If any link is missing, the feature is unusable.
On an earlier generation of models, I did a comparably demanding refactor of the same kind: a single animation-video generation chain touching contracts, the database, task state, interfaces, the worker, the frontend, and the renderer took about 1 hour 11 minutes for the first-pass implementation, then another about 1 hour 04 minutes for browser-side verification and fixes. That data is itself informative—fast to implement, slow to verify—with verification and fixing taking almost as long as the implementation.
This generation’s improvement shows precisely on that curve: it can keep pushing forward within a monorepo structure, complete feature integration across frontend and backend, and handle downstream adaptations of its changes (for example, the interface response structure after adding a field, the frontend consumption logic, and the task-state enum synchronization) without me repeatedly reminding it.
But the genuinely valuable finding came in the manual acceptance stage. All automated tests were green and the feature demo worked, yet while checking the permission matrix line by line, I found the following problems:
- Unauthorized reads: the asset-read interface only checked “is logged in,” not “has permission to access this asset”—anyone who knows a resource identifier could pull someone else’s asset.
- Loose path and naming handling: uploaded filenames were not fully sanitized, leaving risks of directory traversal and absolute-path writes.
- Missing signature-address validation: generated access URLs did not validate expiration time or signature consistency, so a link stayed valid indefinitely once leaked.
- Missing auditing: cross-subject access left no log entry, so it could not be traced after the fact.
This is a textbook P1 problem: the functionality is completely normal, but the attack surface is wide open. It fits exactly the inference from Section 2—security capability is near frontier on basic tasks, but still has gaps on the “complete chain.” Generating an auth function is not hard; what is hard is recognizing that every data entry point in this system must be an auth point, and making all entry points deny by default.
A unified auth entry point should take this form (illustrative pseudocode):
def load_asset(actor, asset_id):asset = asset_repo.find(asset_id)if asset is None:raise NotFound() # Do not leak whether the resource existsif asset.owner_id != actor.id and not actor.can_view(asset.scene_id):raise Forbidden() # Deny by default, not allow by defaultaudit.record(actor.id, "asset.read", asset.id) # Cross-subject access must leave a tracereturn asset
The upload side, correspondingly, should write its constraints as executable assertions rather than agreements buried in comments:
def sanitize_filename(raw):name = pure_name(raw) # Strip the path portion, reject absolute pathsif ".." in name or name.startswith(("/", "\\")):raise InvalidName()return name
The more important engineering action is this: write security constraints ahead of time as hidden acceptance conditions, making them an input the model can see, rather than something discovered by a human after delivery. I would organize the task input like this:
task: secure_asset_import_and_scene_referencehidden_constraints:- "Reading another user's asset without being the owner must return 403, and the response body must not distinguish 'not found' from 'no permission'"- "Uploaded filenames containing ../ or absolute paths must be rejected"- "Access URLs must validate both expiration time and signature consistency"- "Any cross-subject access must be written to the audit log"exit_criteria:- "All automated test cases pass"- "Unauthorized access cases return 403"- "Expired or tampered signature cases return 403"acceptance_owner: human # Explicitly mark the step that cannot be handed off to automation
The last line is the practice I consider most worth promoting: explicitly marking in the requirements which acceptance items are not allowed to be self-assessed by the model. Permission-, security-, and compliance-related items belong to this category.
5. Task B in Practice: “Beatable to the End” Is a Strong Acceptance Signal
The second task provides only the material and the requirements document. The model has to complete the mapping from concept to gameplay on its own: translate the abstract concepts in the teaching material (agents, tool calling, task orchestration, failure retry) into game mechanics, design the level progression and state machine, implement save, grading, and feedback, and ultimately reach the point where it can be played through to the end.
In this task, “beatable to the end” is a severely underestimated acceptance condition. Unlike “the page opens,” completing the game requires: all state transitions to be legal, no deadlock levels, grading logic consistent with the teaching goals, and recoverable failure paths. In essence it is a set of implicit integration tests.
In from-scratch projects, the model’s typical failure modes are three, all of which I observed in mild form in this task:
- Requirement drift: a core mechanism clearly stated in the document gets squeezed out in the mid-to-late stages by a “flashier” mechanism.
- Loss of implicit constraints: a soft constraint like the line in the requirements document “should not let players see the correct answer directly after a wrong answer” tends to be forgotten after the first few levels are implemented.
- Missing state-machine boundaries: non-mainline paths such as end conditions, save recovery, and mid-game exit are under-covered.
The response is not to hope the model gets stronger, but to reshape the input: break the PRD into numbered acceptance items, tag priorities (P0/P1/P2), and require the model to sign off item by item after implementing each one. After doing this, Task B went from “looks playable” to “actually beatable and aligned with the teaching goals.”
6. Turn Evaluation into a Reproducible Process
A one-off subjective impression has no reuse value. I turn this kind of evaluation into six steps:
- Define the task boundary and the acceptance end state: be clear about “what counts as done”; the end state must be observable and determinable.
- Inject hidden constraints: write security, permission, performance, and compatibility requirements as conditions, not scattered across the conversation.
- Limit the scope of change: provide a description of the repository structure and a list of untouchable modules, and observe whether the model refactors out of bounds.
- Layered acceptance: automated tests → manual reproduction of key paths → line-by-line review of the security matrix → architecture and maintainability review.
- Quantify rework cost: record first-pass implementation time, verification-and-fixing time, the number of human interventions, and the reasons for them. This is a harder metric than “looks good.”
- Mark non-transferable items: permission-model design, security-boundary review, data-compliance judgment, and final release decisions—make the responsible party explicit for each.
I fix the scoring dimensions at six: functional correctness, cross-module consistency, verifiability (whether it comes with its own verification loop), security, maintainability, and rework cost. Among these, “verifiability” and “rework cost” are the most often overlooked, yet they best reflect real productivity.
7. Conclusion: Put the Model in the Process, Not in the Acceptance Seat
Across the two tasks, my judgment is that this generation of code models has already crossed the practical threshold for long-horizon software engineering. It can read an existing system, keep pushing cross-module changes forward, and bring a task to a runnable, testable state; from scratch, it can also produce a complete, beatable application. This is work that used to require a human to lead.
But the P1 permission-isolation problem it exposed shows that capability growth is uneven: the ability to generate correct code and the ability to recognize “where something should not be trusted” do not grow in step.
So the reasonable division of labor is: let the model take on long-horizon implementation, boilerplate code, cross-layer adaptation, test completion, documentation, and refactoring; keep permission-model design, security-boundary review, data-compliance judgment, and final delivery decisions on the human side—and lock this boundary into the process through “hidden acceptance conditions + marking non-transferable items.”
One takeaway you can carry away: green tests are only the starting point of an engineering process, not a delivery permit. Whoever can spell out acceptance responsibility clearly is the one who can truly convert a code model’s long-horizon capability into productivity.
