How to Evaluate the Long-Horizon Coding Capability of Code LLMs
Benchmark scores no longer tell you what a code LLM can actually do on a real project. This article argues that meaningful evaluation should target long-horizon coding—cross-module changes, the implicit contracts buried in existing code, and getting a task to a genuinely runnable, testable state. Drawing on two deliberately contrasting tasks (modifying an existing system and building a product from scratch), it shows where current models excel and where they still fail. The key finding is stark: an upgrade driven by post-training environment scaling pushes the boundary on long tasks, yet a permission-isolation defect (IDOR) survived a fully green test suite—proof that self-verification does not cover cross-cutting concerns like authorization. The article closes with a reusable six-metric, five-step evaluation framework, a mandatory human-gate checklist, and clear criteria for which tasks to hand off to the model and which still require a human in the loop.
Don’t Judge a Model’s Coding Ceiling by Benchmark Scores
If “can it write a correct function” is your yardstick for a code LLM, that yardstick stopped working a long time ago. The hard part of real engineering was never syntactic correctness. It is three things: whether the model can read the implicit contracts in existing context, whether it can stay consistent across cross-module changes, and whether it can push a task all the way to a “runnable and testable” state.
This latest upgrade pushes the frontier of what domestic open-source models can do on long-horizon tasks further than before. But a reliable evaluation doesn’t come from a leaderboard score—it comes from how the model behaves on two extreme tasks. And the conclusion is layered:
- Its long-horizon coding capability is now strong enough to test on real projects, especially feature work that spans front end, back end, data layer, and task queues;
- But it hasn’t reached the point where “all tests green” means “safe to ship.” In one acceptance test involving permission isolation, every test passed, yet manual review turned up a P1-level defect;
- On permission boundaries, security constraints, and final acceptance, humans cannot let go—and that won’t change in the near term.
The rest of this article covers the substance of what actually changed in this upgrade, breaks down the design and results of the two tasks, and then distills a reusable evaluation method.
1. What Actually Changed: Scaling the Environment, Not the Base Model
Many model iterations improve because the base model gets bigger. This one doesn’t work that way. The new version shares the same class of base model as the previous generation (hundred-billion-parameter scale)—parameter count and architecture are unchanged. The improvement comes almost entirely from scaling in the post-training stage, and it shows up along three dimensions:
- The scale of the long-horizon task environment grew to tens of times its previous size, sharply increasing how many “task → execution → failure → repair” loops the model sees during training;
- The environment became more diverse, no longer limited to short tasks in a single language or framework;
- Training time was extended significantly, giving the model repeated correction across longer execution chains.
This technical path explains a lot of practical behavior. The model’s accuracy on single-file changes improves only modestly, but its stability on the long chain “read the codebase → locate relevant modules → modify → build fails → self-repair” improves markedly. The payoff from scaling the post-training environment lands on error recovery and convergence speed on long chains—not on single-shot inference accuracy.
The official summary groups the changes into three directions, each of which maps to a different evaluation method:
| Capability direction | What it shows | How to verify it |
|---|---|---|
| Coding engineering | Stays ahead of the open-source field on public code-agent benchmarks, terminal-task benchmarks, and software-engineering bug-fix benchmarks; emphasizes long-horizon and production-grade engineering | Cross-module change tasks + end-to-end runnable verification |
| Security | Extends from “generating code” to white-box code review and vulnerability reasoning; close to frontier closed models on basic security tasks, but still behind on complete exploitation scenarios | Targeted review on planted known-vulnerability samples + red-teaming from an attacker’s perspective |
| Openness | Full weights released after security assessment and model hardening are complete | Local deployment, private-data regression, domain fine-tuning |
One counterintuitive point is worth stressing: “close to frontier on basic security tasks” and “safe to ship” are two different questions. The former is recall on recognizing known patterns; the latter requires the model to reason about “who can access what” at the level of business semantics—and that remains a weak spot.
2. Task Design: Two Orthogonal Evaluations
The worst thing you can do in an evaluation is rely on a single task type. The two tasks here are nearly orthogonal along capability dimensions:
Task A: Modifying an existing system. In a project that already carries a fair amount of legacy code, add the capability to “securely import user image assets and reference them in animation scenes.” It tests whether the model can read the old architecture, identify implicit contracts, handle database migrations, and extend the permission model without breaking existing behavior. The hard part is security constraints.
Task B: Building from scratch. Given only a set of learning materials and a product requirements document, the model must build a knowledge-learning game that can genuinely be played from start to finish. It tests the ability to extract knowledge points from unstructured material, turn a PRD into an implementable feature list, design a difficulty curve and save mechanism, and guarantee that a “beatable” loop actually holds. The hard part is intent recovery and completeness.
One tests comprehension of legacy code and adherence to constraints; the other tests incremental creation and closed-loop delivery. Together they cover the two main scenarios in production.
As a reference baseline, consider the timing from an earlier comparable change: adding an HTML animation video generation pipeline to an internal motion-graphics production platform, spanning seven layers—contract definitions, database, task state machine, API, Worker, web front end, and renderer. The first working version took about 1 hour 11 minutes, and the subsequent browser verification and defect fixes took about another 1 hour 4 minutes. This shows the model can already work continuously inside a monorepo and handle genuinely cross-stack features. Evaluation has to be anchored to this baseline—otherwise you’ll underestimate it and misjudge which layer its weaknesses actually sit in.
3. Task A Review: Why All-Green Tests Missed a P1 Defect
3.1 What the Model Got Right
The change ran the full pipeline with no mid-task abandonment or context loss: it identified the directory structure of the existing asset module, added new tables and migration scripts in the project’s established style, extended the task state machine, completed the API layer, added asynchronous handling in the Worker, wired upload and selection interactions into the front end, and added asset-reference parsing to the renderer. The build passed, all tests passed, and the feature ran correctly in the browser.
For a long-horizon task, that’s already a high level of completion. Keeping interface contracts consistent across seven layers is something the previous generation could rarely do reliably.
3.2 The Defect Found in Manual Acceptance
The problem was in permission isolation. Assets are resource objects that carry ownership, but on the new import and reference path, object-level authorization checks were missing: any user in the same tenant who knew an asset ID could pull assets they didn’t own through the reference endpoint. This is a textbook IDOR (Insecure Direct Object Reference), and in a real environment it’s a P1.
// Defective form: checks "is logged in" but not "is this yours"async function resolveAssetRef(req: Request, res: Response) {const asset = await assetRepo.findById(req.params.assetId);// Missing: if (asset.ownerId !== req.user.id) return res.status(403).end();return res.json({ url: asset.objectKey });}// Correct form: object-level authorization must be explicit and non-omittableasync function resolveAssetRef(req: Request, res: Response) {const asset = await assetRepo.findById(req.params.assetId);if (!asset) return res.status(404).end();if (asset.tenantId !== req.user.tenantId) return res.status(404).end();if (asset.ownerId !== req.user.id && !isSharedWith(asset, req.user.id)) {return res.status(403).end();}return res.json({ url: await signUrl(asset.objectKey, { expiresIn: 300 }) });}
3.3 Why the Model Was Bound to Miss It
This defect wasn’t an occasional slip from insufficient capability—it’s a structural blind spot at the level of methodology, for four reasons:
- The model writes its own tests, and the cases naturally revolve around the happy path. Its tests were all “current user uploads → current user reads,” and that path is genuinely all-green. An attacker’s-perspective case requires actively constructing a “different user” subject, and the model has no motive to build one.
- Permissions are a cross-cutting concern that doesn’t appear in feature requirements. The PRD describes “let users import image assets and reference them in animations”; the permission model is existing, implicit infrastructure. The model will correctly reuse the existing permission middleware, but it won’t proactively question whether every new entry point is actually covered by that middleware.
- There’s no threat-modeling step. Nowhere in the execution chain is there a “list every resource-access point → annotate each one’s authorization check” step. The model’s capability is spent making the feature correct, not making it secure.
- Permission semantics require business understanding. What counts as “the same user,” what counts as “shared,” and what counts as “an admin exception” are internal organizational conventions, and the codebase usually only contains partial implementations of them.
This is the root reason “all tests green” doesn’t equal “shippable”: automated tests verify whether the feature works as expected, whereas a permission defect is a case where the feature works as expected but the expectation itself is missing a rule.
4. Task B Review: Completeness from Source Material to a Playable Product
Task B’s results were clearly better. Both tasks ran through, but only the second was directly acceptable.
Starting from a set of learning materials and a PRD, the model had to complete a long chain of actions: parse the structure of the materials → extract and categorize knowledge points → design the mapping between levels and knowledge points → implement the quiz and grading logic → add progress saving and failure retry → ensure there exists a winning path from start to finish. It completed every step, and the game can be played through end to end.
This kind of task performs well because the acceptance criteria are endogenous: whether it’s beatable is something the model can verify itself by playing through, forming a closed loop. Verifying a permission defect requires an external perspective that the model can’t discover on its own.
The contrast is itself an important evaluation conclusion: the model performs markedly better on “self-verifiable tasks” than on “tasks that need external-constraint verification.” So when assigning work, classify tasks by how they’ll be accepted, not by how complex they are.
5. A Reusable Evaluation Method for Long-Horizon Coding
5.1 Six Metrics
| Dimension | Definition | How to observe |
|---|---|---|
| Task completion rate | Share of tasks that run end to end | Manual confirmation |
| Time to first runnable version | Time from start to first successful build/launch | Timing |
| Cross-module consistency | Whether contracts between the changed layers stay coherent | Code review |
| Automated pass rate | Build + unit + integration test results | CI |
| Manual acceptance defect rate | Defects found in review per thousand lines changed | Severity-tagged defect stats |
| Defect severity distribution | The mix of P0/P1/P2 | Defect tracker |
The first four the model can advance on its own; the last two must be carried by humans. The point of evaluation isn’t how high the first four are, but which categories the last two’s defects fall into—if P1 defects cluster consistently in permissions, encryption, and compliance, then the boundary for task allocation is very clear.
5.2 Five-Step Workflow
- Establish a reference baseline. Use the previous generation’s timing and defect data on the same task as the baseline, so you don’t judge absolute numbers in a vacuum.
- Inject constraints. Explicitly state architectural conventions, code style, prohibited actions, mandatory security rules, and acceptance criteria in the prompt, along with the expected self-testing approach. The more specific the constraints, the more controllable the output.
- Automated verification. The CI pipeline runs the build, static checks, and tests; the model can read failure logs and self-repair.
- Manual acceptance. Review that is independent of the model’s output, focused on cross-cutting concerns that tests don’t cover.
- Defect attribution. Classify each defect into one of four buckets—context comprehension, cross-file consistency, non-functional constraints, product-intent recovery—building an accumulating data asset.
5.3 Mandatory Gate Checklist
Before wiring model output into a real project, the following should be non-skippable human gates:
- Object-level authorization: enumerate every new resource-access entry point one by one and confirm each has an ownership check;
- Multi-tenant data boundaries: whether cross-tenant queries all have proper constraints, and whether indexes include a tenant dimension;
- Upload path: file-type allowlist, size limits, content inspection, unpredictable storage paths, and time-limited signed access links;
- Secrets and configuration: no hard-coded credentials; sensitive config goes through a secret manager, not the codebase;
- Audit logging: sensitive operations are traceable, and sensitive fields never land in logs;
- Dependency security: the provenance, license, and known vulnerabilities of every new dependency.
5.4 Task Allocation Criteria
| Safe to hand off | Needs a human watching |
|---|---|
| Scaffolding and boilerplate generation | Permission and multi-tenant isolation design |
| Mechanical cross-module refactoring | Encryption, signing, key management |
| Filling in tests and documentation | Payment, billing, and audit paths |
| From PRD to a playable MVP | External-facing endpoints that touch compliance |
| Code review for known vulnerability patterns | Final acceptance and go-live decisions |
The pattern is clear: anything whose goal can be verified automatically goes to the model; anything that has to be judged by business convention and an attacker’s perspective needs a human in the loop.
6. The Engineering Value of Open Weights
Releasing full weights after security assessment and model hardening means more for engineering than for evaluation. It unlocks four capabilities: local deployment that keeps data in-house; building a regression evaluation set on your own private codebase to form your own capability baseline; domain fine-tuning against your internal tech stack; and a cost model that turns fixed inference spend into controllable compute investment.
On infrastructure, everything involved is a general-purpose capability category: use a container platform to host inference services with on-demand elastic scaling, object storage to hold weight files and evaluation datasets, a logging and monitoring service to cover request latency, failure rate, and resource utilization, and a message queue to decouple scheduling from execution for long tasks. On Baidu AI Cloud, these capabilities all have corresponding standard product forms you can compose directly, with no custom integration required.
One reminder: open weights don’t mean out-of-the-box readiness. Before deployment you still need to complete a security assessment, configure input/output filtering policies, and run privilege-escalation and prompt-injection tests against your business scenario.
Conclusion
This generation of models genuinely deserves a test on real projects for long-horizon coding—especially on cross-stack feature work and on building verifiable closed loops from scratch, where the efficiency gains are real. But its defect distribution is equally clear: permissions, security, and final acceptance are the three areas where the model’s self-evaluation mechanism fails by design, because these problems only exist under an external perspective.
So the pragmatic way to use it is this: give it long-horizon, cross-module, self-verifiable work, and make security gates a mandatory step in the process rather than part of the model’s output. That’s also where the value of evaluation lies—it isn’t about giving the model a score, but about figuring out where its capability boundary actually falls, and where the humans need to stand near that line.
