How to Scientifically Evaluate a Coding LLM's Real Engineering Ability
Coding LLM benchmark scores are easy to read, but they only go so far. This article argues that engineering ability is better measured by a model's stability on long-horizon tasks, and describes a two-task evaluation design: a refactor in a legacy system with strict security constraints, and a build from zero. Both tasks ran through, yet a high-severity privilege escalation flaw passed automated tests and was caught only during manual review. Readers will find a seven-dimension scorecard, a structured logging format for keeping results reproducible, and practical guidance on which work to delegate to the model and which must stay with human engineers.
The Verdict First: Methodology Matters More Than the Score
Every time a frontier coding model gets a generation update, community discussion tends to zero in on leaderboard numbers. Put the model into a real engineering pipeline, though, and those numbers turn out to be little more than an entry ticket. What actually decides whether the model is usable is its stability on long-horizon tasks: can it read context tangled across several modules, keep changes consistent across the frontend and backend, and carry a task all the way to a “runnable, testable” state?
A typical trait of this round of updates is that the new generation has pushed open-source Chinese models’ coding ceiling higher again, with noticeably greater persistence on long-horizon tasks. Yet it still falls short of “all tests green, ready to ship”. In hands-on runs on two real projects, both tasks went through. But the legacy-system change involving permission isolation exposed a high-severity privilege escalation flaw during manual acceptance.
So this class of model deserves a layered assessment: it is worth putting on real projects, especially for long-horizon coding; but when permissions, security, or final acceptance are involved, humans cannot take their hands off. Here is how we arrived at that judgment.
Why Leaderboard Scores Aren’t Enough to Explain Engineering Ability
Benchmarks usually share a few traits: clear task boundaries, explicit scoring criteria, and relatively clean context. They measure a model’s ability to solve a single point of a problem when the goal is spelled out. Real engineering is the opposite:
- Requirements are vague. A PRD often describes only the intent; boundaries, exception paths, and the permission model all have to be filled in by developers.
- Constraints are hidden. Legacy projects carry many unwritten historical contracts, such as the meaning of a field on a given endpoint or the rules for moving between states. The model has to “understand” them, or it will break something.
- Changes span modules. A single requirement can reach across contract definitions, the database, the task state machine, endpoints, async jobs, the frontend, and the rendering layer.
- The correctness bar is stricter. On a benchmark, “it runs” counts as a pass. In production, “it runs” is only the starting point; what really matters is whether the security boundaries are right.
Judging a model’s engineering ability therefore takes more than counting points. What matters is whether it can withstand three pressures at once, in tasks close to real work: complexity, context length, and hidden constraints.
Evaluation Design: Cross-Validating With Two Very Different Tasks
A single task easily leads to a one-sided conclusion. A more reliable approach is to design two complementary tasks: one that tests “reading legacy code and respecting constraints”, and one that tests “building from zero and shipping”.
Task 1: Modifying a System With Legacy Baggage
Pick a project that has accumulated a large amount of legacy code and enforces clear security constraints, and ask the model to add a capability: secure import of user image assets, plus referencing them in HTML animation scenes. This task really tests:
- Whether the model can understand and reuse the existing layering and contracts rather than starting from scratch.
- Whether it can deliver the feature while honoring the existing security constraints: file validation, content isolation, and permission boundaries.
- Whether its changes stay consistent with the existing code style and state machine.
In tasks like this, defects often hide in the gap between “it runs” and “it oversteps its permissions”, which ordinary automated tests rarely cover.
Task 2: Building a Complete, Deliverable Product From Zero
Supply only the Agent learning materials from a technical learning platform and a PRD, and ask the model to build a knowledge-learning game from scratch that can truly be played to completion. This task tests the full chain:
- Analysis: extracting knowledge points from the materials and turning them into gameplay design.
- Implementation: setting up the project structure from zero and building the core gameplay.
- End-to-end completeness: making the product actually run and actually be completable.
Why Pair Them This Way
One tests reading legacy code and honoring constraints; the other tests creation and delivery. The first stress-tests how deeply the model understands an existing system. The second stress-tests how completely it can carry a product from requirements to delivery. Together, they help avoid the mistaken verdict of “great on a clean project, falls apart on a legacy one”.
Key Finding: Passing Isn’t the Same as Being Deliverable
Both tasks ran through in the end: each compiled, started, and was exercised by automated tests. But during manual acceptance of the first task, a high-severity permission isolation problem surfaced: the functional path worked correctly, but the path that let users overstep their permissions was never blocked.
The root of such problems is that the model optimizes for “make it work and make the tests pass”, not for “make the security boundaries correct”. Permission isolation and privilege escalation defects are often invisible to automated tests: every test passes, but the boundary leaks. This bears out the layered verdict above: on permissions, security, and final acceptance, humans have to hold the line.
Suggested Evaluation Dimensions: Build Your Own Scorecard
Turning these lessons into a scorecard makes evaluations reproducible and comparable. It should cover at least these dimensions:
- Sustained long-horizon progress: Does it keep moving steadily through multi-hour, multi-step tasks, rather than losing sight of the goal midway or repeatedly overturning its own work?
- Depth of context understanding: Can it accurately grasp logic tangled across multiple modules, rather than focusing only on local pieces?
- Cross-module consistency: Do contracts, the database, task state, endpoints, and the frontend stay coherent with one another?
- Constraint compliance: Are constraints, especially security ones, genuinely enforced rather than bypassed?
- Runnable/testable completeness: Can the deliverable actually be started, tested, and played to completion?
- Security boundary correctness: Are permissions, isolation, and privilege escalation properly blocked? (Requires dedicated manual acceptance.)
- Deliverable vs. runnable: How big is the gap between “it runs” and “it can be shipped safely”?
Recording Evaluations in a Reproducible Way
The riskiest thing in evaluation is going by gut feeling. Use a structured record to pin down every task and keep conclusions traceable:
TaskCard = {id: task identifiercategory: refactor | greenfieldconstraints: [security constraints, compatibility constraints, performance constraints...]acceptance: [automated acceptance checks..., manual acceptance checks (permission/security)...]}RunRecord = {task_id: ID of the corresponding TaskCardrun: which run this isduration: end-to-end time takenautotest: pass | fail (which checks passed / which failed)manual_check: { permission: pass | fail, security: pass | fail }issues: problems found and their severity (P0/P1/P2)verdict: runnable_only | deliverable}
The key is keeping “automated tests pass” and “manual acceptance passes” as two separate fields. The first measures engineering completeness; the second measures deliverability. Only when both pass can verdict be marked deliverable.
How to Read a Version Update
When you hear that “the new version is stronger”, ask where the gains actually came from. A notable feature of this upgrade is that the base model was not swapped out; the gains came mainly from scaling up the post-training stage: the scale of long-horizon task environments was expanded by tens of times, with more environment types and longer training runs.
So interpret the release along three dimensions:
- Coding ability: Does it stay ahead on code-oriented benchmarks? Does it put more emphasis on long-horizon tasks and production-grade software engineering?
- Security reasoning: Does the capability extend beyond generating code into white-box code review and vulnerability reasoning? Does it approach frontier closed-source models on basic security tasks, while still trailing in full exploit scenarios?
- Openness and deployment: Are the full weights released only after security evaluations and model hardening? Is the model easy to deploy, verify, and improve on your own?
One point deserves emphasis: higher scores on security evaluations do not mean the model has complete exploit capability, and the two should not be conflated.
The Boundary of Human–Model Collaboration: Security and Acceptance Stay With Humans
Overall, this class of model has crossed the “is it usable?” threshold and reached the “worth putting on real projects” stage, and long-horizon coding in particular deserves trust. But the distribution of its defects dictates how we should collaborate with it:
- Safe to hand off: long-horizon coding, cross-module implementation, building a runnable product from zero, and repetitive refactors.
- Humans must own: permission design, security boundaries, privilege escalation acceptance, and final delivery decisions.
A practical split is to let the model handle “implementation” and “self-testing”, while humans handle “defining constraints” and “accepting boundaries”. In any scenario involving permissions or security, manual acceptance should be built into the process as a separate, non-skippable step.
Summary
Don’t stop at leaderboard numbers when evaluating a coding LLM. A more valuable approach is to stress-test it with two complementary task types, “reading legacy code and honoring constraints” and “building from zero and delivering”, use structured records to separate “it runs” from “it’s deliverable”, and score it with a scorecard covering long-horizon ability, consistency, constraint compliance, and security boundaries. This turns subjective impressions into reproducible conclusions. It also lets you see clearly, before bringing the model into production, which parts of the work to hand over and which to keep firmly in your own hands.
