How to Evaluate LLMs for Long-Horizon Coding and Deliverable Capability
Static benchmark scores tend to overstate the real engineering value of code-capable large language models. This article argues that evaluation should shift from single-problem accuracy to end-to-end delivery, and demonstrates the point through two anonymized hands-on tasks: enhancing security features in an existing codebase, and building a fully completable interactive learning game from scratch. It breaks down three core dimensions of long-horizon coding—context comprehension, cross-module progress, and convergence to a runnable, testable state—explains how post-training strengthens coding ability, and shows through a concrete permission-isolation defect that "tests passing" does not equal "safe to deliver." The article closes with a practical methodology (real projects as the yardstick, human acceptance as the backstop), an FAQ on model selection and permission test design, and guidance on defect grading.
Introduction: Evaluation Should Focus on Long-Horizon Delivery
When assessing a code-capable large language model, relying only on static leaderboards tends to overstate its true engineering value. Leaderboard problems usually have clear boundaries and limited context; a model scores by producing correct code within a small, local snippet. Real software projects are different. A codebase contains historical implementations, implicit conventions, module dependencies, and security constraints. The model must read a large amount of context, advance a task across files, and stay consistent through multiple rounds of modification. Evaluation focus should therefore shift from “single-problem accuracy” to “end-to-end delivery capability.”
This article is based on two anonymized hands-on tasks: one enhances security features within an existing project, and the other builds a completable learning game from scratch. The former tests whether a model can modify a legacy system under constraints; the latter tests whether it can close the product loop from empty material. Together, these tasks let us observe capabilities such as complex-context comprehension, cross-module progression, and reaching a runnable, testable state.
Evaluation Design: Two Complementary Tasks
A single task can rarely cover the full picture of coding ability. A more reliable approach is to set up two task types at once—“modification under constraints” and “construction from a blank slate”—and to define inputs, scoring items, pass conditions, and common failure points explicitly.
| Task | Goal | Input | Acceptance Focus | Common Failures |
|---|---|---|---|---|
| Task 1: Security feature enhancement in a legacy project | Add asset import and front-end animation resource references to an existing project | Anonymized codebase, module descriptions, security specifications, API documentation | Whether it reuses existing abstractions; whether it respects module boundaries; whether it completes permission checks | Blindly adding new files; testing only the happy path; missing privilege-escalation cases |
| Task 2: Building a fully completable application from scratch | Deliver a playable, save-able, completable interactive learning game from the provided material | Domain learning material, requirements document, acceptance criteria | Whether it launches; whether the state machine closes the loop; whether progress saving is reliable; whether the completion condition is clear | UI is complete but the game cannot be finished; state is lost after loading a save; completion conditions are missing |
Task 1: Security Feature Enhancement in a Legacy Project
The input for this task should include an anonymized codebase, module boundary descriptions, data API documentation, and a list of security constraints. The requirement should not merely say “add an import feature”; it should also specify which asset types to support, whether to reuse the existing upload logic, whether permission inheritance is involved, and whether cross-module references to front-end resources are allowed.
Scoring items can be divided into four categories. First, context localization: can the model find the correct service-layer, API-layer, and presentation-layer files? Second, restraint in modification: does it prioritize reusing existing functions, existing validation, and existing exception handling rather than casually adding duplicate modules? Third, module consistency: does the new logic follow the project’s original layering conventions and naming style? Fourth, security completeness: does it cover scenarios such as admin import, a regular user reading their own resources, a regular user reading someone else’s resources, and invalid resource identifiers?
Pass conditions should go beyond “the feature runs.” They should also include: the main-flow test cases pass; permission-related negative cases pass; there are no P1-level security defects; the change does not noticeably break existing APIs; and manual acceptance can produce a reproducible pass record.
Common failures include: the model implements only the new page without hooking into the existing permission system; the model can write main-flow tests but does not add privilege-escalation cases; the model adds a large number of files to complete the task, driving up subsequent maintenance cost.
Task 2: Building a Fully Completable Application from Scratch
The input for this task should include domain learning material, a requirements document, and acceptance criteria. The requirements document should specify which learning modules the application must complete, how to advance to the next level, how progress is saved, how completion is judged, and how abnormal input is handled.
Scoring items should focus on product-loop capability. First, requirements decomposition: can the model turn learning material into levels, question banks, states, or progress structures? Second, state-management quality: does it design a clear state machine that correctly handles start, in-progress, failure, completion, save, and load? Third, runnability: can the application run from start to finish rather than only showing a static UI? Fourth, testability: does it provide verifiable entry points, logs, test scripts, or acceptance steps?
Pass conditions should include: the application launches; the core learning flow can be completed; progress can be saved and restored; the completion condition can be triggered explicitly; abnormal states do not crash the program; and evaluators can reproduce results following the instructions.
Common failures include: the model completes the UI but does not implement completion judgment; state is inconsistent after loading a save; the model demonstrates only part of the flow without covering failure and retry paths; reproducible launch instructions are missing.
Three Core Dimensions of Long-Horizon Coding
Judging Whether the Model Understands Complex Context
The first hurdle in long-horizon tasks is context comprehension. The larger the project, the more implicit conventions there are—naming rules, directory layering, error-handling conventions, interface contracts, and configuration-loading methods. The model must first locate the correct files, then determine which modules a change will affect.
During evaluation, watch three things. First, does the model blindly add new files, or does it prioritize reusing existing abstractions? Second, does it break original module boundaries—for example, bypassing the service layer to operate on data directly, or writing business logic into the presentation layer? Third, can it recognize implicit constraints such as permissions, tenant isolation, resource ownership, and audit logging?
A model that truly understands the context tends to show greater restraint: smaller change scope, explanations that stay close to the existing implementation, and test designs that focus on real call chains.
Observing Cross-Module, Sustained Progress
Real features often span the contract layer, data layer, task state, API layer, execution layer, and presentation layer. Whether a model can stay consistent across multiple files is the dividing line for long-horizon ability. Weaker models tend to exhibit a “fix one spot, forget another” pattern during cross-module changes: the API field changes but the front end is not adapted; the state definition changes but the save logic is not updated; a permission check is added but the exception response format becomes inconsistent.
To evaluate this, ask the model to complete a full chain—for example, “user uploads asset → save record → generate display resource → front end references it → roll back on failure.” If the model can complete only one of these files and the rest of the chain requires repeated human patching, its long-horizon progression ability is still limited.
Judging Whether It Converges to a Runnable, Testable State
The goal of long-horizon coding is not “finish writing the code” but to bring the task to a runnable, testable state. The model should proactively construct a verification path: can it pass the build, start the service, trigger the core flow, and provide reproducible test results?
To evaluate this, see whether the model proactively provides launch commands, minimal test cases, dependency notes, and acceptance steps. Output that is genuinely close to engineering delivery usually explains how to verify success and points out risks that are not yet covered. Output that gives only code without a verification path usually remains at the demo-generation stage.
How Post-Training Improves Coding Ability
From a technology-evolution standpoint, once the base model’s scale is fixed, further improvements in coding ability come mostly from investment in post-training. Common practices include expanding the scale of long-horizon task environments, diversifying environment types, extending training time, and shifting the ability focus from “generating correct code snippets” to “sustained task progression within real engineering.”
In public information and hands-on observation, post-training typically affects three kinds of capability. First, coding ability oriented toward production-grade software engineering and long-horizon tasks. Second, expansion from code generation to code review, vulnerability reasoning, and basic security tasks. A model may approach frontier levels on some basic security tasks but still lag in full exploit development, complex permission models, and real attack-surface reasoning. Third, open weights make deployment and secondary improvement easier.
Therefore, model selection should not look only at whether a model can generate a correct code snippet, but also at whether it excels at sustained delivery under existing constraints.
Capability Boundaries Exposed by Hands-On Testing: Runnable Does Not Mean Deliverable
During the hands-on testing and manual acceptance of Task 1, a typical problem appeared: the feature path worked normally, but there was no protection against privilege escalation. This was logged as a P1-level permission-isolation defect.
The trigger scenario after anonymization is as follows. In an asset-management module, an admin can import assets normally; regular user A owns their own asset; regular user B owns a different asset. The functional cases covered admin import and a regular user reading their own asset, so the main-flow test passed. But manual acceptance found that regular user A could directly construct an asset identifier and call the detail endpoint to read regular user B’s asset. Because the original tests did not cover the negative case of “a regular user reading someone else’s resource,” the privilege-escalation path was never intercepted.
The failing example can be simplified as: request GET /api/assets/asset-b-001; expected response is 403 or PermissionDenied; the actual response is 200 with the asset content, causing unauthorized resource access.
The reproduction steps can be summarized as: first, prepare test data such as user_admin, user_a, user_b, and asset_b; second, log in as user_a or construct a call context; third, construct the identifier of asset_b, which belongs to user_b; fourth, call the asset detail endpoint; fifth, observe whether the endpoint denies access. If the endpoint returns user_b‘s asset content, the case is judged as failed.
Below is a simplified example showing the difference between functional cases and security cases. The example uses pytest style and keeps only the permission logic of the endpoint under test. asset_id denotes the resource identifier, and owner_id denotes resource ownership. Before running, install pytest and place the test file in a tests directory; the minimal run command is pytest -q tests/test_assets.py. If the real endpoint is an HTTP service, you can replace the read_asset call with requests or a test client.
import pytestclass PermissionDenied(Exception):passclass AssetNotFound(Exception):passclass User:def __init__(self, user_id, role="user"):self.user_id = user_idself.role = roleclass Asset:def __init__(self, asset_id, owner_id):self.asset_id = asset_idself.owner_id = owner_id# Simplified endpoint: read_asset(user, asset_id, db)def read_asset(user, asset_id, db):asset = db.get(asset_id)if asset is None:raise AssetNotFoundif user.role != "admin" and asset.owner_id != user.user_id:raise PermissionDeniedreturn asset@pytest.fixturedef db():return {"asset-b-001": Asset("asset-b-001", owner_id="user-b")}@pytest.fixturedef user_a():return User("user-a")@pytest.fixturedef user_b():return User("user-b")@pytest.fixturedef user_admin():return User("admin", role="admin")def test_admin_can_read_asset(db, user_admin):asset = read_asset(user_admin, "asset-b-001", db)assert asset.asset_id == "asset-b-001"def test_user_cannot_read_others_asset(db, user_a):with pytest.raises(PermissionDenied):read_asset(user_a, "asset-b-001", db)def test_user_can_read_own_asset(db, user_b):asset = read_asset(user_b, "asset-b-001", db)assert asset.asset_id == "asset-b-001"
This example focuses on the permission check itself and does not expand into databases, auth middleware, or exception mapping. In real engineering, you should also add scenarios such as not-logged-in, expired session, resource not found, and role changes, according to the API framework. The key is not to let the example replace a full test suite, but to illustrate that functional cases answer “can a legitimate user use it,” while security cases answer “can someone who should not access it be denied.”
Models tend to prioritize making the “green, passing” path hold, yet they may not proactively fill in security cases such as privilege escalation, permission boundaries, and input validation. This is precisely the root of why “all tests green” does not equal “deliverable.”
Evaluation Methodology: Real Projects as the Yardstick, Human Acceptance as the Backstop
Based on the observations above, we can offer a more grounded evaluation method.
First, use real projects as the primary yardstick. Testing with real codebases that contain historical code, security constraints, and module dependencies reflects long-horizon ability better than static leaderboards. An anonymized project should retain its key structure rather than be stripped down to only example logic.
Second, design two complementary tasks. Include both “modification under constraints” and “construction from a blank slate” to cover both ends—understanding a legacy system and delivering from scratch. Relying on only one task easily leads to a one-sided conclusion.
Third, incorporate security into the acceptance checklist. Explicitly require filling in privilege-escalation, permission-isolation, input-validation, and exception-path cases. Do not look only at whether the main flow runs, and do not count only the pass rate.
Fourth, keep a human acceptance step. When permissions, security, and final delivery are involved, humans cannot fully let go. A model can be an efficient executor, but acceptance authority, security judgment, and final responsibility should remain with humans.
Fifth, record failure samples and defect severity. Define separate standards for P0, P1, and P2 issues. For example, unauthorized access to others’ resources, permission bypass, and sensitive-information leakage should be classified as high priority; unclear prompts and non-critical experience issues can be classified as low priority.
FAQ: How to Choose a Model and Design Permission Cases
Q: How do I choose a coding model better suited to real engineering?
A: Prioritize running a small-scale trial with an anonymized real codebase. Do not just have the model write isolated functions; instead, have it complete a cross-module change—for example, adding an endpoint, wiring in permissions, adding tests, and updating documentation. Observe whether it can reuse existing structures, maintain module consistency, and proactively raise risks and verification approaches.
Q: What are common mistakes?
A: Common mistakes include looking only at leaderboard scores; running only the happy-path main flow; treating “compiles” as “runs”; ignoring permission and input validation; asking the model to complete an overly large task at once without providing acceptance criteria; and misjudging all-passing test results as safe to release.
Q: How should I design permission cases?
A: Cover at least six scenarios: normal access, privilege-escalation read, privilege-escalation write, resource not found, role mismatch, and resource-identifier enumeration. For asset-import features, also verify file type, file size, file name, source path, and permission inheritance. For admin-side features, verify the differences among admins, regular users, and unauthorized users.
Q: Why should Task 2 also undergo permission acceptance?
A: Even a standalone learning game may involve progress files, user configuration, or local data. If the model implements progress saving, you need to verify whether the save file can be incorrectly overwritten, whether it distinguishes between different users or level states, and whether data validity is checked when loading a save. A permission mindset is not just a server-endpoint issue; it is also a state-isolation issue.
Conclusion
Across the two anonymized projects in this article, long-horizon coding ability has begun to show its value in supporting real development trials: the model can understand a degree of project context, advance features across multiple files, and bring tasks to a runnable, testable state. But “runs” does not automatically equal “safe to deliver.” Permission isolation, input validation, exception paths, and security acceptance still require an explicit checklist and human oversight.
Future evaluation should expand the sample to cover more project scales, permission models, and failure scenarios. Only with sufficient human acceptance, defect recording, and security-boundary documentation can long-horizon coding ability move gradually from “demo-usable” to “reviewable before production.”
