Evaluating LLM Coding Ability: From Short Tasks to Long-Horizon Development
A practical framework for assessing large language models on software engineering work beyond single-shot code generation. It covers task scope, short-task versus long-horizon evaluation, context management, architecture versus implementation, requirement coverage, and fair benchmark design.
When discussing the coding ability of large language models, it is easy to fall into a common trap: treating the result of a single code generation as equivalent to the model’s real development capability. In enterprise and personal projects, the work is rarely as simple as “write a function.” It usually requires understanding existing code, decomposing requirements, designing interfaces, adding tests, handling exceptions, debugging, and keeping the entire system runnable after multiple rounds of revision.
Therefore, evaluating a coding model cannot rely only on a benchmark score or on whether a snippet looks elegant. A more useful approach is to place the model in a development workflow close to real work and observe whether it can sustain completion of an entire task.
1. Define What Is Being Evaluated: Code Generation Is Not Full Development
Real software development includes at least the following stages:
- Understand requirements and constraints;
- Analyze the existing project structure;
- Design module boundaries and data flow;
- Write implementation code;
- Write or supplement tests;
- Run tests and locate errors;
- Iterate based on feedback;
- Conduct code review and risk checks;
- Confirm that all requirements have been implemented.
Single-turn code generation covers only a small part of this. A model may perform well on short function completion but gradually forget constraints during long, multi-file, multi-turn interactions. It may also propose an architecturally reasonable plan but miss key branches during implementation.
So the real questions are not simply “can it write code?” but:
- Does it accurately cover the requirements?
- Does it proactively verify its own implementation?
- Does it keep decisions consistent?
- Does it lose key information as context grows?
- Does it identify errors it introduced?
- Does it change strategy when blocked?
2. Small Feature Tasks: Differences Are Often Limited
For tasks such as CRUD operations, parameter validation, data transformation, and simple API wrappers, most capable models can produce a basic implementation when requirements are clear and project conventions are complete.
The evaluation focus for these tasks should not be whether the code style fully matches personal preference. Four metrics matter more:
1. Functional Completeness
Does the code cover all paths in the requirement, including success paths, failure paths, boundary inputs, and exception handling? Many models perform well on the main flow but overlook null values, duplicate requests, insufficient permissions, timeouts, and concurrency.
2. Runnability
Does the generated code compile, pass static checks, and pass basic tests? Code that cannot run has no practical value, even if its structure looks good.
3. Change Scope
Does the model modify only the necessary files, avoid unrelated dependencies, and preserve existing interfaces? In real projects, excessive modification is often more dangerous than a local defect.
4. Maintainability
Do naming, layering, error handling, and logging follow project conventions? For small tasks, improving code quality from seven out of ten to nine out of ten may not produce proportional benefits. But if data loss, permission bypass, resource leaks, or other severe issues appear, the evaluation changes qualitatively.
Small requirements are therefore better evaluated by whether they meet the engineering baseline, rather than by over-comparing subtle style differences between models.
3. Long-Horizon Tasks Are Where Differences Really Emerge
When a task expands to dozens of files, multiple modules, and a large amount of historical context, differences in model capability become much more obvious. The difficulty in long-horizon workflows is not only whether the context window is large enough, but whether the model can manage context effectively.
A typical long-horizon task may include:
- Reading an existing project;
- Designing a new module;
- Modifying the database schema;
- Adding service interfaces;
- Supplementing tests;
- Handling compatibility logic;
- Running tests and fixing issues;
- Producing a final change summary.
In this process, models usually encounter three problems.
1. Context Compression Causes Requirement Loss
When historical information approaches the context limit, the model compresses older content. Compression itself is not the main concern. The real danger is that the compression process lacks awareness of priorities.
If the requirement list, interfaces that must not be changed, special business rules, and unfinished items are not saved separately, the model may forget them in later stages. This can appear as:
- A feature is represented only by an empty function;
- Boundary conditions that were already discussed are not implemented;
- Tests cover the main flow but miss exception branches;
- Naming or interface constraints established earlier are broken by later modifications.
The solution is not simply to enlarge the context window, but to establish a structured task state, for example:
Requirements:[ ] User authentication[ ] Permission checks[ ] Idempotency handling[ ] Exception logging[ ] Unit tests[ ] Integration testsCurrent phase: permission module implementationCompleted: authentication interface, data structuresTo verify: duplicate requests, expired credentialsMust not change: existing public interfaces
Updating a state file or task summary before each context compression is usually more reliable than relying solely on model memory.
2. Longer Reasoning Does Not Necessarily Improve Quality
Some models repeatedly reason about a problem, compare multiple approaches, and even keep questioning their previous conclusions. Moderate analysis helps reduce errors, but excessive looping consumes context and time.
For engineering tasks, a more reasonable approach is to require the model to output the following during the analysis phase:
- Current understanding;
- Candidate approaches;
- Selection rationale;
- Risk list;
- Next action.
Once an approach meets an acceptable standard, move on to implementation and verification instead of extending the discussion indefinitely. Engineering development seeks verifiable correctness, not formal length of thought.
3. Self-Introduced Errors Can Create Loops
Models sometimes introduce an error first, then keep patching around that error, eventually falling into a “modify—fail—continue modifying” loop. Common causes include:
- Not rereading the actual error;
- Modifying only the caller without checking the interface definition;
- Replacing runtime results with guesses;
- Continuing to add complexity in the wrong direction.
When two or three consecutive repair rounds still fail to solve the problem, the model should be forced to stop local patching and re-execute the following steps:
- Restore a minimal reproduction;
- Read the complete error message;
- Compare the actual code with assumptions;
- Revert the most recent ineffective change;
- Redesign a smaller fix.
This is a fault isolation strategy, and it is usually more effective than letting the model continue freely.
4. Evaluate Architecture Design and Code Implementation Separately
Different models do not necessarily have the same strengths in architecture design and code implementation.
The architecture stage values:
- Whether the model can identify system boundaries;
- Whether it can distinguish core issues from secondary issues;
- Whether it considers data consistency, permissions, security, and observability;
- Whether it can propose an appropriately complex solution;
- Whether it can explain trade-offs instead of stacking technical buzzwords.
The implementation stage values:
- Whether it accurately modifies the target files;
- Whether it follows existing code style;
- Whether it completes exception handling;
- Whether it can run tests;
- Whether it can revise code based on actual feedback.
In practice, a good collaboration pattern is: first ask the model to generate multiple architecture options, then require it to perform constraint analysis, and finally have a human confirm the direction. After the direction is fixed, ask the model to split tasks and implement them.
Do not let the model start large-scale modifications without a clear architecture conclusion. Otherwise, each subsequent code change may overturn the previous design, causing a large amount of wasted work.
5. Look Beyond “How Much Code Was Written” to “What Was Missed”
Code evaluation is easily misled by surface results. A large amount of submitted code does not mean the task is complete. More important is to establish a requirement coverage check.
Requirements can be divided into three categories:
Core Requirements
These directly determine whether the feature works, such as API behavior, data writes, permission checks, and key business flows.
Constraint Requirements
These determine whether the system is safe and stable, such as idempotency, transaction boundaries, timeouts, retries, compatibility, and resource release.
Verification Requirements
These determine whether the result is credible, such as tests, logs, monitoring, error reproduction, and regression checks.
Review should check each item rather than looking only at the diff. Pay particular attention to the following signals:
- Unimplemented placeholder functions appear;
- Tests cover only the success path;
- Features mentioned in the requirement document have no corresponding code or tests;
- The model claims “done” without showing verification results;
- There are many code changes, but no explanation of which requirements they satisfy.
A simple and effective method is to require the model to provide a requirement-file-test mapping in its final output:
Requirement: duplicate requests must not create duplicate recordsImplementation location: creation logic in the service layerVerification: idempotency test, concurrency testResult: passed / failed / pending manual confirmation
This traceability is more valuable than a long summary.
6. Designing Fairer Model Evaluations
To compare different models, use the same project, the same requirements, the same tool permissions, and the same acceptance criteria, and record the entire process.
Evaluation tasks can be divided into four groups:
Group 1: Short Tasks
Test function completion, API modification, bug fixing, and simple test writing, mainly to observe basic coding ability.
Group 2: Medium Tasks
Require implementation of a complete feature across multiple files, observing project understanding, dependency analysis, and change control.
Group 3: Long-Horizon Tasks
Ask the model to continuously complete requirement analysis, architecture design, coding, testing, and fixing, focusing on context management and task persistence.
Group 4: Review Tasks
Provide code with hidden issues and ask the model to find security, performance, concurrency, and maintainability problems, observing its ability to detect issues.
Evaluation metrics can include:
- Requirement coverage;
- Test pass rate;
- Number of severe defects;
- Ratio of ineffective changes;
- Information retention after context compression;
- Ability to recover from errors;
- Number of human interventions required to complete the task;
- Maintainability of the final code.
It is important to emphasize that a single result does not represent the model’s full capability. Model performance is also affected by prompts, task decomposition, project conventions, context length, and tool configuration. A reasonable conclusion should therefore be “more suitable for certain tasks under a certain workflow and constraints,” rather than a simple declaration that one model leads across the board.
7. Practical Recommendations for Developers
If you adopt fully automated or semi-automated coding, consider establishing the following safeguards:
- Write clear requirements, constraints, and acceptance criteria first;
- Split large tasks into small stages that can be rolled back;
- Ask the model to plan before modifying code;
- Run tests after each completed stage;
- Use an independent requirement checklist to prevent omissions;
- Keep human review for database, permission, payment, and deletion operations;
- Have the model show test results rather than only conclusions;
- When consecutive failures occur, roll back and re-localize the problem;
- Write key decisions into persistent documentation;
- Have humans perform final business acceptance and risk confirmation.
Overall, differences between coding models may not be obvious on short tasks. The real dividing line is stability in long-horizon work: whether the model can remember requirements, control changes, identify omissions, escape error loops, and continuously complete verification.
Therefore, when choosing and using a model, do not ask only “which model writes code better?” Ask further: “Which model is better suited to my project structure, task length, acceptance method, and level of human involvement?” Only by measuring model capability within the full engineering process can evaluation results become truly useful.
