Long-Horizon Coding Tasks: Evaluating and Tuning Large Language Models for End-to-End Software Development
This article examines long-horizon coding tasks, where large language models are asked to complete end-to-end software engineering work over dozens of tool calls and large token budgets. Based on a comparison benchmark involving mainstream general-purpose LLMs and roughly 200 million tokens consumed in a single stage, it argues that coding ability should be decomposed into multiple dimensions, including local code quality, task planning, context management, constraint following, self-correction, and execution efficiency. It then presents an evaluation and tuning methodology covering benchmark design, token accounting, key metrics, context compaction, machine-enforced constraints, cost control, and multi-model orchestration.
1. Introduction: What a “200 Million Token” Coding Benchmark Reveals
With large models increasingly used for code generation, developer attention is shifting from “completing a few lines of code” to “handing an entire requirement to a model for end-to-end completion.” Examples include building a module skeleton or refactoring an existing service according to a specification document. These tasks often last from tens of minutes to several hours, span dozens of tool calls, and can consume tens of millions or even hundreds of millions of context tokens in a single stage. The industry usually refers to such work as long-horizon tasks.
In a comparison benchmark that the author participated in, several mainstream general-purpose large language models were evaluated. In an engineering task that consumed approximately 200 million tokens in a single stage—measured using one vendor’s accounting method—model performance differed significantly:
- Pure coding experience: differences among models were limited and mainly reflected execution style and response speed.
- Long-horizon completion: rankings changed noticeably; some models separated from others in requirement coverage, compression management, and stability.
- Tuning prerequisite: the author did not perform any targeted tuning on any model, so the transferability of the conclusions is limited.
This phenomenon shows that “coding ability” is not a single metric. It includes at least the following partially independent sub-dimensions:
- Local code quality: whether functions are correct, idiomatic for the language, and free of obvious defects.
- Task planning and decomposition: whether requirements can be broken into reasonable, verifiable subtasks and dependencies identified.
- Context management: whether critical information is retained and handed off without loss as the window approaches its limit.
- Constraint following: whether hard requirements such as TDD, interface conventions, and directory structures are continuously obeyed throughout long processes.
- Self-correction: whether the model can recover from compilation errors and test failures instead of entering a loop.
- Execution efficiency: whether the same quality is achieved with less time and fewer tokens.
This article does not judge any specific product. Instead, it abstracts the experience from the benchmark into a methodology for evaluating and tuning long-horizon coding agents, for technical teams to use when selecting models, designing tool chains, and orchestrating agents.
2. Evaluation Design: Turning “Which One Is Stronger?” into a Measurable Question
2.1 Evaluate “Model + Engineering Framework,” Not the Model Alone
The final output of a long-horizon task is determined jointly by three factors:
Final completion = f(model base capability, agent engineering framework, task and acceptance specification)
Comparing models without fixing the latter two factors can severely distort conclusions. Changing only the context-management strategy for the same model can alter requirement coverage by tens of percentage points. Therefore, the first step is to fix the following variables: tool set, file access scope, execution environment, prompt templates, and context-length configuration. Clearly identifying which layer changed allows the problem to be attributed to the correct layer.
2.2 Building the Evaluation Set
- Requirement specification: Each task should have an explicit acceptance checklist, such as “implement interface X and satisfy contract tests Y.” The specification document serves both as the evaluation basis and as model input.
- Difficulty tiers: At minimum, classify tasks into function-level, module-level, and repository-level tiers. Record the number of files involved and the estimated number of interaction rounds for each task.
- Combine static and dynamic evaluation: Static benchmarks, such as offline code-generation problems, allow rapid horizontal comparison. Dynamic benchmarks, such as fixing defects or implementing features in real repositories, better reflect long-horizon performance. The two serve different purposes and should not replace each other.
- Independent validation: Test cases should be written by evaluators or maintained separately from the implementation. This avoids a self-validation loop in which the model writes tests and then passes them on its own.
2.3 Key Pitfall: Inconsistent Token Accounting
Different models use different tokenizers. The same text may produce noticeably different token counts across models, especially for Chinese text and code. Directly comparing token counts from different vendors is like measuring height with different rulers. The following approaches are recommended:
- Billing basis: Use each model’s billing accounting method to reflect actual expenditure.
- Unified basis: Re-count all inputs and outputs using a tokenizer defined by the evaluator for horizontal comparison.
- Auxiliary metrics: Record rounds, number of tool calls, and actual execution time. These metrics are independent of the tokenizer.
2.4 Core Metrics
| Metric | Definition | Question answered |
|---|---|---|
| Requirement coverage | Number of correctly implemented and acceptance-passed requirement items / total requirement items | Whether the task was “fully” completed |
| Acceptance pass rate | Proportion of deliveries that pass all acceptance cases in one attempt | Delivery quality |
| Repair cost | Time and number of changed lines required for manual repair | Usability |
| Number of compactions and information retention rate | Number of compaction triggers; proportion of key facts that can still be correctly referenced after compaction | Long-horizon stability |
| Instruction-following decay | Change in the frequency of hard-rule violations as the number of turns increases | Strength of normative constraints |
| Loop and idle rate | Number of repeated failed steps / total number of steps | Whether execution is stuck in ineffective loops |
| Throughput and latency | Output tokens per second, first-token latency, and end-to-end time per token | Usage experience |
| Resource efficiency | Tokens and time consumed to produce output of equivalent quality | Cost-performance ratio |
2.5 Experimental Controls
- Repeated runs: Large-model outputs are stochastic, so a single result is insufficient. Repeat each task at least 3–5 times and report mean and variance.
- Same-condition comparison: Use the same budget cap, timeout, and tool permissions.
- Blind evaluation: Hide model identity during manual scoring to avoid brand expectations influencing judgment.
3. Context Management: The Decisive Factor for Long-Horizon Tasks
3.1 Trigger Timing: Later Is Not Always Better
Compaction refers to summarizing historical dialogue and tool outputs when context usage approaches the window limit, in order to free up space. A frequently observed behavior is that some models start compaction when context usage reaches about 75%–80%. If a model’s reasoning chain is very long, context consumption can be fast, leading to frequent compaction. Each compaction introduces information loss; frequent compaction is equivalent to repeated memory loss.
The opposite extreme is compacting too late, causing truncation at the most critical moment. The compaction threshold should therefore be adjusted based on the following factors:
- Whether the remaining budget is sufficient to complete the current stage.
- Whether there is critical information that must not be lost, such as interface signatures or pending tasks.
In engineering practice, the compaction threshold can be made configurable and tuned based on measurements. The following configuration is illustrative only; actual field names and available parameters should follow the official documentation of the selected model or platform:
# Illustrative only: parameter names and value ranges must follow the official documentation of the selected model/platformcontext:window_tokens: 1000000 # Increase the usable context window; this can significantly reduce compaction frequencycompaction_trigger_ratio: 0.85 # Compaction trigger threshold; values closer to 1 mean fewer compactions but higher riskreasoning:max_thinking_tokens: 8000 # Limit per-step thinking length to prevent reasoning chains from consuming context too quicklybudget:max_turns: 200 # Maximum total number of turnsmax_runtime_seconds: 7200 # Maximum total runtime in seconds
3.2 Information Most Likely to Be Lost During Compaction
Based on benchmark observations, the most common major omissions during compaction include:
- Incomplete requirements: The compaction summary may preserve only what has been completed and omit queued tasks.
- Decisions already made: For example, “use a unified layered structure” or “wrap all errors with exceptions.” After compaction, the model may act on its own instead.
- Interface contracts: Function signatures and data structure definitions may be summarized into vague statements such as “implemented the relevant functionality.”
- Failed approaches: The model may forget that an approach has already failed and try it again.
- Original constraints: Hard requirements such as TDD and directory conventions may be diluted or disappear entirely.
3.3 Countermeasures: Structured Handoff, Layered Compaction, and External State
The core idea is: do not expect the model to remember everything; externalize the state. The following general practices can be used with any model:
(1) Structured handoff package
Before compaction, the framework—not the model—generates a machine-readable state file. After compaction, this file is reinjected as part of the system prompt:
def build_handoff(state):"""Called by the orchestration framework before compaction to externalize critical state."""return {"remaining": state.todo, # List of remaining tasks, each with acceptance conditions"decisions": state.decisions, # Confirmed architecture and interface decisions"contracts": state.contracts, # Function signatures, data structures, and exception conventions"failed": state.failed, # Failed approaches and reasons, to avoid repeating mistakes"files": state.touched, # Modified files and their responsibilities"next_step": state.next_action, # Next action}
(2) Layered compaction: Prioritize preserving “recent raw dialogue + task list + decision records,” and only summarize older history. Keep the near context faithful and compress distant history.
(3) External state: Write requirement lists and decision records into repository files such as REQUIREMENTS.md and DECISIONS.md. The framework re-reads and reinjects them after each compaction. No matter how the model compacts, the source of truth remains in the file system.
(4) Phase isolation: Split long-horizon tasks into phases, such as planning → implementation → review → acceptance. Phases communicate only through handoff packages. Each phase starts from a clean context, which controls the size of a single context and reduces interference between phases.
4. Constraint Following: From Written Guidelines to Executed Guidelines
4.1 Typical Constraint Decay
A frequent problem observed in the benchmark is that models strictly follow TDD, directory conventions, and interface agreements early in development, but adherence visibly declines as the number of turns increases. The main manifestations are:
- Empty-implementation escape: Using empty functions or placeholder implementations to “skip” expensive features, then omitting them during compaction. Even with strict test-first requirements, the requirement omission rate can exceed 20%.
- Self-looping: Getting stuck in an erroneous logic path that the model itself created, with difficulty breaking out.
- Rule dilution: When dozens of rules are stacked into a prompt, the model may remember only a few of them.
4.2 Use Machine-Enforced Constraints Instead of Natural-Language Constraints
Natural-language guidelines are diluted by context; machine-enforced constraints are not. The following combination is recommended:
- Gate scripts: Before each submission, run formatting checks, static analysis, type checking, and all tests. If any check fails, the process cannot proceed to the next phase.
- Contract tests: Use independently maintained contract tests to define interface behavior. Any implementation that does not satisfy the contract fails.
- Loop detection: Deduplicate tool calls. If consecutive repeated failing commands are detected, force termination and roll back.
import hashlib, timeclass LoopGuard:"""Executed at the orchestration layer; it does not rely on the model's self-awareness."""def __init__(self, window=5, max_runtime=7200):self.seen, self.window, self.max_runtime = {}, window, max_runtimeself.start = time.time()def check(self, step_signature: str):if time.time() - self.start > self.max_runtime:raise RuntimeError("Runtime budget exhausted; switch to the wrap-up process")self.seen[step_signature] = self.seen.get(step_signature, 0) + 1if self.seen[step_signature] >= self.window:raise RuntimeError(f"Repeated step detected: {step_signature[:80]}")
4.3 Layer Constraints Instead of Stuffing Everything at Once
Constraints should be divided into three layers, with high-priority constraints visible every turn:
- L0 hard constraints: Non-negotiable rules such as “no empty implementations” and “all contract tests must pass.” Put them in the system prompt and validate them mechanically through gates.
- L1 architecture conventions: Directory structure, layering conventions, and naming style. Put them in specification files inside the repository and reference them when tasks start.
- L2 preference suggestions: Preferences such as code-style choices, which can be reasonably balanced.
This works better together with a “requirement list + completion self-check.” After completing each item, the model checks it off in the list and outputs a completion report. The framework compares the list with acceptance results and automatically requests follow-up work when omissions are found. This is far more effective than simply writing “please make sure to complete all requirements” in a prompt.
5. Execution Efficiency and Cost Control
5.1 Focus on Per-Step Cost, Not Only Throughput
Slower responses during peak periods and fluctuating end-to-end time per token are common experience problems in long-horizon tasks. The reason is that long-horizon tasks are more sensitive to stability and unit cost: if one stage slows down, the entire pipeline is delayed.
During evaluation, record:
- Output speed in tokens per second and first-token latency.
- Performance differences between peak and off-peak periods.
- Number of tokens consumed per completed requirement item.
If the same model performs noticeably differently at different times, engineering should consider strategies such as queuing, degradation, and retries, rather than attributing the fluctuation to model capability.
5.2 Use Budget Control Instead of Unlimited Execution
Set three budgets for each task: turn budget, token budget, and time budget. When the task approaches a limit, it enters a wrap-up process: prioritize delivering core requirements, and record the rest as pending tasks. This controls cost and avoids the case where 90% of the work is done but the process is truncated with 10% remaining.
6. Multi-Model Collaboration: Put Each Capability in the Right Place
A noteworthy observation from the benchmark is that different models have clearly different “personalities,” and this can be used for division of labor:
| Capability profile | Typical behavior | Suitable role |
|---|---|---|
| Planner | Objective and balanced design; good at task decomposition | Main planning and execution |
| Reviewer | Careful; able to identify omissions and boundary conditions | Independent review and gap filling |
| Over-deliberative | Repeatedly scrutinizes details and may become overly focused on local issues | Limited-scope specialized review |
6.1 A General Orchestration Pattern
- Planning phase: The planner model produces task decomposition and interface design, outputting a structured plan such as YAML or JSON.
- Execution phase: Requirements are implemented one by one in phase-isolated contexts. Each requirement has clear acceptance conditions.
- Review phase: The reviewer model checks the code diff against the requirement list item by item and outputs a problem list ranked by severity.
- Arbitration phase: Review comments are triaged. Local-detail issues can be fixed in batch, while architecture-level issues require returning to the planning phase for reassessment. This avoids “losing the big picture while focusing on small details.”
- Acceptance phase: Machine gates perform tests, static checks, and completion comparison, rather than relying on model self-assessment.
6.2 Orchestration Principles
- Context isolation: Reviewers should not share the full history with executors, to avoid being biased by preconceived conclusions.
- Structured handoff: Phases exchange only structured artifacts such as plans, diffs, and problem lists.
- Single source of truth: Requirement lists and decision records are maintained by the framework; models only read and update them.
- Rollback: Persist artifacts from each phase. If a failure occurs, retry from the last stable point.
7. Deployment Checklist: From Evaluation to Production
- Build an internal evaluation set: Cover function-level, module-level, and repository-level tasks, each with acceptance cases.
- Unify measurement standards: Record token consumption under both the billing basis and a unified tokenizer basis.
- Externalize state: Turn requirement lists, decision records, and handoff packages into files. Make compaction policies configurable.
- Use machine-enforced constraints as a safety net: Gate scripts, contract tests, loop detection, and budget control.
- Clarify role division: Decouple planning, execution, review, and acceptance; avoid relying on a single model for all stages.
- Establish monitoring: Record token consumption per turn, number of compactions, and the distribution of failure causes. Use log services and monitoring alerts for trend analysis. Intermediate artifacts from long tasks can be persisted to object storage for replay and checkpoint recovery.
8. Conclusion
The core conclusion from this long-horizon coding benchmark is: In long-horizon tasks, differences in model “personality” often have a greater impact on the final output than differences in capability. Over-deliberative models may become overly focused on local details; models that suppress details may lose requirements during compaction; meticulous models add clear value during review. However, these characteristics can be constrained and guided through engineering practices.
Therefore, instead of arguing about “which model is the strongest,” it is more useful to reframe the question as “how can a model plus an engineering framework reliably complete long-horizon tasks?” Managing context well, mechanizing constraints, externalizing state, and assigning roles appropriately is a reliable path to improving delivery quality in real projects.
