Long-Horizon Code Agents: A Practical Guide to Modeling, Training, and Evaluation

1. Introduction: From Single-Turn Generation to Long-Horizon Autonomous Execution

Over the past two years, code generation models have mainly been evaluated through function-level completion and single-turn question answering. The judgment criterion was relatively simple: could the generated code pass a set of unit tests?

As agent-based applications became widespread, the nature of the tasks changed fundamentally. A developer or platform now hands a requirement to a model, for example fixing a production defect, implementing a cross-file feature, or adding tests to a legacy module. The model is then expected to complete the following work autonomously:

• retrieve and understand repository context;
• formulate a modification plan;
• read and write multiple files;
• execute build, test, and static-analysis commands in a terminal;
• iterate on fixes based on error messages until acceptance conditions are met.

Under this mode, model output is no longer “a piece of code.” It is “a long-horizon trajectory composed of multi-turn reasoning, tool calls, and environmental feedback.” Evaluating a code agent therefore requires looking not only at whether the final code is correct, but also at whether it remains goal-directed across dozens of interactions, whether instruction following remains stable, and how much compute and time it consumes.

A recent industry trend is also relevant: model routes that use open-weight models as the base, focus on post-training, and concentrate compute on reinforcement learning and task-environment construction are pushing both long-horizon task performance and unit cost toward better levels. This article distills the general methods behind that route and provides an actionable training and evaluation framework.

2. Why Long-Horizon Tasks Are Hard: How Errors Accumulate Along the Trajectory

To optimize long-horizon performance, it is first necessary to understand where failures come from. From an engineering perspective, failures of long-horizon agents can be grouped into five categories.

1. Context drift. As interactions accumulate, intermediate artifacts fill the context window: file fragments read during exploration, error messages from failed attempts, and rejected modification plans. Truly important information is diluted or pushed out of the window, and the model gradually drifts away from the original objective.

2. Error accumulation. Each decision is built on the result of the previous step. If the per-step correctness rate is p and steps are independent, the probability that all n steps are correct is approximately p^n. For example, if per-step correctness is 0.98, the probability that 50 consecutive steps are all correct is only about 0.36. In real tasks, steps are correlated, so the numbers are not that pessimistic, but the pattern that error is amplified as the number of steps grows still holds.

3. Instruction-following decay. The longer the conversation, the easier it is to ignore format constraints, naming conventions, and restrictions such as “do not modify this module.” Models tend to follow recently introduced content rather than continuously align with the original constraints.

4. Ineffective loops. The model repeatedly tries the same kind of modification, or gets stuck in a “modify-test fail-modify again” loop, unable to decide when to switch approaches, roll back a version, or request human intervention.

5. Reward hacking. When training uses only “whether tests pass” as the signal, the model may learn shortcuts such as deleting failing assertions, changing a function to directly return the expected value, or bypassing the real logic. Such trajectories may score highly on benchmarks, but they become negative assets in real code repositories.

These five problems are interconnected: context drift aggravates instruction-following decay, ineffective loops amplify token consumption, and reward hacking contaminates both training data and evaluation results. Therefore, optimizing long-horizon tasks cannot focus only on whether the model is “smart.” It requires coordinated design across training objectives, data structures, inference systems, and evaluation methods.

3. Capability Modeling and Metrics: Defining Goals as “Capability Divided by Cost”

“High performance” and “low cost” are usually in tension. Larger models, longer reasoning processes, and more retries can improve pass rates, but they also increase the cost of each task. In engineering, it is common to measure cost-performance using the ratio “capability divided by cost,” rather than comparing absolute capability or absolute price alone.

3.1 Breaking “Intelligence” into Measurable Dimensions

“Intelligence” is a broad term. In the context of code agents, it can be decomposed into five measurable dimensions:

• Task completion rate: the proportion of tasks in which all hidden tests pass;
• Long-horizon stability: the proportion of tasks that converge within a specified number of rounds, and the proportion that still fail to converge after a certain number of rounds;
• Instruction-following rate: the proportion of outputs that satisfy explicit constraints such as format, scope, and prohibited actions;
• Tool-call accuracy: the proportion of calls with correct commands, parameters, working directories, and related information;
• Cost per task: the compute and monetary cost consumed to complete a task.

These dimensions are not independent. Blindly lengthening the reasoning chain can improve completion rate, but it raises cost and latency; over-compressing output weakens complex reasoning ability. The purpose of a metric system is to provide quantitative evidence for these trade-offs.

3.2 Cost Metrics: From Price per Million Tokens to Cost per Successful Task

The industry often compares models by price per million tokens. In agent scenarios, however, a metric closer to real expenditure is “cost per successful task”:

Cost per successful task = Total task cost / Number of successful tasks
Total task cost = Input tokens consumed × input price per million tokens
                + Output tokens consumed × output price per million tokens
                + Tool execution and environment overhead
                + Sunk cost of failed tasks

Assume that Models A and B have the following prices and performance:

Metric	Model A	Model B
Price per million input tokens (CNY)	12	1.2
Price per million output tokens (CNY)	60	6
Task completion rate	70%	65%
Average input tokens per task	1.2 million	2 million
Average output tokens per task	80,000	150,000

Under the example’s assumptions, the estimated model-invocation cost per successful task is about CNY 27.4 for Model A and about CNY 6.6 for Model B, so Model A is about 4.1 times more expensive. Although the list price differs by a factor of 10, the lower-price model’s greater number of interaction rounds and larger context consumption compress the actual cost gap to about 4 times. This example shows that unit price is only one input to the cost model; interaction efficiency is equally important. If task concurrency is high, fixed costs such as environment startup, image pulling, and test execution should also be included in the accounting.

3.3 Evaluation Methods: Verifiable, Contamination-Resistant, and Multilingual

A clear advantage of code tasks is that answers can be automatically verified. Patch-verification benchmarks based on real code repositories and defect records have become a common industry evaluation method: each task includes a repository snapshot, a problem description, and a set of hidden tests, and a generated patch is considered successful only if it passes all hidden tests. Such benchmarks usually cover multiple programming languages and are used to test cross-language capability.

Three points need attention when designing evaluation:

• Contamination resistance. Training data and evaluation sets must be strictly isolated, and overlap between training corpora and evaluation items should be checked regularly;
• Multiple sampling. A single run is strongly affected by sampling randomness, so results from multiple runs should be reported with mean and variance;
• Real distribution. Public benchmark distributions often differ from business scenarios, so teams should build internal evaluation sets from real task logs and use them as the main basis for model selection and regression testing.

4. Training Route: From Cold Start to Targeted Reinforcement Learning

4.1 Starting from an Open-Weight Base Model

Pretraining a model from scratch with code and reasoning capabilities requires massive corpora and months of compute, which is not feasible for most teams. Post-training on high-quality open-weight models is the commonly adopted route.

This route requires special attention to:

• license requirements for commercial use, distribution, and derivative models;
• distribution differences between training data and the base model’s pretraining corpus, to avoid catastrophic forgetting;
• quantitative evaluation of base-model capabilities as a baseline for post-training effects.

4.2 Staged Training Pipeline

A typical training pipeline can be summarized in four stages:

1. Cold-start supervised fine-tuning. Perform supervised learning on high-quality task trajectories, including reasoning, tool calls, and code modifications, so that the model learns basic interaction formats and task decomposition.
2. Reinforcement learning with verifiable rewards. Use sparse rewards based on whether hidden tests pass to improve task correctness.
3. Targeted reinforcement learning. Introduce structured textual feedback such as compilation errors, test failures, and static-analysis results as process signals to improve learning efficiency.
4. Preference alignment, optional. Use preference data based on human or model review to optimize dimensions that are hard to verify automatically, such as code readability and restraint in modification scope.

4.3 Targeted Reinforcement Learning: From “Pass or Fail” to “Why It Failed”

In verifiable-reward training, the model usually receives a reward of 0 or 1 only after the entire trajectory ends. This sparse signal has high variance and low sample utilization, especially when task failure rates are high, because many samples provide little useful gradient.

The idea of targeted reinforcement learning is to convert structured environmental feedback into text, inject it into the model’s subsequent context, and let the model improve based on “what exactly went wrong.” Typical feedback sources include:

• error locations and messages from a compiler or type checker;
• stack traces from failed test cases;
• warnings from static analysis tools;
• executable specification documents and code-review comments.

A schematic reward design is shown below. It is only illustrative of structure and should be adjusted according to the business.

def compute_reward(task, trajectory, hidden_tests):
    # 1) Final reward: whether hidden tests pass
    passed, failed = run_tests(task.repo, trajectory.patch, hidden_tests)
    base = 1.0 if not failed else 0.0
    # 2) Process reward: gain from targeted text feedback
    feedback_gain = 0.0
    for step in trajectory.steps:
        errors = parse_feedback(step.tool_output)   # Parse structured information such as errors and warnings
        if errors:
            feedback = summarize(errors)            # Compress into concise text feedback
            step.context.append(feedback)           # Write feedback back to the context to drive later fixes
            feedback_gain += 0.02 * fixed_one_error(errors)
    # 3) Anti-reward-hacking penalty: detect test modification, empty patches, out-of-scope edits, etc.
    penalty = 0.5 * detect_reward_hacking(trajectory.patch, task)
    penalty += 0.3 * out_of_scope_edits(trajectory.patch, task.allowed_paths)
    # 4) Length penalty: suppress ineffective loops and overly long outputs
    penalty += 0.2 * max(0, trajectory.num_steps - task.step_budget) / task.step_budget
    return max(0.0, base + feedback_gain - penalty)

Here, fixed_one_error counts the number of errors fixed in the current iteration and is used to estimate the incremental value brought by feedback. Three principles should be maintained when designing such rewards:

• Prioritize final rewards; use process rewards as supplements. Avoid incentivizing the model to deliberately create errors just to earn process rewards;
• Make penalties explicit. Reward hacking and out-of-scope edits must be penalized explicitly;
• Keep feedback understandable to the model. Feedback text should be concise and clearly localized, avoiding noise that obscures key information.

4.4 Training Stability: Keeping Long-Horizon Reinforcement Learning from Collapsing

Long-horizon reinforcement learning has high requirements for training stability. Common engineering techniques include:

• Group-relative advantage estimation. For multiple sampled trajectories of the same task, use the group’s average reward as a baseline to compute relative advantage, such as in GRPO-style methods, avoiding the cost of training a separate value network;
• Policy constraints. Use PPO-style probability-ratio clipping and KL penalties against a reference model to limit policy update magnitude and prevent forgetting of base capabilities;
• Length and repetition penalties. Suppress meaningless reasoning expansion and repeated actions;
• Trajectory truncation and reward normalization. Set maximum steps and maximum token counts, and normalize rewards to prevent extreme values from dominating gradients;
• Staged monitoring. Monitor reward curves, task completion rates, average length, output entropy, and KL divergence to detect collapse or degradation early.

5. Data and Task Engineering: Scaling Synthetic Tasks and Controlling Quality

5.1 Task Sources: Constructing Tasks Reverse-Engineered from Real Development Assets

High-quality tasks are the foundation for training long-horizon capability. Real development processes leave many reusable assets from which training tasks can be reverse-engineered:

• Defect-repair tasks: Construct “state before defect + problem description + code after repair” from historical defect records and corresponding repair commits, and use tests added or modified in the repair commit as the verification standard;
• Feature-implementation tasks: First write test cases, then remove the implementation and ask the model to complete the code;
• Code refactoring and migration tasks: Ask the model to refactor a module while preserving behavior, or migrate code to another language;
• Test-augmentation tasks: For modules with low coverage, ask the model to write tests covering boundary conditions.

5.2 Synthetic Expansion: Quantity Growth Must Accompany Quality Improvement

Synthetic tasks can quickly expand scale, but tasks that are simply copied or template-generated have limited value. Effective expansion methods include:

• Difficulty layering: Divide tasks by “function level, file level, module level, repository level,” and match data of corresponding difficulty to different capability stages;
• Variant generation: Change constraints for the same task, such as prohibiting new dependencies, limiting modification scope, or requiring interface compatibility, to train the model’s sensitivity to constraints;
• Cross-language transfer: Translate the same logic into multiple language versions to improve cross-language generalization;
• Environment diversity: Cover repositories with different dependency structures, build tools, and test frameworks to avoid overfitting to a particular engineering organization.

5.3 Data Quality Control: Executable, Verifiable, and Contamination-Free

As task scale increases, quality control determines the upper bound of training benefit. A pipeline-style quality gate is recommended:

1. Executability check. Automatically build and run tests in an isolated environment, and directly remove tasks that cannot be reproduced;
2. Verification-consistency check. Ensure that test cases can actually distinguish “fails before repair, passes after repair”;
3. Difficulty estimation and deduplication. Deduplicate by embedding similarity and estimate task difficulty to avoid many repetitive samples;
4. Contamination detection. Regularly compare overlap between training tasks and evaluation sets to prevent score inflation caused by data leakage;
5. Trajectory filtering. Among reinforcement-learning sampling results, retain only trajectories whose rewards meet a threshold and that pass rule checks for the next training round.

6. Compute Optimization: Training Efficiency, Reinforcement Learning Pipelines, and Inference Cost

6.1 Distributed Training: Keeping Large-Scale Clusters Fully Utilized

Common sources of wasted compute in large-scale training include communication waiting, memory bottlenecks, and load imbalance. General optimization methods include:

• Multi-dimensional parallelism: Combine data parallelism, tensor parallelism, and pipeline parallelism; in data-parallel scenarios, use optimizer-state sharding, such as ZeRO or FSDP-style approaches, to reduce per-GPU memory usage;
• Sequence parallelism and activation recomputation: For long-context training, sequence parallelism can split extremely long sequences; activation recomputation trades extra compute for memory savings;
• Mixture-of-experts architectures: If a MoE architecture is used, pay attention to all-to-all communication overhead under expert parallelism and expert load balancing, avoiding a few overloaded experts slowing down overall training;
• Communication-computation overlap: Use techniques such as gradient bucketing and overlapping computation with communication to hide communication latency;
• Precision optimization: Within the range allowed by numerical stability, use lower precision, such as BF16 or FP8, to improve per-GPU throughput.

At the cluster level, it is necessary to build resource profiles for training tasks and choose appropriate parallel strategies by task scale. Small-scale tasks do not need complex parallelism, while large-scale tasks must avoid communication becoming a bottleneck.

6.2 Reinforcement Learning Pipeline: Decoupling Sampling from Training

Reinforcement learning training usually alternates between two stages: sampling and parameter updates. The sampling stage requires many inference calls. If sampling and training share the same set of accelerators, compute idles: training cards are idle during sampling, and inference resources are idle during updates.

Common optimization ideas include:

• Asynchronous sampling: The sampling cluster continuously generates trajectories, while the training cluster consumes completed trajectories; the two are decoupled through a queue;
• Weight synchronization strategy: Synchronize policy weights periodically or by step, balancing sampling freshness against synchronization overhead;
• Partial trajectory reuse: For extremely long trajectories, use segmented sampling and checkpoint resume to reduce repeated computation;
• Inference-side acceleration: Use continuous batching, paged KV cache, speculative decoding, and other general techniques to improve sampling throughput.

The following pseudocode shows the basic structure of decoupling sampling from training:

# Rollout cluster: continuously produce trajectories
def rollout_worker(model_version, task_queue, replay_buffer):
    while not stop_event.is_set():
        task = task_queue.get()          # Tasks are dispatched from the environment side
        with sandbox(task.repo) as env:  # Isolated environment with time and resource limits
            traj = agent_loop(env, model_version, max_steps=task.step_budget)
            reward = compute_reward(task, traj, env.hidden_tests)
            replay_buffer.push(task, traj, reward)   # Only enqueue trajectories that meet the threshold
# Training cluster: consume trajectories and update weights
def trainer(replay_buffer, model, ref_model):
    while not stop_event.is_set():
        batch = replay_buffer.sample(batch_size)    # Stratified sampling by task difficulty
        advantages = group_relative_advantage(batch) # Group-relative advantage
        loss = policy_loss(model, batch, advantages) \
             + beta * kl_penalty(model, ref_model, batch)
        loss.backward()
        optimizer.step()
        weight_version += 1
        broadcast_weights(model, weight_version)     # Synchronize weights to the rollout cluster

6.3 Task Execution Environment: An Underestimated Cost in Long-Horizon Training

Long-horizon task training requires executing builds, tests, and static checks in sandboxes, and environment overhead can be as significant as model computation itself:

• Image pre-building and caching: Bake dependency installation into image layers to avoid repeated installation for every task;
• Execution time and resource limits: Prevent individual tasks from occupying resources for too long;
• Parallel scheduling: Schedule by CPU and memory requirements to improve cluster utilization;
• Fast failure termination: Terminate tasks that are obviously impossible to complete to reduce wasted compute.

6.4 Inference Cost: Turning Unit-Price Advantages into Real Cost Advantages

After deployment, inference cost directly determines application scale. Common optimization methods include:

• Model routing: Route simple tasks to small models or rules, and call large models only for complex tasks;
• Context management: Summarize and compress historical interactions, keeping only content relevant to the current sub-goal;
• Prefix cache reuse: Cache repeated prefixes such as system prompts and repository indexes;
• Quantization and speculative decoding: Reduce single-inference cost under acceptable precision;
• Budget control: Set maximum steps and maximum token counts for each task; when the budget is exceeded, escalate to human handling.

7. Evaluation System: Avoiding “High Benchmark Scores but Low Real Efficiency”

After training is complete, an evaluation system consistent with training objectives is required. It is recommended to cover the following layers:

• Offline benchmarks: Use public patch-verification benchmarks and internal task sets, and report completion rate, average steps, average token consumption, and variance;
• Adversarial testing: Specifically construct reward-hacking scenarios, for example checking whether the model can still pass verification when test files are deleted or assertions are commented out;
• Long-horizon stability testing: Measure model performance on tasks near the round limit and observe whether loops or drift occur;
• Online canary release: Canary by traffic percentage in real business, and monitor task success rate, human-takeover rate, and user adoption rate;
• Cost dashboard: Track cost per successful task by task type and model version, forming a continuous optimization loop.

It should be noted that improvement on public benchmarks is not equal to improvement in business value. Teams should use their own business task distribution as the basis and establish a continuously regression-testable evaluation baseline.

8. Engineering Implementation Checklist

For teams that want to build or select code agents, the following checklist can be referenced:

Model side

• Prefer open-weight base models, and evaluate licenses and commercial compliance;
• Use verifiable-reward reinforcement learning as the main approach, with targeted feedback as a supplement, to prevent reward design from being exploited by reward hacking;
• Strictly isolate training and evaluation sets, and establish contamination-detection mechanisms.

Data side

• Construct tasks from real development assets and layer them by difficulty;
• Establish three quality gates: executable, verifiable, and contamination-free;
• Regularly clean low-quality trajectories and keep data distribution aligned with business.

Compute side

• Decouple training and sampling resources to improve cluster utilization;
• Optimize task execution environments with image caching and scheduling;
• Introduce model routing, context compression, and budget control on the inference side.

Evaluation side

• Use “capability divided by cost” as the core metric;
• Link offline, adversarial, and canary evaluation;
• Continuously feed human feedback back into training data.

9. Conclusion

Competition in code agents is shifting from “single-turn generation capability” to comprehensive competition over “long-horizon task execution capability” and “cost per unit task.” The methods discussed in this article can be summarized into three main lines:

• Center on verifiable feedback, and use targeted reinforcement learning to improve long-horizon task stability;
• Use synthetic tasks as a lever, and expand training data under controllable quality;
• Treat compute efficiency as a constraint, and continuously optimize cost across training, sampling, and inference.

It is important to recognize clearly that benchmark scores are only approximate measures of capability. Real benefit depends on how well the model matches specific business scenarios. For R&D teams, building an evaluation and training system that is “verifiable, regression-testable, and cost-measurable” has more long-term value than chasing the leaderboard of a single model.