Balancing Cost and Performance in Long-Horizon Agent Reinforcement Learning

    Long-horizon coding and terminal agents must complete multi-step workflows involving log analysis, file edits, test execution, and plan revision. This article explains how to evaluate these agents using task success rate and per-task cost, how full-parameter reinforcement learning can improve long-range decision-making, what a reproducible training recipe should contain, and how to establish fair evaluation, deployment safeguards, and a practical implementation path.

    Long-horizon agents are not solving the same problem as conversational models that can write a few lines of code. Terminal and coding agents must read logs, modify files, run tests, and revise their plans based on failures across many interactions. A task may span dozens or even hundreds of steps. A small improvement at each individual step does not guarantee that the full trajectory will remain stable; the agent may still fail near the end.

    For this reason, training objectives should not focus only on single-turn accuracy. They should measure whether the complete task reaches a successful closed loop, and how much reasoning and environment interaction that loop consumes.

    Defining the Problem: Success Rate and Per-Task Cost

    In practice, two categories of metrics should be tracked together.

    The first is task success rate. For automatically evaluated benchmarks, this is commonly measured as the average pass rate across multiple independent trials, reducing the impact of a single unusually favorable sample. Success should be determined by tests, validation scripts, or the final environment state—not by the model claiming that the task is complete.

    The second is per-task cost. At minimum, it includes three components:

    • Computation used for model inference and generation
    • Environment execution time, including compilation, testing, and container startup
    • Additional trajectories caused by failures and retries

    Reporting only the highest success rate does not show whether a system can fit into routine software development. Reporting only low cost does not establish long-horizon reliability for real repository changes.

    A useful way to compare strategies is to plot them on a success-rate versus per-task-cost plane. Using a logarithmic cost axis often makes the comparison clearer. The solutions that are not simultaneously dominated by another solution in both success rate and cost form the Pareto frontier. The practical goal of training is not to increase reasoning intensity without limit. It is to move the same base model along this frontier: either raise success rate or significantly reduce cost while maintaining a similar success rate.

    What Full-Parameter Reinforcement Learning Adds

    Many general-purpose foundation models already have basic capabilities in syntax, library usage, and short-range patching. Their weakness is often long-horizon credit assignment. An early decision to inspect the wrong file may not become visibly harmful until dozens of steps later. If the only reward is a sparse signal at the end of the task, the model receives little useful feedback about the large amount of ineffective exploration in the middle.

    Full-parameter reinforcement learning means updating the main policy network rather than training only a small adapter. The objective is to change long-range decision habits, not merely surface wording.

    This approach is useful when:

    • The desired behavior differs substantially from the pretraining distribution, such as strict terminal constraints or a prohibition against inventing file paths.
    • The workflow must consistently follow a pattern such as reproducing the failure, making the smallest necessary change, and running regression tests.
    • Inference should not depend on extremely long chains of thought and excessive compute; the policy itself should become shorter and more accurate.

    The trade-offs include memory usage, communication overhead, and training stability. In engineering work, the training recipe should therefore be documented as an auditable checklist rather than summarized vaguely as “reinforcement learning was used.”

    What a Reproducible Training Recipe Should Include

    A recipe that others can verify should specify at least five areas: data, environment, rewards, optimization, and evaluation.

    Tasks and trajectories. Tasks should cover four kinds of actions: reading code, modifying code, running commands, and interpreting failures. Difficulty should be stratified. Overly easy examples encourage template-like behavior, while overly difficult tasks with extremely sparse rewards can produce almost no useful gradient signal.

    Trajectories should preserve the original environment observations rather than only the model’s summaries. Otherwise, the policy may learn how to describe success instead of how to reach it.

    Action space. High-risk operations should be restricted to an allowlist, such as inspection, editing, testing, and commit-message generation. Unrestricted network access and arbitrary deletion should be blocked. Actions should be parseable and replayable. When an action fails, the environment should return a structured error instead of an unprocessed terminal dump.

    Rewards. Terminal rewards should reflect whether tests pass, whether the patch is minimal, and whether unrelated files were changed. Process rewards should be limited and concrete. Small positive signals may be assigned for successfully reproducing a defect, locating the correct module, or preserving existing regression tests.

    Avoid rewarding long answers or confident wording. Such signals encourage idle output rather than productive behavior.

    Optimization and stability. Long-horizon tasks have high variance. Training may require group normalization, truncated advantages, and limits on the size of individual updates. Extremely long failed trajectories should receive a length penalty so that the policy cannot exploit process rewards through unlimited retries.

    Checkpoints should not be selected solely by training reward. They should also be evaluated on a fixed holdout set using success rate and average step count.

    Compute budget. The recipe should state the accelerator scale, training duration, number of sampled trajectories, and maximum number of interaction steps. Transparent budgeting helps distinguish algorithmic improvements from gains caused by incomparable amounts of compute.

    Specific hardware types and training days may remain in internal records. For external comparisons, it is more useful to report the compute required per successful task.

    A framework-agnostic outline of the sampling and update loop might look like this:

    1. for batch in task_stream:
    2. trajectories = []
    3. for task in batch:
    4. state = env.reset(task)
    5. steps = []
    6. for t in range(max_steps):
    7. action = policy.sample(state)
    8. state, done, info = env.step(action)
    9. steps.append((state, action, info))
    10. if done:
    11. break
    12. reward = score(task, env.final_state(), steps)
    13. trajectories.append((steps, reward))
    14. advantages = normalize(trajectories)
    15. policy.update(trajectories, advantages, kl_penalty)
    16. evaluate_on_heldout()

    This pseudocode describes the data flow and does not represent a proprietary interface from any particular platform. In production, equivalent constraints should be implemented using the team’s existing training and container environments.

    Evaluation: Establish a Common Measuring Stick First

    One of the most common mistakes in long-horizon agent comparisons is using one environment for training and another for published results. Command timeouts, dependency installation, test parallelism, and retry limits can all change the pass rate.

    A rigorous evaluation should:

    1. Fix the task list, random-seed range, and maximum number of steps.
    2. Run each task independently multiple times, reporting the mean and a measure of variation.
    3. Distinguish official public scores from scores produced by an internally built execution framework. If a baseline’s public score comes from a different framework, state that the numbers are not directly comparable or rerun the baseline in the same framework.
    4. Report average steps, average generated output, and the distribution of failure types instead of publishing only one pass-rate number.

    Transfer evaluation is equally important. An improvement limited to terminal-operation tasks may simply reflect overfitting to the command patterns of that environment. The policy should be tested on another type of software engineering task—for example, receiving a defect description and repository snapshot and producing a patch that passes hidden tests.

    Improvement in both settings provides stronger evidence that the policy has learned localization and validation skills rather than memorized benchmark patterns.

    How to Compare Costs Fairly

    Self-hosted deployments and usage-based APIs have different pricing models. Placing the electricity cost of one organization’s machines next to another provider’s list price can lead to misleading conclusions.

    A more reliable approach is to:

    • Estimate self-hosted model cost using a public, verifiable equivalent unit price, or report token volume, interaction steps, and wall-clock time as separate physical measures.
    • Evaluate comparison systems using their recommended configurations, including high-reasoning settings when applicable. Do not compare against only an expensive configuration of the other system to make the local system look cheaper.
    • Use “one task attempt” as the cost denominator, including failed trials rather than counting only the successful trajectory.
    • If the gain comes from shorter trajectories, separate the contribution of policy quality from the contribution of early stopping and caching.

    The practical interpretation of a Pareto improvement is straightforward. Once systems reach the same success-rate tier, a significantly lower per-task cost makes a system a strong candidate for the default coding-assistant backend. If cost is similar but success rate is higher, it may be better suited as an escalation path for difficult defects. If neither condition holds, the system should not be described publicly as having reached the frontier.

    Engineering Safeguards Before Production

    A high training score does not mean that an agent is ready for repository access. Before deployment, teams should add the following controls:

    • Use a least-privilege execution environment. Network access, secrets, and production data should be hidden by default.
    • Run every patch through the project’s existing tests and static checks before human review.
    • Record the task, patch, and test results for later investigation rather than retaining only the model’s final response.
    • Add step and budget circuit breakers for trajectories that make no progress.
    • Monitor success rate, average cost, and rollback rate—not merely the number of calls.

    These capabilities can be implemented with general container platforms, object storage, logging, and monitoring systems. They do not require a particular proprietary product. When cloud infrastructure is used for training and evaluation, resource isolation, image versions, and experiment records should be part of the recipe so that the findings remain reproducible on another set of machines.

    A Practical Implementation Sequence

    There is no need to apply full-parameter reinforcement learning across every repository at once. A more controlled path is:

    1. Select a subset of tasks with automatically verifiable outcomes.
    2. Standardize the evaluation framework and rerun the base model to establish baseline success rate and per-task cost.
    3. Run small-scale reward ablations to verify the combination of terminal test rewards and limited process rewards.
    4. Perform full-parameter updates within a controlled step budget and observe whether ineffective retries decrease as well as whether success rate improves.
    5. Test the same checkpoint on another category of software engineering tasks to measure transfer.
    6. Preserve the configuration, random seeds, and cost details at every stage.

    The conclusion should remain measured. The value of a long-horizon agent does not depend on a particular model name. It depends on whether the training recipe makes success rate and cost explainable, verifiable, and affordable within an engineering workflow.

    When evaluation criteria are consistent, cost accounting is honest, and permission and testing gates are in place, reinforcement learning can improve software development productivity instead of producing a high score that nobody else can reproduce.