Methodology and Engineering Practices for Evaluating Agents on Long-Horizon Tasks

Why Long-Horizon Tasks Are the Key Test for Agents

Short-turn question answering is relatively easy to evaluate: provide a question and compare the answer. However, the real value of an AI agent lies in long-horizon tasks, where it must interact with an environment across many turns, execute commands, read outputs, correct mistakes, adjust its direction, and ultimately complete an end-to-end goal. The challenges include:

• Large state space: Every action changes the environment state, and errors accumulate.
• Multiple valid paths: The same goal can be reached through different execution paths, so evaluation cannot focus only on the final result.
• Diverse failure modes: Failures may come from command syntax errors, wrong direction, or giving up midway.

Therefore, an evaluation system designed for long-horizon tasks must measure three dimensions at the same time: whether the task is ultimately completed, whether the process is reliable, and whether the agent can recover after going off course. This article discusses the design principles and engineering practices behind such an evaluation methodology, based on publicly available agent evaluation work.

Designing the Evaluation Metrics

Net Improvement Rate: Quantifying “Usefulness” as a Single Number

Task success rate alone can be inflated by a few easy samples. A common alternative is the net improvement rate, which compares the task state before and after the agent acts. Only actions that produce real forward progress receive positive credit, while ineffective or destructive actions receive negative credit. The result is normalized into a comparable ratio.

A simplified calculation is shown below:

def net_improvement(before_score, after_score, max_score):
    """
    before_score: score of the task in its initial state
    after_score:  score of the task state after the agent acts
    max_score:    theoretical maximum score for the task
    returns a normalized net improvement rate, approximately in [-1, 1]
    """
    if max_score == before_score:
        return 0.0
    return (after_score - before_score) / (max_score - before_score)

This metric penalizes both “spinning in place” and “making things worse,” so it better reflects the agent’s incremental contribution in long-horizon tasks. In practice, the net improvement rate is often computed as a weighted average across a batch of tasks to produce a model-level score.

Command Execution Reliability: Bash Reliability

In long-horizon tasks, agents rely heavily on command-line interactions with the system. Command execution reliability measures how consistently the agent generates commands that are syntactically correct, use valid arguments, and produce parseable execution results. Common measurements include:

• First-attempt command success rate.
• Share of failures caused by syntax or argument errors.
• Share of failures that the agent can self-correct and retry successfully.

A low score here usually means the agent is unreliable in a real terminal environment. Even if the plan is correct, it may fail during execution.

Steerability: Recovering from the Wrong Direction

Another major risk in long-horizon tasks is going off course, where the agent keeps investing effort in the wrong direction. Steerability, also referred to as correction capability, measures whether the agent can quickly understand and adjust its subsequent actions when an external correction signal is given, such as a user pointing out the wrong direction or the environment returning abnormal feedback.

Evaluation typically includes a “correction injection” step: a directional prompt is inserted midway through the task, and the evaluator observes whether the agent converges back to the correct path over the following steps. This metric is especially important for interactive development scenarios.

Real Session Data: Keeping Evaluation Grounded

Offline test sets can easily lead to overfitting. A more credible approach is to collect real user sessions involving long-horizon tasks, then anonymize and label them as evaluation samples. In publicly reported evaluation work, the dataset can reach more than 17,500 real long-horizon task sessions, covering categories such as workflow processing, programming, and daily conversation.

Real data provides three main benefits:

1. Realistic distribution: Task difficulty, instruction ambiguity, and environment noise are closer to production.
2. Attributable failures: Real failure cases can inform better metric design.
3. Scenario-specific ranking: Results can be reported separately for work, programming, chat, and other categories, avoiding the problem of a single overall score hiding weak areas.

From Rankings to Model Selection: How to Read Evaluation Results

In evaluations of open-weight models, a typical result might look like this: a certain open-weight model ranks first among open models and thirteenth among all evaluated models, while also ranking first among open models in work, programming, and chat scenarios. Its overall net improvement rate is +5.1%, an increase of 4.2 percentage points over the previous generation, and it ranks near the top among open models on command execution reliability and steerability.

For engineering teams, such results carry at least three takeaways:

• Balanced performance matters more than a single standout area: Leading results across all three scenarios suggest a more balanced capability profile, making the model suitable as a general-purpose agent foundation.
• Generational improvement is meaningful: A 4.2 percentage point increase in net improvement rate indicates real progress in the ability to move long-horizon tasks forward, rather than improvement driven only by better phrasing.
• Sub-metrics are explainable: Strong command execution reliability and steerability directly correspond to stability and maintainability in real deployments.

It is important to emphasize that rankings are an input to model selection, not a conclusion. Teams should validate against their own task distribution:

Model Selection Validation Checklist (Example)
1. Select 50-100 internal business tasks and construct end-to-end sessions
2. Use a unified environment image and toolset, and fix random seeds
3. Track: task completion rate / net improvement rate / average interaction turns / recovery rate after correction
4. Classify failure cases by cause (planning error / command error / giving up)
5. Compare candidate models and produce a weighted decision matrix

Engineering Recommendations for Deployment

Use Guardrails to Improve Command Execution Reliability

Do not assume the agent will write the correct command on the first attempt. A practical approach is to add guardrails at the orchestration layer: perform lightweight syntax checks before command execution, parse exit codes and standard error output after execution, and feed structured error information back to the model to guide self-correction. This mechanism can significantly increase the proportion of effective commands.

Use Checkpoints to Reduce the Cost of Going Off Course

Long-horizon tasks should include staged checkpoints. After each sub-goal is completed, save a state summary. If the system detects that multiple consecutive steps have made no progress, or if a user correction is received, roll back to the most recent checkpoint and replan. This is more economical than restarting from the beginning and gives the agent more room to benefit from its correction capability.

Use Layered Evaluation to Guide Iteration

Evaluation can be split into three layers: unit level, which covers a single command or tool call; subtask level, which covers a verifiable sub-goal; and end-to-end level, which covers the complete task. The unit level helps locate defects, the subtask level measures compositional ability, and the end-to-end level reflects the real user experience. All three layers are needed to attribute changes in net improvement rate to specific parts of the pipeline.

Conclusion

The core of evaluating long-horizon agents is not to produce an impressive score, but to build a measurement system that is explainable, reproducible, and attributable. Net improvement rate answers whether the agent is useful. Command execution reliability answers whether it is stable. Steerability answers whether it can recover after going off course. When evaluation samples come from real sessions, metrics can be broken down by scenario, and conclusions can be translated into engineering measures such as guardrails and checkpoints, evaluation becomes more than a ranking exercise: it becomes a driver of productivity. Teams that are selecting or building an agent foundation should establish their own task sets and layered evaluation pipelines as early as possible, so that every model iteration can be guided by evidence.