Long-Horizon Agent Evaluation: From One-Shot Answers to Continuous Execution

Why One-Shot Answers Are Not Enough for Agent Evaluation

Traditional model evaluations often use one question and one answer as the basic unit. They typically focus on factual accuracy, reasoning ability, or code generation quality. These tests are easy to standardize and useful for quickly comparing basic model capabilities, but they do not adequately represent the complexity of real development and operational work.

Actual tasks are rarely as simple as “provide a question and return an answer.” A user may describe only a goal, such as fixing a feature, organizing a set of data, or checking whether a program can be deployed in a specific environment. An agent must clarify the objective and constraints, inspect the environment, break the work into steps, select tools, execute actions, read the results, and adjust its approach when errors occur. Whether the task succeeds depends on the reliability of the entire process—not just whether one response appears correct.

Long-horizon evaluation therefore focuses on a complete loop: understand the goal, create a plan, call tools, verify the result, correct deviations, and continue until the acceptance criteria are met. This is closer to how agents operate in real workflows, but it is also more difficult to control and reproduce.

What Capabilities Do Long-Horizon Tasks Evaluate?

1. Goal Understanding and Task Decomposition

Real-world requirements often include implicit constraints. When modifying code, for example, the agent may need to make tests pass without breaking existing interfaces. When processing files, it may need to preserve the original data or follow a specific directory structure. An evaluation should check whether the agent can identify the task boundaries, prerequisites, and risks—not merely whether it starts executing quickly.

Task decomposition is equally important. A complex objective should be divided into verifiable steps with sensible dependencies. If an agent omits a critical step or makes changes before confirming the prerequisites, later steps may succeed while the overall task still deviates from its actual goal.

2. Tool Use and Environmental Adaptation

Terminal-based tasks require an agent to understand its current environment and use commands, scripts, or other tools at the appropriate time. The key question is not simply whether the agent can call a tool, but whether it can take the right action based on the returned result.

For example, after a command fails, the agent should distinguish between insufficient permissions, an incorrect path, a missing dependency, and a problem in the program itself. Repeating the same operation without analysis is not reliable recovery. Environmental inspection, appropriate tool selection, and checks before and after execution all affect the final outcome.

3. Error Recovery and Iteration

Errors are common in long-running tasks. An evaluation that records only the first execution result may overlook whether an agent can recover. More meaningful observations include whether the agent can extract useful clues from an error message, avoid repeating ineffective approaches, revalidate the result after making a correction, and roll back or adjust its actions when side effects appear.

These capabilities determine whether an agent can continue making progress in a non-ideal environment. A single accidental success does not demonstrate stable execution. Similarly, a single failure should be analyzed together with its cause and the conditions of the task.

4. Final Results and Process Quality

Process performance and final results should be evaluated separately. An agent may eventually complete a task after many ineffective attempts, or it may follow a concise process but fail an important acceptance condition. Looking only at completion rates can hide efficiency and safety problems, while looking only at the process may overlook whether the deliverable is correct.

A more complete evaluation should consider result accuracy, execution cost, operational safety, reproducibility, and compliance with constraints. For tasks involving file modifications or script execution, evaluators should also check for unauthorized changes, data loss, and temporary artifacts that were not cleaned up.

How to Interpret Benchmark Scores

Different benchmarks cover different types of tasks, so their scores should not be combined into a single ranking of overall capability. Terminal-operation evaluations emphasize environmental interaction and continuous execution. Code-repair benchmarks focus primarily on defect localization and patch quality. Knowledge-work evaluations examine multi-step information processing. Long-context evaluations test whether a model can retrieve and integrate relevant information from large volumes of text. Advanced mathematics and safety evaluations emphasize formal reasoning and risk identification, respectively.

Even when two benchmarks report percentages, their task sizes, success criteria, execution environments, prompting methods, and permitted tools may differ. Score differences can be interpreted only when the test conditions are comparable. If a result comes from a particular configuration or may be affected by data contamination, it should be treated with additional caution. It should not be used to make a broad claim about overall superiority in a task category.

A more reliable way to read benchmark results is to ask three questions:

• What is being measured? Is the task relevant to real work, and are the acceptance criteria clearly defined?
• Are the test conditions consistent? Did the models use the same tools, context, execution time, and retry rules?
• Are the results stable? Were the tasks run multiple times, and were failure types and variation ranges reported?

Scores become useful for decision-making only after this context is understood. Otherwise, a single number may mix together task difficulty, environment configuration, and random variation.

Why Long-Context Results Deserve Separate Analysis

Long-context capability does not simply mean that a model can accept more text. In practical tasks, the model must also locate key facts within a large amount of material, distinguish similar information, retain constraints, and use the relevant details correctly in later reasoning or actions. As context length increases, information redundancy, distractors, and relationships across distant sections make processing more difficult.

Long-context evaluations should therefore measure more than input size. They should consider how difficult it is to locate key information, whether evidence is distributed across multiple sections, whether several sources must be combined, and whether the answer accurately cites the relevant facts. If a test only makes the input longer without testing retrieval and synthesis, its conclusions may be limited.

For engineering work, long-context capability can help agents handle large codebases, lengthy requirements documents, and multi-turn task records. It does not replace information management, however. Important constraints should be stored in a structured form where possible, and key decisions should be documented. Before executing irreversible operations, the agent should verify the target and impact scope again. Even when a model can process long inputs, clear task state and explicit acceptance criteria remain necessary.

Recommendations for Building an Internal Evaluation Set

Extract Scenarios from Real Tasks

Enterprises and development teams can derive evaluation tasks from common workflows instead of relying exclusively on public benchmarks. Scenarios may cover code inspection, configuration troubleshooting, data processing, log analysis, and document generation. Sensitive information should be removed, and each task should clearly define its objective, environmental prerequisites, and completion conditions.

Define Explicit Acceptance Criteria

Every task should have a verifiable outcome, such as specified tests passing, an output file meeting structural requirements, or a configuration reaching the expected state. When fully automated evaluation is not possible, human review can be used. In that case, the review criteria should be documented to reduce subjective differences between evaluators.

Record Both Results and Process

In addition to final success rates, teams should record execution steps, tool-call counts, retries, error categories, task duration, and any unauthorized actions or side effects. Process data helps identify the source of a problem. The agent may have misunderstood the task, used a tool incorrectly, lacked sufficient environmental information, or encountered unclear acceptance criteria.

Run Repeated Tests and Analyze by Category

Agent tasks involve a degree of randomness, so a single run should not be treated as a stable conclusion. Important tasks should be executed multiple times and analyzed by task type, failure mode, and difficulty level. Evaluation reports should also document environment versions, tool permissions, time limits, and scoring methods to support reproduction and future comparison.

From Evaluation Results to Production Deployment

Benchmark scores are useful for identifying capability trends and screening candidate solutions, but they should not be treated as direct evidence of reliability in production. Real systems also require permission isolation, operation auditing, data protection, emergency stopping, and human approval. High-impact actions—such as deleting data, publishing changes, or modifying production configuration—should have clear authorization boundaries and confirmation procedures.

Deployment can use graduated automation. Low-risk, easily verifiable steps may be executed automatically. Operations with a broader impact should require human confirmation. When the system cannot make a reliable determination, it should stop and request additional information. Recording execution traces and preserving rollback capabilities can improve efficiency while limiting risk.

Ultimately, the value of long-horizon evaluation is not to declare one model the “strongest.” Its purpose is to reveal whether a system can reliably complete objectives through sustained interaction. When selecting and using an agent, teams should consider task types, error-recovery capabilities, context requirements, and safety constraints, and continue validating the system against real business tasks. These findings are more representative of practical engineering value than any single benchmark score.