Evaluating Research AI Agents: From Task Success Rates to Cost Governance

1. Why Research AI Agents Need Process-Oriented Evaluation

Traditional question-answering benchmarks mainly test whether a model can generate a correct response. Research work, however, rarely consists of a single text-generation step. Researchers may need to inspect data, understand code, configure an environment, run experiments, analyze errors, and organize the results into conclusions that others can review. A failure at any stage can undermine the credibility of the final answer.

Interactive terminal-based evaluation focuses on this continuous execution capability. The agent receives a research objective and operates within a defined computing environment, tool-permission set, and resource budget. It must ultimately deliver a verifiable result. This result might be a repaired program, processed data, a valid experimental output, or an analytical report with a complete evidence chain.

Such evaluations therefore test more than whether an agent “knows how to use a terminal.” They also examine task understanding, planning, tool operation, scientific computation, error recovery, and result verification.

A success rate on a specific benchmark only describes performance under that benchmark’s task distribution and runtime conditions. It should not be treated as a direct measure of overall research capability. To make evaluation useful for technical decisions, scores must be analyzed together with task composition, execution budgets, acceptance procedures, and operating costs.

2. Define the Evaluation Target Beforehand

The performance of a research agent is usually determined jointly by the model, execution framework, tool environment, and verification mechanism. Even when the same model is used, changes to context organization, time limits, or accessible data can significantly affect task completion.

The first step is therefore to define exactly what is being measured.

If the goal is to compare models, prompt structure, tool permissions, computing resources, and retry policies should be kept as consistent as possible. If the goal is to select a deployable system, different solutions may use different execution strategies, but their resource consumption and configuration differences must be disclosed. The first approach emphasizes controlled variables; the second emphasizes end-to-end delivery efficiency. Their conclusions should not be mixed.

A useful capability breakdown includes:

1. Task understanding: Can the agent identify the inputs, constraints, objectives, and acceptance criteria?
2. Execution planning: Can it decompose a complex objective into verifiable steps?
3. Environment operation: Can it correctly read files, call programs, and manage intermediate artifacts?
4. Scientific reasoning: Can it select an appropriate method and correctly interpret experimental behavior?
5. Self-verification: Can it detect anomalies, recognize insufficient evidence, and initiate additional checks?
6. Delivery quality: Can it submit a complete, reproducible result in the required format?

This decomposition helps avoid a common mistake: treating a well-formatted and fluent report as proof that the research task has actually been completed.

3. Building Research Tasks with Sufficient Discriminative Power

High-quality tasks should originate from technical problems found in real research workflows. They must still be de-identified, scoped, and standardized so that different systems start from comparable conditions.

A coverage matrix can include the following task categories.

Data Processing

The agent may need to identify missing values, duplicate records, inconsistent units, or mismatched fields, then produce a data artifact that satisfies predefined constraints. Evaluation should not only check whether an output file exists. It should also verify data integrity and whether the processing rules were applied correctly.

Computation and Numerical Validation

Tasks may require the agent to implement a computational procedure, inspect convergence behavior, or explain why results change under different parameters. For floating-point calculations, an acceptable error range should be defined in advance. Numerical outputs should not be judged through character-by-character comparison.

Experiment Reproduction

The task can provide a method description, input data, and partial execution materials, requiring the agent to reconstruct key steps. The objective is not necessarily to reproduce one exact number mechanically. Instead, evaluation should confirm that the method, parameters, data split, and evaluation procedure meet the requirements.

Program Diagnosis

The agent may be given a failed run, abnormal output, or performance degradation and asked to identify and fix the problem. Hidden tests should include boundary inputs so that the system cannot pass by making a narrow change tailored only to visible examples.

Result Interpretation

The agent must form conclusions from actual execution results and describe the limits of the evidence. When statistical support is insufficient or an experiment is incomplete, it should express uncertainty clearly rather than inventing a plausible-sounding conclusion.

Task difficulty should also be tiered. Short workflows are useful for locating basic capability gaps, while multi-stage tasks are better for observing planning, context management, and error recovery. Overall success rates are meaningful only when the task distribution contains a reasonable range of difficulty.

4. Design Acceptance Rules as a Trustworthy Evidence Chain

The central question in research evaluation is not what the agent claims to have completed, but what the evaluation system can independently prove it completed.

Acceptance can be divided into four layers:

1. Artifact completeness: Confirm that the required files, data, or reports were generated and can be read successfully.
2. Structure and constraint validation: Check field types, record counts, value ranges, directory structures, and required experiment records.
3. Semantic correctness: Determine whether the result satisfies the task objective. For example, verify that a data transformation preserves required information, that a repaired program passes undisclosed tests, or that a numerical result falls within the permitted error range.
4. Scientific validity: Check for data leakage, inappropriate evaluation procedures, and conclusions that go beyond the available evidence. Some tasks require expert review at this level and cannot be fully evaluated automatically.

Acceptance materials should be isolated from the agent’s workspace so that the system under test cannot read reference answers or modify the evaluation logic. For executable tasks, record key file hashes or summaries, runtime parameters, and verification logs so that the source of each conclusion can be traced.

5. Metrics Beyond Task Success Rate

Task success rate can be defined as:

Task success rate = Number of tasks that pass acceptance ÷ Total number of valid evaluation tasks

This metric is intuitive, but reporting only one overall score can hide differences in task difficulty and resource conditions. A more complete report should include the following.

Category-Level Success Rates

Report performance separately for data processing, program diagnosis, numerical computation, and other categories. This helps identify capability gaps.

Completion Time

Separate model waiting time, tool execution time, and end-to-end duration. Otherwise, differences in the runtime environment may be mistaken for differences in reasoning ability.

Failure Types

Classify failures as misunderstanding, environment error, computation error, timeout, missing verification, or security violation. This supports targeted improvement.

Result Stability

Run stochastic tasks multiple times independently and observe both the success proportion and its variation. The number of repetitions should be reported. Under appropriate statistical assumptions, confidence intervals can also be provided.

Resource Consumption

Record the costs of model calls, computation, storage, and human review so that technical performance can be evaluated against an actual budget.

For partially completed tasks, staged scores can help with diagnosis. However, partial credit must be clearly distinguished from complete task success. Otherwise, an average score may conceal delivery failures.

6. Comparing Cost and Effectiveness Fairly

A low per-call price does not necessarily produce a low delivery cost. A solution may have inexpensive individual calls but require more retries, longer tool execution, or more frequent human intervention, resulting in a higher total cost.

A consistent accounting boundary is recommended:

Total evaluation cost = Model invocation cost + Computing and storage cost + Retry cost + Human review cost

If some costs cannot be measured accurately, they should be explicitly listed as excluded rather than assumed to be zero. Cost comparisons should use the same task set, pricing basis, and resource conditions. Reductions should not be calculated across incompatible statistical scopes.

A more useful business metric is:

Cost per successful task = Total cost of all valid evaluation attempts ÷ Number of successful tasks

The numerator must include spending on failed attempts. Otherwise, the actual delivery cost will be underestimated. When no task succeeds, a finite cost per successful task should not be reported.

It is also useful to compare performance across multiple budget settings rather than selecting a single operating point. Gradually increasing the budget and observing whether success rates continue to improve can reveal where marginal returns begin to decline. Production systems should generally select a configuration that meets quality requirements within a controlled budget, rather than simply pursuing the highest possible score.

7. Runtime Environments and Security Isolation

Research agents may execute programs, write files, or install dependencies. Evaluation environments must therefore follow the principle of least privilege.

Each task should use an isolated working directory and a recoverable environment. Computing resources, runtime duration, and file-access scope should be limited. For tasks that do not require network access, external connectivity should be disabled. When external resources are necessary, the permitted scope should be defined clearly and access should be logged.

Input data should be separated from acceptance logic, and sensitive information should not be placed in the task environment. Logs should also be sanitized to prevent credentials or personal information from being exposed during execution recording.

Task files and program outputs may contain misleading instructions. The evaluation framework should treat these materials as data to be analyzed, not as instructions authorized to modify security rules or expand permissions.

If isolation fails, the acceptance program malfunctions, or infrastructure problems occur, the incident should be marked and investigated separately. It should not automatically be classified as an agent capability failure.

8. Turning Evaluation Results into Engineering Improvements

A practical process can be organized into five steps.

First, define the business objective. Identify the most important task types, the minimum acceptable success rate, and the maximum cost and time limits.

Second, build a small, high-quality seed task set. Prioritize reliable acceptance procedures, then gradually expand coverage.

Third, fix the environment and execution protocol. Save the task version, system configuration, resource limits, and execution records for every experiment.

Fourth, perform layered analysis. If failures are concentrated in environment preparation, improve environment instructions and dependency management. If they occur mainly during numerical validation, strengthen the checking mechanisms. If excessive retries are the main issue, improve termination conditions and budget controls.

Finally, convert representative failures into regression tests while maintaining an independent test set. This prevents the system from overfitting to known tasks.

The value of evaluating research AI agents is not to produce an isolated ranking. It is to establish a quality system that is explainable, reproducible, and traceable. Only by analyzing task difficulty, evidence reliability, success rates, resource costs, and security boundaries together can evaluation results provide a sound basis for model selection and production engineering decisions.