AI Agent Evaluation: Reliability, Correctability, and Net Improvement Rate

    Why is evaluating AI agents so hard? This article breaks down the gap between static benchmarks and long-horizon agent evaluation, then presents a practical four-dimension framework built on task realism, fine-grained metrics, layered comparison, and improvement measurement. You'll learn how to define Shell execution reliability and correctability, how to compute a Net Improvement Rate with confidence intervals to decide whether a model upgrade is worth shipping, and how to build a reproducible internal evaluation pipeline—from a 20-task smoke set to a 200+ task release gate. It also covers risk control, sandboxing, common pitfalls, and a minimal implementation you can start with today.

    AI Agent Evaluation: Reliability, Correctability, and Net Improvement Rate

    Evaluating AI agents is hard—not because running a few examples is difficult, but because task realism, environment state, failure attribution, and upgrade payoff are hard to compare directly. Static leaderboards reflect some capabilities, but they rarely cover the dynamic interaction, side-effect control, and self-correction of long-horizon agents.

    The conclusion up front: evaluation should follow a four-dimension framework—task realism, fine-grained metrics, layered comparison, and improvement measurement. On metrics, be explicit about Shell execution reliability and correctability. On payoff, use a computable Net Improvement Rate with confidence intervals to decide whether a model upgrade is worth releasing.

    1. Why It’s Hard: Static Benchmarks vs. Long-Horizon Evaluation

    Static benchmarks typically provide fixed inputs, fixed outputs, and one-shot scoring—well suited to single-turn tasks. Long-horizon agents must operate continuously across file systems, terminals, browsers, tool calls, and multi-turn state. The differences come down to four dimensions:

    • Task realism: Static benchmarks are usually isolated problems; long-horizon evaluation needs real workflows, dependencies, and side effects.
    • Environment dependencies: Static benchmarks are usually isolated, whereas agent evaluation is affected by container images, permissions, network, and residual state.
    • Feedback loop: Static benchmarks only look at the final answer; agent evaluation must also look at intermediate commands, retries, correction, and human takeover.
    • Payoff comparison: Static benchmarks look at absolute scores; upgrade evaluation must look at net improvement over the baseline and the risk of regression.

    2. The Four-Dimension Framework: From “It Runs” to “It’s Trustworthy”

    1. Task realism: Tasks should come from real scenarios or high-quality constructed scenarios, each with a clear goal, constraints, initial state, and acceptance criteria. Cover common cases, long-tail cases, failure recovery, permission denial, and missing data.
    2. Fine-grained metrics: Don’t rely on a single overall score. Break it down into reliability, correctability, efficiency, safety, and cost.
    3. Layered comparison: Separate public leaderboards, internal regression sets, and online shadow tasks. Model, prompt, tool permissions, and environment versions must be recorded so that tool changes aren’t mistaken for model improvements.
    4. Improvement measurement: Whether an upgrade is effective can’t be judged by average score alone—compute the Net Improvement Rate, regression rate, and confidence interval.

    3. Key Metric: Shell Execution Reliability

    Shell execution reliability can be defined as: in a controlled sandbox, the probability that an agent completes command execution and file operations according to task constraints while meeting success criteria, permission boundaries, and stability requirements. A workable definition:

    Inputs include task ID, task description, initial file system, environment image, timeout, permission policy, resource quota, allowed network policy, and success criteria. Outputs include the action trace, each command, exit codes, stdout, stderr, artifact file hashes, a final-state snapshot, and audit logs.

    Success conditions must all hold simultaneously: complete the target task within the timeout; key command exit codes match expectations; acceptance artifacts exist and pass validation; no unauthorized access, out-of-bounds deletion, or writes to sensitive directories; and no destructive state requiring manual repair. Failure conditions include: timeout, command syntax error, missing dependency, permission denial, error exit, wrong artifact, repeated invalid attempts, mistaken deletion, mistaken writes, unreachable external network dependency, and non-reproducible results.

    Timeout, permissions, and environment dependencies must be fixed as parameters—for example, per-command timeout, total task timeout, write-permission directory whitelist, read-permission scope, network toggle, dependency package versions, and image digest. Failure types are best categorized as: timeout, syntax, dependency, permission, logic, side-effect, environment-drift, and non-reproducible. Measurement metrics include task success rate, command-level success rate, first-attempt success rate, average attempts, P95 completion time, dangerous-action rate, and manual-intervention rate.

    4. Key Metric: Correctability

    Correctability answers whether an agent can detect, locate, and fix its own errors after making them—not just perform well on the happy path. Evaluation should actively inject deviations: missing files, wrong permissions, command failures, network interruptions, test failures, or negated user instructions.

    Metrics can include error detection rate, correction success rate, mean time to recovery, repeated-error rate, manual-takeover count, and unauthorized-correction rate. If an agent continues executing high-risk commands after an error, record it as a serious correctability failure. If it can stop, explain the cause, roll back, or request minimal permissions, its correctability is high.

    5. The Core Payoff: How to Compute Net Improvement Rate

    The Net Improvement Rate measures how much an upgraded model improves over the baseline across a task set. Let the baseline model be B, the upgraded model be U, the task set be T, and the total number of tasks be n. For each task t, define a success function s(m, t) that returns 1 for success and 0 for failure.

    • Baseline success rate: pB = Σs(B, t) / n
    • Upgrade success rate: pU = Σs(U, t) / n
    • Net Improvement Rate: N = pU − pB

    For multi-dimensional weighting, define N = Σ w_k (pU_k − pB_k), where w_k is the weight for each metric and the weights sum to 1.

    Sample size: A minimum smoke set of at least 20 tasks, each repeated 3 times; a pre-production regression set of at least 100 tasks, each repeated 3 times; for a formal release, at least 200–500 tasks, each repeated 5 times, stratified by scenario. Success determination should be as automated as possible, with dual review or sampling spot-checks where needed.

    Significance: Don’t rely on point estimates alone. Apply paired bootstrap or similar methods on task-level results to compute a 95% confidence interval. The upgrade can be considered effective when the lower bound of the confidence interval is greater than 0 and the Net Improvement Rate exceeds a preset regression threshold. If any key safety metric drops—even when the average Net Improvement Rate is positive—release must be blocked.

    6. From Public Leaderboards to an Internal Reproducible Pipeline

    Public leaderboards are useful for initial screening, but when citing them you must record the leaderboard name, version, date, task set, and scorer version to avoid disputes over timeliness and reproducibility. The internal pipeline should run through these steps:

    1. Freeze the task set: Establish a taskset_version with task sources, acceptance scripts, and boundary conditions.
    2. Pin the environment: Use immutable images and record image digests; fix dependency versions, tool versions, and permission policies.
    3. Set random seeds: Fix the seed for tasks that require randomness and record the candidate set.
    4. Set repetition counts: At least 3 times for key tasks; 5 times recommended for formal evaluation.
    5. Baseline run: Run the baseline model in the same environment with the same tool permissions.
    6. Candidate run: Run the upgraded model or new policy—without changing the evaluation scripts on the fly.
    7. Log retention: Save commands, exit codes, stdout, stderr, file hashes, timing, error types, and approval records.
    8. Compute metrics: Generate success rate, Net Improvement Rate, confidence interval, regression rate, safety failure rate, and P95 latency.
    9. Regression decision: Set release gates—for example, zero severe safety failures, key-task regression within threshold, and Net Improvement Rate confidence-interval lower bound greater than 0.
    10. Release or roll back: If it passes, roll out gradually; if it fails, keep the report and roll back the configuration.

    7. Risk Control and Boundary Conditions

    Agent evaluation involves command execution and must run in a sandbox using least-privilege accounts—never run directly in production or with high-privilege accounts. Data should be desensitized so logs don’t leak keys, tokens, user privacy, or sensitive paths. Network access should be off by default or whitelisted; dangerous commands should have blocking policies configured; all evaluation runs should keep audit logs and set resource limits to prevent infinite loops, full disks, or external-dependency abuse.

    8. Common Mistakes and Metric Selection

    Common mistakes include: running only public leaderboards with no internal regression; task sets that are too small; success determination based on subjective human scoring; ignoring model regression and side effects; container-image drift; not recording random seeds; no confidence intervals; mistaking tool-permission changes for model-capability gains; and looking only at averages while ignoring long-tail failures.

    When selecting metrics, rank them by risk: safety and correctability first, then core-task reliability, then efficiency and cost. For high-value tasks, establish dedicated acceptance criteria; for exploratory tasks, expand coverage, but don’t use them as the sole basis for release.

    9. A Minimal Implementation

    A practical minimal evaluation project can start with 20 high-value tasks: 10 routine tasks, 5 failure-recovery tasks, and 5 permission or error-injection tasks. Each task is repeated 3 times and run in the same sandbox image under least privilege.

    The acceptance criteria are: logs are complete and traceable; success determination is automated; and the output includes baseline success rate, upgrade success rate, Net Improvement Rate, 95% confidence interval, and failure classification.

    In the expansion phase, grow to 100 tasks and add stratification; in the release phase, grow beyond 200 tasks and add regression thresholds and safety gates. This approach lets you start fast while gradually building a traceable, recomputable agent evaluation system.