AI Agent Evaluation in Practice: Four Approaches and How to Choose
A practical guide to putting AI agent evaluation into production. It explains why agent evaluation ultimately comes down to engineering rather than the model itself, separates the design layer (datasets, metrics, judges) from the execution layer, and walks through the four stages of the execution pipeline: batch dispatch, per-case judging, score aggregation, and version comparison. It then compares four ways to run evaluation—manual, script-based, framework-based, and platform-based—along with their trade-offs in effort, reproducibility, and scalability, shows how to classify evaluation tools by role rather than hype, and offers a decision checklist plus answers for common scenarios such as early-stage manual evaluation, fast iteration with regression testing, multi-team governance, data that cannot leave the internal network, and comparing multiple models or frameworks.
1. The Methods Are Ready, So Why Is There Still No Evaluation Report?
Many people say that agent evaluation ultimately comes down not to the model but to engineering. The reason lies in how agents work: their output is hard to judge against a single reference answer. A question-answering task can be compared against a ground truth, but an agent goes through a multi-step process of “reason, call a tool, observe the result, reason again,” often accompanied by external actions such as data queries, file operations, and API calls. Given the same input, a different prompt version, a different model, or a different tool description can produce a completely different trajectory. For this reason, agent evaluation must address three things:
- Is the behavior usable: Did the task complete, were the call parameters correct, did the agent overstep its bounds?
- Is the path reasonable: Are there redundant steps, ineffective loops, or timeouts and retries?
- Is the result reproducible: Pin down the versions of the prompt, model, tools, code, and data, so that any difference can be attributed clearly.
A related and common problem is tool selection. There are many open-source agent development frameworks, they update quickly, and they serve different purposes, so teams often compare them repeatedly yet still cannot settle on one. In reality, selection is just one input to the evaluation pipeline. Without repeatable evaluation, switching frameworks becomes an experiment with no control group—the gains are hard to explain and there is no basis for rollback. This is exactly the question of “what does evaluation actually run on” that this article sets out to answer.
Evaluation can be split into a design layer and an execution layer that sits outside that design:
| Layer | Question it answers | Corresponding content |
|---|---|---|
| Design layer | What to test, how to measure, who judges | Evaluation sets (stratified sampling, difficulty mix, version management); metrics (effectiveness, efficiency, safety, cost); scorers (rule checks, model judges, human review) |
| Execution layer | How to dispatch in batches, judge case by case, aggregate scores, compare two versions | Evaluation orchestration and scheduling, judging pipeline and result retention, aggregation statistics and reporting, version comparison and regression gating |
The design layer determines whether an evaluation is meaningful; the execution layer determines whether it can be repeated reliably. No matter how detailed the methodology is, once the execution layer is missing, the evaluation set is just a static document, evaluation degrades into scattered manual spot checks, and there is no quantitative evidence strong enough to support a release decision. So this article first clarifies “which ways evaluation can be run,” then explains how to classify tools, and finally gives selection answers you can apply directly.
2. The Four Key Stages of the Evaluation Execution Pipeline
Take a “regression evaluation comparing Model A against Model B” as an example. The execution pipeline needs to complete four steps in sequence:
- Batch dispatch: Read the evaluation set, send inputs, expected results, and scoring requirements to the system under test at a certain concurrency level, while collecting inputs and outputs, tool-call trajectories, latency, and token usage.
- Per-case judging: Follow different branches depending on the scorer type. Rule checks compare structured fields directly; a model judge scores case by case against the rubric and gives reasons; human review enters a pending-review queue.
- Score aggregation: Aggregate by metric dimension and by case group, distinguishing overall values from subgroup values, and keep every detail so it can be traced back.
- Version comparison: Align the old and new results along the same dimensions, compute the difference per dimension, and pinpoint the specific cases where “the new version got better” and “the new version got worse.”
An illustrative configuration (fields are examples, not any product’s API):
name: customer-assistant-v3-vs-v4dataset: cases/customer-v2.jsonl # contains stratified labels: difficulty, scenario, multi-turntargets:- name: v3endpoint: /v1/chat # model service under test, connected via a unified protocol- name: v4endpoint: /v1/chatjudges:- type: rule # required fields, format, tool-call parameter validation- type: llm # model judge, scores case by case against the rubricrubric: rubrics/helpfulness.md- type: human # sampled human review, used to calibrate automatic judgingsample_rate: 0.1sampling:temperature: 0seed: 42 # fix randomness when the model supports itrepeats: 3 # resample high-variance cases to test stabilityguardrails:min_pass_rate: 0.85 # block release if below the thresholdmax_regression_cases: 5 # maximum allowed number of "regression cases"output:results: runs/{name}/results.jsonlreport: runs/{name}/report.html
import asyncio, json, os, timefrom pathlib import Pathasync def call_target(cfg, case, attempt=0):"""Unified call protocol for the system under test; timeout retries with exponential backoff and a cap."""try:resp = await request_with_timeout(cfg["endpoint"], case["input"])assert_no_pitfall(resp) # pitfall detection: timeouts, empty responses, fallback canned replies, etc.return respexcept Exception:if attempt >= 2:return {"error": "call_failed"}await asyncio.sleep(2 ** attempt)return await call_target(cfg, case, attempt + 1)async def run_target(cfg, cases, sem):rows = []async def one(case):async with sem: # concurrency limit protects downstream servicest0 = time.time()outs = [await call_target(cfg, case) for _ in range(cfg["repeats"])]return {"case_id": case["id"],"tags": case["tags"],"outputs": outs,"verdicts": await judge_all(case, outs), # rule + model judge"latency_ms": (time.time() - t0) * 1000,}rows = await asyncio.gather(*(one(c) for c in cases))out = Path("runs") / cfg["name"]out.mkdir(parents=True, exist_ok=True)with open(out / "results.jsonl", "w", encoding="utf-8") as f:for r in rows:f.write(json.dumps(r, ensure_ascii=False) + "\n")return rowsasync def main():cases = load_jsonl("cases/customer-v2.jsonl")sem = asyncio.Semaphore(int(os.getenv("MAX_CONCURRENCY", "8")))cfg = load_config("eval.yaml")results = {t["name"]: await run_target(t, cases, sem) for t in cfg["targets"]}report = compare(results) # overall score, per-dimension score, significantly worse casesassert report["pass_rate"] >= cfg["guardrails"]["min_pass_rate"]assert report["regression_count"] <= cfg["guardrails"]["max_regression_cases"]Path("runs/report.html").write_text(render_html(report), encoding="utf-8")if __name__ == "__main__":asyncio.run(main())
These four stages map to four tool roles:
- Execution and scheduling: responsible for dispatch and retries.
- Judging and scoring: responsible for rule checks and model judges.
- Experiment tracking: responsible for recording versions and results.
- Collaboration platform: responsible for centralized display, review, and gating.
The selection advice and scenario recommendations that follow can all be aligned against these four roles.
3. Four Evaluation Approaches: Principles, Limits, and Improvements
Once the evaluation set and scoring rubric are finalized, the goal is usually to move evaluation toward scale while keeping human effort focused on where judgment is genuinely needed. In practice within companies, the common approaches fall into four categories: manual, script-based, framework-based, and platform-based. They are not substitutes for one another but four ascending steps in terms of human effort, reproducibility, and scalability.
3.1 Manual Approach: A Chat Window Plus Human Eyes
How it works: Paste the evaluation-set inputs one by one into a chat interface, judge pass or fail against the rubric, and record the results in a table. A more disciplined version gives every reviewer a unified template with fields such as case ID, input, output, the rubric point that was hit, the verdict, the failure reason, and notes.
Value: This is the approach closest to real usage, and it is the only source for calibrating the rubric. Ambiguities in the rubric often surface only through actual judgment—for example, whether “a helpful answer” means complete information, a clear conclusion, or performing the action the user expected. Without judging a batch of samples by hand, the vague wording in the rubric is hard to spot. So manual review at the start is not inefficient; it is a necessary investment.
Failure mode: The problems almost all come down to “repetition.” In the first round reviewers are energetic and judge relatively strictly; on a rerun, facing the same cases, people unconsciously loosen their standards, and toward the end they “give it a rough glance and pass it.” The result is that the pass rate for the new version goes up, but in reality it is an illusion caused by drift in the judging standard. On top of that, manual review cannot run in parallel and struggles to support regression evaluation.
Improvements:
- Clarify the purpose: Position pure manual review as “calibrating the rubric, training reviewers, and establishing an annotation baseline,” not as the workhorse for daily regression evaluation.
- Build a gold standard: Have reviewers annotate independently in pairs, and check disagreements using metrics such as simple agreement rate and an inter-annotator agreement coefficient. Categories with high disagreement go back into the rubric for redefinition; once stable, they are consolidated into a gold-standard dataset to serve as a reference for calibrating automatic judging.
- Keep judging consistent: First run a batch of anchor cases together to align the standard; break the rubric down by dimension, with each rubric point stating the observable conditions for “satisfied/not satisfied”; rotate review in batches and cross-check a certain proportion of double-annotated samples.
3.2 Script-Based Approach: A Self-Built Batch Pipeline
How it works: Write a general-purpose program that reads an evaluation set in a format such as JSONL or CSV, loops over the system under test and saves the results, then calls a scoring function to grade them and finally aggregates everything into a structured report. The run_target function in the example above is the minimal working form.
Advantages: Fully controllable, with the lowest cost to change. Swapping the system-under-test interface, changing the scoring logic, or adding a new statistical dimension is just a few lines of code, with no dependency on any external system.
Weaknesses: Teams quickly find themselves reinventing the wheel in the same few places, including:
- Data loading and resume-from-checkpoint;
- Concurrency control and failure retries;
- Cost estimation and metric calculation.
Every team’s implementation quality varies. A common pitfall is computing only the average score while failing to retain per-case details, so that later, when you want per-dimension analysis or version comparison, you can only rerun everything.
Engineering essentials:
- Decouple calling from judging: Split the system-under-test call and the scoring logic into two steps. When scoring rules change, there is no need to call the model again, saving substantial time and call costs.
- Fix the sampling parameters: Model-based systems under test have sampling randomness. Write temperature, random seed, and number of samples into the configuration and note them in the report; otherwise a “two-version comparison” may just be comparing two random fluctuations.
- Exception and pitfall detection: Beyond call failures, identify pitfall situations such as empty replies, refusals to answer, truncation, infinite loops, and repeatedly calling the same tool, and count them separately. If they are all lumped together as “failures,” the real capability gap gets masked.
- Traceable identity and grouping: Give each case a stable ID plus scenario and difficulty labels, and give each run a unique run ID, writing results row by row to a JSONL file, which naturally supports resume-from-checkpoint and incremental comparison.
- Define the statistics up front: Compute pass rates grouped by label and observe whether a few hard cases are dragging down overall performance, so that structural weaknesses are not hidden by the average score.
3.3 Framework-Based Approach: Leveraging a Mature Evaluation Framework
How it works: Use a community-standard evaluation framework or open-source evaluation library. These usually already have the following built in:
- Dataset loading;
- A unified interface for calling the system under test;
- Common metrics and model judges;
- Experiment recording and report generation.
Once the team connects its system under test and evaluation set, it gets a fairly complete pipeline.
Advantages:
- Less redundant implementation, shortening the distance from “manual evaluation” to “repeatable evaluation.”
- Metrics that have been validated by the community, with definitions consistent with common industry standards, making reports easier to discuss across teams.
- Most support running inside the pipeline, with configurable switching between the system under test and the scoring scheme.
Risks: These risks basically all come from “coupling” and need to be identified in advance:
- Data format binding: Once the evaluation set is imported into a framework’s private format, the migration cost goes up.
- Mismatch in the system-under-test abstraction: A framework may assume the input and output are single-turn text Q&A, whereas agent evaluation needs trajectories, tool-call records, and multi-turn conversations, which may require you to extend it yourself.
- Mismatch in metric meaning: General-purpose metrics do not necessarily reflect the business goal—for example, a general relevance score is not equivalent to the “is the information accurate” that the business cares about.
- Implicit calls: Some frameworks automatically make another model call during the scoring stage, making scoring costs unpredictable and results irreproducible.
Principles for use:
- Keep the evaluation set’s master data in a framework-agnostic, general format, treating the framework only as a consumer;
- Connect the system under test through an adapter layer, rather than writing business code into the framework’s plugin system;
- Verify the framework’s implicit behaviors one by one, especially whether the model judge’s model version and parameters are pinned down.
3.4 Platform-Based Approach: An Evaluation Management Platform and Observability System
How it works: Use a centralized evaluation or observability platform to manage datasets, run records, result dashboards, and collaborative review together. There are two typical forms: a self-built internal evaluation platform; and a combination of general cloud product categories—such as object storage, message queues, container platforms, log services, and monitoring and alerting. Taking Baidu AI Cloud as an example, object storage can hold datasets and evaluation results, a container platform can run evaluation tasks, a message queue can distribute and buffer tasks, a log service can retain the judging process, and monitoring and alerting can set thresholds and fire alerts on regression metrics. Put together, these capabilities form a governable execution pipeline.
Advantages:
- Multi-person collaboration and review-task assignment;
- Results retained centrally, easy to trace and audit;
- Connection to the release process, with gates that can be set;
- Large-scale concurrency, permissions, and data security can be governed centrally.
Cost: The cost of building and maintaining it is high. Platformization often means the process becomes fixed, and when rules change it requires changes on the platform side.
Recommendation: Introduce a platform only once evaluation has become routine and requires multi-person collaboration or compliance record-keeping. And whether you build or buy, first define the four types of objects: datasets, run records, result metrics, and permission boundaries. Otherwise the platform will be built, but it will still be full of historical data with inconsistent definitions.
3.5 Comparing the Four Approaches
| Approach | Human effort | Reproducibility | Scalability | Main risk | Typical stage |
|---|---|---|---|---|---|
| Manual | High, grows linearly with the number of cases | Depends on reviewer state | Low | Judging-standard drift, fatigue | Rubric calibration, annotation baseline |
| Script | Medium at first, low afterward | Medium-high (depends on fixed configuration) | Medium | Reinventing the wheel, inconsistent statistics | Fast iteration, daily regression |
| Framework | Low | High (configurable, reproducible) | Medium-high | Abstraction mismatch, implicit behavior | Building the evaluation system |
| Platform | High during build, low during operation | High (central retention and gating) | High | Build and maintenance cost, rigid process | Multi-team collaboration, routine operation |
The key is to understand the progression. Manual work produces a “judgment standard,” scripts produce a “repeatable process,” frameworks produce “standardized capability,” and platforms produce “governance and collaboration.” Skipping an earlier level and jumping straight to a later one typically ends with the tools being complete but no one daring to use the results to make decisions.
4. How to Classify Tools: By Role, Not by Popularity
Evaluation-related tools can be grouped into five functional roles:
- Case and data management: building evaluation sets, deduplication, stratified labels, gold-standard maintenance, and dataset version management.
- System-under-test access: adapting agent development frameworks and model inference services, unifying interface protocols, timeout retries, concurrency control, and call-cost accounting.
- Judging and scoring: the three types of scorers—rule checks, model judges, and human review—plus multi-scorer fusion and consistency checking.
- Execution and scheduling: batch orchestration, concurrency and rate limiting, resume-from-checkpoint, and task queuing.
- Experiment tracking and reporting: version records, metric aggregation, group comparison, report generation, and gate validation.
When selecting, first confirm whether each category is already covered by an internal system, to avoid duplicate construction. A common mistake is conflating “agent development frameworks” with “evaluation tools”: a development framework solves how an application is built, while an evaluation tool answers whether the application is good; the two need to connect, but their responsibilities must not be confused.
5. A Selection Method: A Ready-to-Use Decision Checklist
The recommended approach is to complete selection in five steps:
- Map the existing pipeline: What form is the system under test (single-turn Q&A, multi-turn conversation, an agent with tool calls)? What is the size, format, and update frequency of the current evaluation set? Who runs the evaluation now, and how often?
- Inventory existing capabilities: Does the company already have a reusable logging and monitoring system, object storage, or task scheduling? Reuse existing capabilities first, and avoid building a separate facility just for evaluation.
- Identify the missing pieces: Compare one by one against the four stages from Section 3, and fill only what is genuinely missing.
- Set admission criteria: Including whether the system-under-test version can be pinned, whether per-case results can be retained, whether statistics can be grouped by label, whether comparable reports can be output, and whether data can stay inside the internal network.
- Validate on a small scale before scaling up: Pick 50–100 representative cases for a pilot, get two rounds running, and then decide whether to expand.
The corresponding admission conditions can be written as a checklist:
- Cases can be versioned, with the evaluation set managed separately from the code;
- The system under test is accessed through a unified protocol, so switching models or prompts only changes configuration;
- Each result retains the input, output, trajectory, latency, token usage, and the judging reason;
- Metrics can be viewed grouped by difficulty, scenario, and task type;
- Support for aligning and comparing results across two or more versions, outputting a list of “regression cases”;
- The judging process is reproducible, with the rubric and model-judge versions traceable;
- Support for gate thresholds, so that releases can be blocked when metrics fail to meet targets;
- Sensitive data is masked, with clear permission and audit boundaries.
6. Selection Answers for Typical Scenarios
Scenario one: Just getting started, evaluation still relies on manual work, and the evaluation set has a few hundred cases. Start with the script-based approach. Spend a day or two building a minimal batch pipeline, fix the sampling parameters, and output per-case results and grouped statistics. Keep manual review, but change its goal to calibrating the rubric and establishing a gold standard.
Scenario two: Fast iteration, prompts or models change every week, and regression evaluation is needed. Use a combination of scripts and a framework, and connect evaluation to the continuous integration process: a code commit triggers an evaluation task, the report is archived as an artifact, and a merge is blocked when metrics fail. The focus is on making “two-version comparison” a default action rather than an ad hoc task.
Scenario three: Multiple teams collaborate, evaluation needs are diverse, and unified governance is required. Introduce an evaluation management platform, or self-build one based on general cloud product categories: object storage for datasets and results, a container platform to run evaluation tasks, a message queue to distribute tasks, a log service to retain judging trajectories, and monitoring and alerting to watch regression metrics. Unify data definitions and permission boundaries, and avoid each team building its own isolated system.
Scenario four: High data-security requirements, and evaluation data cannot leave the internal network. Prefer components that can be deployed privately, or self-build inside the internal network based on a container platform. The model judge also needs to be assessed for whether it can be replaced by an internally deployed model, or fall back to a combination of rule checks and human review.
Scenario five: Need to compare multiple models or multiple framework options. The adapter layer of an evaluation framework delivers the most value here. Abstract the system under test behind a unified call interface, and reuse the evaluation set, scorers, and report templates, switching only the target under test, avoiding rewriting scripts for every comparison.
7. Closing: Tools Handle Execution, Not Definition
The essential difference among the four approaches is who takes on the evaluation step:
- Manual: humans handle all the judgment and produce the standard.
- Script: the machine executes and humans define the rules, producing a repeatable process.
- Framework: standardizes the rules and the process, lowering implementation cost.
- Platform: brings the process into a system of collaboration and governance.
No matter which one you choose, the three questions—what the evaluation set represents, what counts as a pass, and how the score is calculated—must still be answered by the team itself. Tools can only execute for you; they cannot rescue a poorly thought-through “what to test and how to judge.” So a sensible path to adoption is: first get the execution pipeline working with a minimal batch script, calibrate the rubric with human review, and then bring in frameworks and platforms as needed to handle collaboration and governance. Whether the judgment standard is mature is not measured by how complete the platform is, but by whether the difference between two evaluation runs can be clearly explained.
