Evaluating AI Agents: Benchmarks, Costs, and True Capabilities

I. From “Answering” to “Doing”: What AI Agents Actually Change

In the past, interacting with large language models felt like talking to an armchair strategist: you asked questions, and it provided well-structured answers, but you still had to do the actual work yourself. The value boundary of the model stopped at suggestions and drafts.

A new generation of flagship models is rewriting this boundary. Empowered by agent technology, models are starting to get their hands dirty. Their capability stack can be roughly divided into five layers:

• Task Planning: Breaking down ambiguous goals into executable sequences of steps.
• Tool Invocation: Selecting external capabilities such as APIs, code interpreters, and retrieval systems as needed.
• Environment Operation: Directly driving real environments like browsers, terminals, and office software.
• Result Self-Verification: Checking output correctness through methods like screenshot comparison or execution receipts.
• State and Memory Management: Maintaining intermediate conclusions throughout long-chain tasks.

Consequently, tasks like opening software, filling out forms, organizing data, scheduling, scraping web pages, analyzing data, writing and running code, and even self-checking for page errors can now be executed end-to-end by the model. The working paradigm has shifted from “request-response” to a closed loop of “perceive—plan—execute—verify”:

# Agent execution loop (illustrative, not runnable code)
state = TaskState(goal="Organize this week's customer follow-up records and generate a summary report")
for step in range(MAX_STEPS):
    thought = llm.plan(state)                    # 1. Plan: decide the next step
    action  = llm.select_tool(thought, toolset)  # 2. Act: select tool, provide parameters
    result  = executor.run(action)               #    Execute: browser/terminal/API
    ok, fb  = verifier.check(action, result)     # 3. Verify: check if the result is correct
    state.update(thought, action, result, fb)    # 4. Update state
    if ok and state.goal_reached():
        break

For developers and enterprise users, this shift is far more significant than just “smoother responses”: deliverables change from “text” to “results,” and the engineering focus shifts from prompt engineering to task orchestration, tool governance, and safety guardrails. Long-process tasks like coding, reporting, document consolidation, and office automation are where agent capabilities first deliver tangible value.

II. Benchmarks Look Impressive, But Check “How They Were Run”

2.1 Same Model, Two Different Report Cards

A rather representative phenomenon has emerged in public industry discussions: a new-generation model scored around 60% on abstract reasoning benchmarks under a standardized testing framework. However, when tested using a customized tool provided by the vendor, its score skyrocketed to near perfection. What is the difference? The customized framework allows the model to retain hidden reasoning states during multi-step tasks, utilizes stronger context management, and provides a more complete toolset.

Translated into engineering terms: agent tasks are long-chain, where each step depends on intermediate conclusions from the previous ones. If a standard framework truncates the context and restricts tools every turn, the model has to “rethink” from scratch, causing errors to accumulate with each step. The customized framework essentially gives it a scratchpad it can check at any time. The same driver will naturally have different lap times on a track depending on whether they are driving a modified race car or a stock civilian car.

2.2 Beyond Benchmarks: Three Pitfalls

• Data Contamination: Evaluation questions might have leaked into the training corpus, inflating scores.
• Benchmark Saturation: Once leaderboards approach perfect scores, top models lose differentiation.
• Goodhart’s Law: When a metric becomes an optimization target, it ceases to objectively reflect capability.

It is worth emphasizing that even evaluation institutions admit that maxing out abstract reasoning benchmarks does not mean AGI has arrived. Benchmarks can serve as a necessary condition for filtering, but they should never be the sufficient condition for final selection.

III. Third-Party Evaluations: Look at Capability Structure, Not Single-Point Scores

Vendor narratives naturally lean towards showcasing their strongest areas. The value of third-party evaluations lies in completing the full capability panorama. Synthesizing common conclusions from independent industry evaluations, a few patterns are worth remembering:

1. Asynchronous Capabilities: A model’s coding agent capabilities might reach the first tier, while its general intelligence tests show only a slight improvement over the previous generation, or even lag behind same-generation competitors. “Strong in one area” does not equal “strong across the board.”
2. Task-Specific Fluctuations: Complex knowledge-based tasks are improving, but certain professional tasks might experience generational regressions—training objectives involve trade-offs, and capability shifts are the norm.
3. More Concise Output, Not Necessarily Lower Cost: New-generation models often output fewer tokens, but multi-step reasoning and multiple tool calls amplify the total volume. Calculating costs per task might actually reveal higher expenses.

Selection advice: Cross-reference multiple independent evaluation sources, and build a private evaluation set reflecting your own business as early as possible. Do not let public leaderboards make decisions for you.

IV. Calculating Costs: From “Token Unit Price” to “Cost Per Task”

4.1 The Unit Price Illusion

The invocation unit price for new flagship models is often 2 to 3 times that of the previous generation. “The model uses fewer tokens” might be true, but the saved volume is easily consumed—even surpassed—by the increased number of reasoning steps and tool calls. Industry evaluations have shown cases where the unit price increased, but the actual cost per task rose by over 70%.

4.2 The Correct Way to Calculate Costs

Cost per task = Σ(input tokens per step × input price + output tokens per step × output price) × (1 + retry rate) + tool execution/manual fallback costs

Here is a snippet of illustrative code to clarify this (parameters are for demonstrating the algorithm, not real product pricing):

def task_cost(steps, price_in, price_out,
              avg_in=4_000, avg_out=800, retry_rate=0.2):
    """steps: average reasoning rounds; price unit: currency / million tokens"""
    per_step = (avg_in * price_in + avg_out * price_out) / 1e6
    return steps * per_step * (1 + retry_rate)
# Previous generation: lower unit price, but more steps
old = task_cost(steps=10, price_in=4,  price_out=20)
# New generation: higher unit price, but fewer steps
new = task_cost(steps=8,  price_in=10, price_out=50)
print(f"Previous generation cost per task ≈ {old:.2f}")   # ≈ 0.38
print(f"New generation cost per task ≈ {new:.2f}")       # ≈ 0.77

A 20% reduction in steps, yet the cost doubled—this is the pitfall of focusing solely on token unit prices.

4.3 Three Engineering Practices to Control Costs

• Tiered Routing: Route simple tasks to smaller models; only assign complex, long-process tasks to flagship agents.
• Budget Guardrails: Set maximum steps, maximum budgets, and timeout circuit breakers for each task.
• Cost Ledger: Continuously record costs and success rates on a per-task basis, making “cost-effectiveness” an observable metric.

V. Deployment Practices: An Actionable Evaluation Process

1. Define Task Profiles: Outline the process length, fault tolerance, result verifiability, and environmental dependencies of target tasks to determine if they are suitable for an agent.
2. Build a Private Evaluation Set: Extract 50 to 200 end-to-end tasks from real business operations. Focus on five metrics: end-to-end success rate, average steps, cost per task, P95 latency, and false trigger rate for destructive operations.
3. Sandbox First, Gradual Rollout: Test in an isolated environment first, then gradually release. Apply least privilege; retain manual confirmation for critical operations like deletion, sending, and payment, and retain full-chain audit logs.
4. Leverage General Cloud Capabilities: Enterprises can utilize general capabilities provided by cloud platforms—such as Baidu AI Cloud’s model services, function compute, container platforms, log services, and monitoring alerts—to build the execution, observability, and guardrail systems for agents, avoiding redundant infrastructure building from scratch.

VI. Conclusion: The Capabilities Are Real, But the Math Must Be Clear

The shift from “answering” to “doing” is the most substantial paradigm upgrade for large models in recent years, delivering maximum value for long-process workflows like coding, reporting, and office automation. However, there is only one rational way to judge them: Look at the framework for benchmarks, look at the structure for capabilities, and look at the task for costs. Treating evaluation as an engineering problem rather than a matter of faith is the only way to ensure agents actually get the work done for you—instead of just inflating your bill.