A New Lens on AI Agent Evaluation: Balancing Task-Level Cost and Domain Expertise

1. From Exams to Real Work: The Shift in Evaluation Paradigms

Over the past two years, LLM evaluation systems have largely been built around single-turn question-answering accuracy. This approach has its merits: it is standardized, reproducible, and easy to compare across models. But it has a fundamental flaw: it measures what a model knows, not what a model can accomplish.

Tasks in real business scenarios typically have the following characteristics:

• Multi-step: A complete task must be decomposed into several sub-steps, and failure at any one step can cause the entire task to fail.
• Tool-dependent: The model needs to call external APIs, read and write files, execute code, and query databases.
• State management: Intermediate results must be passed and validated between steps.
• Fault tolerance: The model must be able to recover and retry when faced with incomplete input or abnormal responses.

As a result, a new evaluation mindset has gradually become an industry consensus: move evaluation from exams back to real work—rather than looking at single-turn answer scores, assess whether a model can reliably deliver results in a real task environment. Leaderboards of this kind are often called agent capability evaluation benchmarks, and their core metrics are no longer accuracy alone but a combination of task completion rate, delivery quality, execution cost, and stability.

The implication for developers is direct: the scores you reference when selecting a model should be as close as possible to the business workflows you actually intend to run.

2. Task-Level Cost: A Long-Underestimated Selection Dimension

In agent scenarios, the way cost is calculated has fundamentally changed. Traditional API calls are billed by token, whereas the cost of an agent task is the sum of multi-turn reasoning + tool calls + retry overhead. A single task may trigger dozens of model calls, and the price difference of a single call can be magnified dozens of times.

A more reasonable cost unit is cost per task, i.e., the average inference and call cost of completing one task on a standard task set. This metric has two key properties:

1. It is strongly correlated with task complexity. On simple tasks, cost differences between models may not be obvious, but on complex tasks with long chains and multiple tools, the differences become significant.
2. It is not linear with performance. In industry practice, it is common to see cases where “performance is slightly higher but cost doubles,” as well as options where “performance is close but cost is an order of magnitude lower.”

This leads to an important analytical tool: the Pareto frontier.

On the performance–cost two-dimensional plane, if a model achieves the best results at the same cost, or the lowest cost at the same results, it lies on the Pareto frontier. For engineering teams, only models on the Pareto frontier are truly worth including in the candidate pool—because they represent the optimal trade-off points under current technical conditions.

A typical judgment logic can be expressed in pseudocode:

def is_pareto_efficient(candidates):
    """
    candidates: list of (performance, cost)
    Returns candidate points on the Pareto frontier
    """
    efficient = []
    for i, (p1, c1) in enumerate(candidates):
        dominated = False
        for j, (p2, c2) in enumerate(candidates):
            if i == j:
                continue
            # j is no worse than i in performance, no higher than i in cost,
            # and strictly better in at least one dimension
            if p2 >= p1 and c2 <= c1 and (p2 > p1 or c2 < c1):
                dominated = True
                break
        if not dominated:
            efficient.append((p1, c1))
    return efficient

The value of this logic is that it helps teams quickly filter out dominated options and focus limited evaluation resources on models that are genuinely competitive.

3. Domain Expertise: Why Finance Is a Litmus Test

General agent evaluations can reflect baseline capabilities, but they cannot answer a key question: in highly specialized domains, is the model’s delivery quality reliable?

Financial analysis is a typical “high-barrier + high fault-tolerance requirement” scenario. It covers:

• Fund research and securities analysis
• Quantitative strategy construction and backtesting
• Asset allocation and portfolio management
• Corporate finance and accounting

These tasks place demands on models that go far beyond “generating a coherent paragraph of text”:

1. Process compliance: Financial analysis has mature workflows, and models need to understand and follow them rather than improvise.
2. Data rigor: Numbers, units, and time conventions must be accurate; any hallucination can invalidate a conclusion.
3. Explainability: Analytical conclusions need to be traceable to data sources and reasoning chains.
4. Understanding of domain terminology: Models need a correct understanding of financial statement items, valuation methods, risk metrics, and similar concepts.

Therefore, whether a model possesses “professional financial analysis capability” cannot be inferred from general leaderboards; it must be evaluated on workspaces and task sets built around real financial scenarios. A typical approach is to construct thousands of simulated workspaces covering tens of thousands of high-quality tasks, allowing the model to complete analysis workflows under conditions close to a real working environment.

On this basis, expert crowdsourced evaluation is an important supplementary validation method. Experts with financial industry backgrounds score model outputs across multiple dimensions, capturing quality dimensions that automated metrics struggle to cover, such as logical consistency, reasonableness of conclusions, and professionalism of expression.

4. A Reusable Agent Evaluation Workflow

Combining the above analysis, here is an evaluation workflow that engineering teams can directly reference:

Step 1: Define the Task Set

Extract representative tasks from actual business operations and stratify them by complexity:

Level	Task Characteristics	Example
L1	Single-step, no tools	Information extraction, format conversion
L2	Multi-step, single tool	Data query + summarization
L3	Multi-step, multiple tools, stateful	Full analysis report generation
L4	Long chain, requires fault tolerance	Strategy backtesting + exception handling

Step 2: Collect Multi-Dimensional Metrics

Record the following data for each task:

{
  "task_id": "fin-0427",
  "level": "L3",
  "success": true,
  "quality_score": 0.86,
  "latency_ms": 18400,
  "token_cost_usd": 0.42,
  "tool_calls": 7,
  "retries": 1
}

Step 3: Calculate Task-Level Cost

def cost_per_task(records):
    total_cost = sum(r["token_cost_usd"] for r in records)
    return total_cost / len(records)
def success_rate(records):
    return sum(1 for r in records if r["success"]) / len(records)

Step 4: Plot the Pareto Frontier

Use success rate as the vertical axis and cost per task as the horizontal axis, filter out frontier models, and then combine expert scores to make the final decision.

Step 5: Small-Traffic Gray-Scale Validation

Run candidate models on bypass traffic in the production environment, compare their delivery quality and cost against the existing solution, and switch over fully only after confirming that the evaluation conclusions are reproducible.

5. Practical Recommendations for Developers

1. Don’t look at a single score alone. Overall leaderboard rankings reflect comprehensive capability, but your business may be concentrated in a vertical domain, so you need to look at domain-specific performance.
2. Bring cost into the first round of screening. When performance gaps are within 1%, cost differences are often what actually decides the choice.
3. Prefer models on the Pareto frontier. They represent the current optimal trade-off, not “the strongest in some single dimension.”
4. Use real tasks for final validation. Any leaderboard is only an approximation; your own business workflow is the only standard.
5. Build a continuous evaluation mechanism. Models iterate quickly, and today’s conclusions may become invalid three months later. Evaluation should be a repeatable pipeline, not a one-time action.

Conclusion

Model selection in the agent era is essentially a multi-objective optimization problem: performance, cost, stability, and domain fit are all indispensable. The shift of evaluation systems from exams to real work is precisely what gives this optimization problem more reliable inputs. For specialized domains such as finance, only models whose delivery quality and task-level cost have been validated on real task sets deserve to be included in the production pipeline. The Pareto frontier, meanwhile, is the ruler that helps us quickly locate the optimal trade-off among numerous options.