Model Agent Capability Evaluation and Selection Methodology

Introduction: From Parameter Comparison to Capability Evaluation

After a model release, many teams begin by opening the parameter table: context window size, maximum output length, input and output prices, and leaderboard rankings. They use these figures to decide which model to adopt. This approach was acceptable in the single-turn question-answering era, but it often fails in agent scenarios.

The reason is that agent performance is not determined by a single dimension. It is jointly determined by the coherence of multi-step decisions, the accuracy of tool calls, stability during long tasks, and cost per completed task. A parameter table can answer whether a capability exists, but it cannot answer whether that capability is sufficient or worth the price.

A more reliable approach is to turn model selection from a one-time purchase decision into a repeatable, measurable, and regression-friendly engineering process. This article answers four questions:

1. What to evaluate: how to break agent capability into observable dimensions.
2. How to test: how to design task sets, define pass criteria, and control experimental variables.
3. How to calculate cost: how to compare capability with cost and latency in the same coordinate system.
4. How to use the results: how to convert evaluation conclusions into tiered routing, fallback mechanisms, and gradual iteration.

1. Two Tiers Are Often Compared, but the Comparison Method Needs Adjustment

Around the release of each flagship model, the market often shows a typical pattern: an earlier flagship represents the capability ceiling, while a later mainstream model offers performance close to the flagship at a significantly lower unit price. The price gap between the flagship and mainstream offerings is also often significant. For example, the ratio of entry-level price to standard price may be about 1:2, and the priority tier may be about 1:4. These numbers are only illustrative of pricing structure; actual prices for each tier should follow the official documentation.

As a result, the selection question is often simplified to: “Is the more expensive model worth it?”

If the task is writing marketing copy or extracting fields, the conclusion is usually one-sided: the mainstream model is more cost-effective. Once the task enters an agent scenario, the conclusion becomes more nuanced. Agent task cost is not calculated linearly by the number of calls. It accumulates according to “steps × context per step × output per step.” Capability is also not evenly distributed. Differences become amplified in local metrics such as long-horizon task completion rate, tool-call precision, and self-repair ability after failure.

The correct comparison is not to ask, “Which is stronger, A or B?” Instead, ask three more specific questions:

• On the same task set, which model has a higher success rate? “Success” must be machine-verifiable, not based on subjective impressions.
• To reach the same success rate, which model has lower cost? This should include token consumption, retry counts, and the proportion of human intervention.
• To achieve the same result, which model has more controllable latency? For interactive applications, P95 latency often affects experience more than average latency.

The following sections expand on these points.

2. Agent Capability Dimensions

To evaluate whether a model is good, it is necessary to break the question into observable metrics. The following seven dimensions cover the aspects that most engineering teams care about.

2.1 Instruction Following and Output Controllability

Every agent output may be consumed by downstream programs. Therefore, stable formatting is a baseline capability. Evaluation should check whether fields are complete, types are correct, enum values fall within the expected set, and no explanatory filler is included.

A practical technique is to construct adversarial requirements: include a main task and several distracting constraints in the prompt, then observe whether the model misses or rewrites the constraints on its own.

2.2 Tool-Call Precision

Tool calls are the skeleton of an agent. Evaluation should distinguish three levels:

1. Whether to call: Does the model call a tool unnecessarily when no tool is needed?
2. Whether the call is correct: Are the function name, parameter names, and parameter types correct? Are enum values valid?
3. Whether the order is reasonable: When multiple tools have dependencies, can the model call them in the correct order? When there are no dependencies, can it call them in parallel to reduce latency?

Parameter-level accuracy is the core metric of this dimension. It is recommended to separately track the “parameter schema compliance rate” and the “semantic correctness rate.” The former can be checked programmatically, while the latter requires business assertions.

2.3 Coherence and Self-Repair in Long-Horizon Tasks

Long-horizon tasks are characterized by many steps, complex state, and accumulating errors. Evaluation should include three types of samples:

• Baseline tasks: Clear execution paths, used to confirm basic capability.
• Fault-injected tasks: Artificially inject error codes, timeouts, or empty results into tool returns, and observe whether the model can identify the issue and provide a reasonable remedy instead of repeating invalid calls or giving up directly.
• Long-context tasks: Place key constraints in the middle of a long document or across multiple files to test whether the model can maintain constraints when attention is diluted.

A counterintuitive phenomenon is worth noting: after the context window is expanded, task success rate does not necessarily improve. The reason is that “effective context” and “available context” are not the same thing. Retrieval noise can significantly reduce performance. Therefore, evaluation should fix the context organization strategy and separately evaluate “window size” and “context quality.”

2.4 Reasoning Budget and “Thinking” Configuration

New-generation models commonly provide controllable reasoning depth, often described as “adaptive thinking” or “extended thinking budget.” The evaluation focus is not whether thinking is enabled, but how much reasoning budget is cost-effective.

It is recommended to run budget-gradient experiments on a fixed task set, plot a “budget-accuracy” curve, and find the inflection point. After the inflection point, additional budget yields diminishing returns while cost increases linearly. The output of this step can often directly save more than 30% of call costs. This ratio depends on task type and must be measured in practice.

2.5 Speed and Latency Distribution

Latency should be divided into time to first byte and full response time. In agent scenarios, end-to-end completion time is the metric users perceive, and it is composed of “steps × per-step latency.” Evaluation should record P50 and P95 latency, and P95 should be used as the basis for an experience budget ceiling.

At the same time, high-throughput tiers usually come with higher unit prices. Selection therefore requires a joint trade-off between cost per task and P95 latency.

2.6 Cost Structure

The cost per task can be approximated as follows:

Cost per task ≈ Σ(input tokens_i × input price + output tokens_i × output price)
               + converted cost of retries and human intervention

Note that input tokens amplify as the number of steps increases: at each step, historical messages, tool results, and system prompts are sent back into the context. This means that the hit rate of prompt caching may have a greater impact on total cost than a 2× difference in input price. When evaluating selection, it is essential to clarify the pricing after cache hits.

2.7 Safety and Controllability

Agents have execution capabilities, so the risk surface expands accordingly. Evaluation should include refusal of unauthorized operations, recognition of dangerous commands, defense against prompt injection, and prevention of sensitive information leakage.

It is recommended to turn these items into “redline cases.” If any redline case fails, the model should be considered unqualified and should not participate in average score calculation.

3. Evaluation Set Construction: Derive Backward from Business Tasks

The most common failure in evaluation is using public leaderboard questions as one’s own task set. The correct approach is to derive task sets backward from real business tasks and organize them by difficulty level:

Level	Task Characteristics	Evaluation Method	Suggested Sample Size
L1 Single-step	Single-turn generation, extraction, classification	Rule assertions	100+
L2 Multi-step	Tool calls, multi-turn completion	Tool traces + result validation	50+
L3 Long-horizon	Cross-file, cross-system modification	End-to-end acceptance + regression testing	10–20
L4 Adversarial	Injection, privilege escalation, abnormal input	Safety rules	30+

The reason for layering is that differences measured with L1 tasks will be averaged out. Only by listing L3 and L4 separately can capability boundaries be exposed.

Scoring functions should be as machine-verifiable as possible. The following is an illustrative scaffold. It is independent of any specific vendor SDK and can be connected to any model service:

from dataclasses import dataclass, field
from typing import Callable
@dataclass
class TaskCase:
    task_id: str
    level: str                      # L1 / L2 / L3 / L4
    prompt: str
    rubric: Callable[[dict], float]  # Takes the final output of the model and returns a score from 0 to 1
    max_steps: int = 12
    tags: list[str] = field(default_factory=list)
@dataclass
class CaseResult:
    task_id: str
    level: str
    score: float
    steps: int
    input_tokens: int
    output_tokens: int
    latency_p95: float
    tool_param_valid: float
    redline_pass: bool
def run_suite(model_client, cases, budget=None):
    results = []
    for c in cases:
        runner = AgentRunner(model_client, max_steps=c.max_steps, budget=budget)
        final = runner.run(c.prompt)
        r = CaseResult(
            task_id=c.task_id, level=c.level,
            score=c.rubric(final),
            steps=runner.steps,
            input_tokens=runner.usage.input_tokens,
            output_tokens=runner.usage.output_tokens,
            latency_p95=runner.latencies_p95,
            tool_param_valid=runner.tool_param_valid_ratio,
            redline_pass=runner.redline_pass,
        )
        results.append(r)
    return results

Based on this scaffold, three core metrics can be calculated:

• Weighted pass rate: Σ(score × level weight). The weight of L3 should be significantly higher than that of L1.
• Effective step ratio: minimum steps required to complete the task / actual steps. This is used to measure verbosity.
• Cost per task: Directly plug token usage into the pricing formula.

def task_cost(r, price_in, price_out, retry_rate=0.0, human_rate=0.0,
              retry_cost=1.0, human_cost=10.0):
    base = r.input_tokens * price_in + r.output_tokens * price_out
    return base * (1 + retry_rate * retry_cost) + human_rate * human_cost

When these three metrics are placed in the same table, selection discussions have a common language.

4. Tiered Selection: Replace “Either-Or” with a Three-Tier Structure

Most teams struggle with the question of whether to use the flagship model or the mainstream model. A more practical structure is to turn this into a three-tier combination:

1. Flagship tier: Handles unattended long-horizon tasks, complex codebase modifications, and high-risk decision review. The evaluation criterion is the “zero-human-intervention completion rate.”
2. Mainstream tier: Handles the vast majority of daily calls, including conversation, retrieval-augmented question answering, and routine tool orchestration. The evaluation criterion is the “pass rate at cost per task.”
3. Lightweight tier: Handles structured tasks such as classification, extraction, format conversion, and routing decisions, as well as batch offline jobs. The evaluation criterion is “throughput and unit price.”

Why does the mainstream tier often carry 70%–85% of call volume? Because task distribution is long-tailed: most requests do not fall into the hardest 5%. The value of the flagship tier is not that it performs better on average, but that it does not overperform or underperform on complex tasks. This point must be reflected in evaluation. If only the average score is reported, the cost disadvantage of the flagship tier will be significantly underestimated. Only when L3 completion rate is listed separately can its value become clear.

4.1 Routing Strategy

Routing decisions should be made in the lightweight tier to avoid paying flagship-level costs for routing itself:

def route(task):
    if task.needs_long_horizon or task.risk_level == "high":
        return "flagship"
    if task.requires_tool_orchestration or task.token_estimate > 32_000:
        return "mainstream"
    return "lightweight"

Routing signals should be as structured as possible: task type labels, estimated token count, whether execution has side effects, and historical failure counts. Using rules rather than a model for routing makes the routing layer itself auditable.

4.2 Degradation and Fallback

A tiered structure must be paired with a degradation path. Otherwise, a single service fluctuation can escalate into a business incident:

• When the flagship tier times out or is rate-limited, automatically degrade to the mainstream tier and mark the result as “completed via degradation.” The business can then decide whether manual review is needed.
• When the mainstream tier fails twice consecutively, transfer the request to a human queue and write the sample into the regression set.
• Set independent quotas and circuit-breaker thresholds for each tier to prevent a failure in one tier from dragging down the entire chain.

5. Latency and Experience Budget

Latency budgets should be derived backward from perception thresholds, not forward from model speed. Common practices include:

• If time to first byte exceeds about 1 second, users may perceive lag. Therefore, interactive scenarios must enable streaming output.
• Single-turn completion time should be controlled within 5–8 seconds. Excess time should be filled with intermediate feedback such as progress indicators and step summaries.
• Long-horizon tasks should use asynchronous job mode, moving long-running work out of the interactive path and delivering results through message queues and task status queries.

During evaluation, record both “time to first byte” and “completion latency.” The two often produce different rankings: some models have fast first-byte latency but noticeable degradation during long output. During selection, it is more effective to clarify which segment matters more than to make a general judgment about “fast” or “slow.”

6. Four Levers for Cost Optimization

Assuming capability requirements are met, cost-reduction methods are usually ranked by return on investment as follows:

1. Prompt caching: Include stable system prompts, tool definitions, and long-document prefixes in the cache. After a cache hit, the pricing for repeated input drops significantly. This is especially effective for multi-step agents because historical messages are highly repetitive.
2. Context compression: Summarize or structurally trim tool return results, keeping only necessary fields. In practice, this step can often reduce input tokens by 40%–60%. The exact reduction depends on the task type.
3. Budget tiering: Assign different reasoning budgets to tasks of different difficulty. Use low-budget tiers for simple tasks and increase the budget only for complex tasks.
4. Batch processing: Use batch APIs for offline, non-real-time evaluation and batch generation jobs. This usually provides a clear unit-price discount, but the tradeoff is that completion time becomes less controllable.

7. Controllability Engineering: Put Uncertainty in a Cage

Regardless of which tier is selected, the stability of agent deployment depends on engineering constraints rather than model self-discipline:

• Structured output: Prefer server-side structured output capabilities so that the model generates according to a given schema. The client should still perform secondary validation. If validation fails, perform targeted retries instead of rerunning the entire task.
• Least privilege: The tool layer should grant the minimum necessary permissions for each task. High-risk operations should go through secondary confirmation or an approval workflow.
• Externalized state: For long-horizon tasks, write progress, completed steps, and to-do lists into external storage such as object storage or a relational database. This supports resuming from checkpoints and avoids context bloat.
• Observability: Report each call’s input and output, token usage, tool traces, and latency to logging and monitoring systems. Alert rules should cover three types of events: sudden increases in failure rate, single-task cost exceeding a threshold, and P95 latency degradation. Evaluation data should be retained long-term as a regression baseline.

8. Regression and Canary Release: Keep Selection Conclusions Valid

Conclusions from a one-time evaluation will become invalid as model versions are updated. It is recommended to establish three mechanisms:

1. Versioned evaluation sets: Store the task set in a code repository and manage it under version control together with model configurations.
2. Full regression before each model upgrade: Use weighted pass rate as the threshold. If the result is below the baseline, traffic switching should not be allowed.
3. Canary traffic release: Gradually increase traffic in stages: 5% → 20% → 50% → 100%. At each stage, observe cost and quality metrics. Roll back immediately if anomalies occur.

9. Common Pitfalls

• Using average scores for decisions: L1 tasks will average out differences in long-horizon capability.
• Ignoring input-side amplification: In multi-step agents, input tokens accumulate with each step. Output price is often mistakenly treated as the main cost item.
• Treating context window size as capability: A large window does not mean effective utilization. Context organization strategy should be evaluated separately.
• Ignoring fallback paths: Relying on a single tier can turn a rate-limit event into a business outage.
• Assuming one evaluation lasts forever: Both models and toolchains continue to evolve, so conclusions need to be refreshed periodically.

10. Conclusion

Agent selection is essentially an engineering decision, not a leaderboard question. By breaking capability into observable dimensions, breaking tasks into verifiable cases, and breaking cost into attributable items, teams can build a flagship–mainstream–lightweight three-tier structure with a fallback chain. This allows them to find a sustainable balance among effectiveness, cost, and latency.

The methods described above can be directly implemented in various cloud environments. Evaluation scripts and task sets can run on container platforms. Evaluation data and traces can be stored in object storage and databases. Cost and quality metrics can be continuously tracked through logging services and monitoring alerts.

When building Agent applications on Baidu AI Cloud, teams can similarly use its model service capabilities and general cloud product capabilities to solidify the closed loop of evaluation, selection, canary release, and regression, ensuring that every model upgrade is supported by data.