Model Agent Capability Evaluation and Selection: From Evaluation Sets to Cost Closure

1. The root cause: the evaluation object changes in agent scenarios

Single-turn QA evaluation is relatively simple: provide a set of prompts and compare answer quality. Agent scenarios change the basic unit of evaluation. The evaluated object is no longer an “answer,” but an execution trace: the model repeatedly reads state, calls tools, observes returned results, revises its plan, and continues until the task is completed or abandoned.

This creates three direct consequences:

1. Errors are amplified. A 95% single-step accuracy rate drops to roughly 60% theoretical success in a 10-step chain (0.95^10). For long-horizon tasks, differences between model tiers are often not “slightly smarter,” but compounding effects.
2. Cost is no longer determined by prompt length alone. Agents repeatedly carry the same context into each reasoning step, so input token consumption is often much higher than output token consumption. If a model supports extended thinking or reasoning modes, thinking tokens may also be billed separately.
3. “Sounds correct” does not mean “task completed.” A model may produce an elegant plan but fail during execution, or it may have a rough plan but strong error recovery. Evaluation must be anchored to verifiable environment states.

Therefore, the core selection question should be restated as: Under a given task distribution, which model tier achieves an acceptable task success rate and controllable behavior at the lowest cost per successful task?

2. Eight dimensions for evaluating agent capability

Do not rely only on public benchmark scores. Place candidate models into a unified evaluation framework and collect data across the following dimensions:

Dimension	Observation method	Key metrics
Tool-calling correctness	Single-step and multi-step tool-selection test sets	Correct tool-selection rate, parameter-filling accuracy, invalid-call rate
Long-horizon planning and persistence	Tasks requiring 8–30 steps to complete	Task success rate, average steps, no-progress steps
Error recovery	Injected tool errors, empty results, timeouts	Recovery rate, retry effectiveness, whether the model gets stuck in loops
Context management	Very long documents plus multi-turn tool feedback	Recall of key information, use of middle-segment information
Controllability and structured output	Strict schema constraints, boundary conditions	Schema-valid rate, out-of-policy output rate, reasonableness of refusals
Instruction following and format stability	Repeat the same prompt N times	Consistency, format drift rate
Latency and throughput	Fixed-concurrency load testing	P50/P95 end-to-end latency, time to first token
Cost per successful task	Combined with real token usage	Cost per successful task, cost per 1,000 steps

Among these, error recovery is the most easily overlooked and one of the most discriminating factors. The gap between flagship-tier and balanced-tier models usually does not come from “getting it right the first time,” but from “whether the model can detect its own mistake and provide an alternative path.”

3. How to build evaluation sets: layered, traceable, and regression-ready

3.1 Four layers

• L1 capability probes (50–100 cases): single-step tool selection, parameter extraction, and format constraints. Used to quickly eliminate clearly unqualified candidates.
• L2 business closed-loop set (30–60 cases): real tickets, real data, and real side effects, such as database writes, message sending, and external API calls. This is the primary basis for selection.
• L3 long-horizon and robustness set (10–20 cases): multi-turn tasks, mid-course plan changes, conflicting instructions, and missing information.
• L4 regression set: collected from real production failures and used as a gate after version upgrades.

The key benefit of layering is cost control. L1 can be run frequently. L2 should be run for each selection decision or major version change. L3 should be reserved for the final candidate stage.

3.2 Required metadata for each case

from dataclasses import dataclass, field
@dataclass
class Case:
    id: str
    layer: str                 # L1/L2/L3/L4
    prompt: str
    tools: list[str]
    assert_fn: str             # Points to an executable validation function
    sandbox: str               # Isolated environment identifier; must be resettable
    max_steps: int = 12
    budget_tokens: int = 400_000
    tags: list[str] = field(default_factory=list)

Three hard requirements are mandatory:

• The environment must be resettable to avoid cross-contamination between cases.
• Assertions must be executable; success or failure must not depend on human judgment.
• Side effects must be rollbackable. L2 cases must run inside a sandbox.

4. Evaluation execution harness

A unified harness is a prerequisite for comparable metrics. Use the same harness, the same tool implementations, and the same context-construction strategy, and replace only the model adapter layer.

@dataclass
class Trace:
    case_id: str
    success: bool
    steps: int
    tool_calls: int
    invalid_calls: int
    in_tok: int
    out_tok: int
    latency_ms: int
    stop_reason: str           # done / max_steps / budget_exceeded / api_error
def run_case(model, case, env) -> Trace:
    env.reset(case.sandbox)
    msgs = [{"role": "user", "content": case.prompt}]
    t = Trace(case.id, False, 0, 0, 0, 0, 0, 0, "api_error")
    for _ in range(case.max_steps):
        resp = model.call(msgs, tools=env.tool_schemas(case.tools))
        t.steps += 1
        t.in_tok += resp.usage.input_tokens
        t.out_tok += resp.usage.output_tokens
        if resp.tool_calls:
            for call in resp.tool_calls:
                t.tool_calls += 1
                if not env.validate(call, case.tools):
                    t.invalid_calls += 1
                msgs.append(env.execute(call))     # Real side effects occur inside the sandbox
        else:
            msgs.append({"role": "assistant", "content": resp.text})
            break
    else:
        t.stop_reason = "max_steps"
    t.success = env.assert_fn(case.assert_fn)(case.id)
    t.latency_ms = env.elapsed_ms
    return t

When aggregating results, do not report only averages. Averages can hide long-tail failures:

def summarize(traces, unit_price_in, unit_price_out):
    n = len(traces)
    ok = [t for t in traces if t.success]
    cost = lambda t: (t.in_tok * unit_price_in + t.out_tok * unit_price_out) / 1e6
    return {
        "success_rate": len(ok) / n,
        "avg_steps": sum(t.steps for t in traces) / n,
        "invalid_call_rate": sum(t.invalid_calls for t in traces)
                             / max(1, sum(t.tool_calls for t in traces)),
        "p95_latency_ms": sorted(t.latency_ms for t in traces)[int(n * 0.95) - 1],
        "cost_per_all": sum(cost(t) for t in traces) / n,
        "cost_per_success": (sum(cost(t) for t in traces) / len(ok)) if ok else None,
        "failure_breakdown": {r: sum(1 for t in traces if t.stop_reason == r)
                              for r in {"max_steps", "budget_exceeded", "api_error"}},
    }

cost_per_success is more useful for decision-making than unit price. A model that is 40% cheaper may end up costing more if it requires 30% more retries and manual fallback.

5. Scorecard: structure subjective judgment

For aspects that cannot be automatically asserted, such as “plan quality” and “expression clarity,” use a weighted scorecard plus human spot checks rather than letting a single large model decide the winner:

• Automatically assertable items (60% weight): task completion, schema validity, zero out-of-policy or dangerous operations, timeout rate.
• Trace-quality items (25% weight): whether steps are redundant, whether there are meaningless repeated tool calls, and whether the model unnecessarily expands into long reasoning.
• Risk items (15% weight, including veto conditions): unauthorized writes, context leakage, and missing confirmation for irreversible operations.

If a large model is used to assist scoring, three practices are essential: provide an explicit rubric with item-by-item rationale, fix the scoring prompt and temperature, and manually review 20% of samples to calibrate consistency.

6. Model-tier mapping by task characteristics

Most vendor model matrices follow a “flagship—balanced—lightweight” structure. Within the same tier, there may also be a standard mode and a high-speed mode, where the high-speed mode has a higher unit price but lower latency. Rather than obsessing over a single benchmark score, map models to tasks based on task characteristics:

Task characteristics	Recommended tier	Reason
Single-point classification, extraction, summarization, intent recognition	Lightweight tier	Latency and cost are sensitive; insufficient capability risk is low
High-concurrency customer-service assistance, batch document processing	Balanced tier + batch processing	The cost-performance inflection point usually falls in the middle tier
Multi-tool orchestration, cross-system code modification, long-document analysis	Flagship tier	Error recovery and long-horizon persistence determine success
Interactive completion, latency-sensitive chains that care about first-token latency	Balanced tier + high-speed mode	Paying a higher unit price for lower latency is often better than dropping a tier

Hybrid routing is often the optimal cost solution, rather than choosing only one model:

def route(task):
    if task.est_steps <= 2 and task.risk == "low":
        return "lite"                    # Lightweight tier
    if task.ctx_tokens > 300_000 or task.risk == "high":
        return "flagship"                # Flagship tier, including irreversible operations
    return "balanced"                    # Default to the balanced tier
# Escalation cascade: if the balanced tier shows no progress twice in a row, escalate to the flagship tier
def escalate(trace_history, model_name):
    if len(trace_history) >= 2 and not any(t.success for t in trace_history[-2:]):
        return "flagship"
    return model_name

A key constraint for escalation strategies is to set an escalation limit, such as at most one escalation per task. Otherwise, failed tasks may repeatedly consume budget across tiers.

7. Three hidden items in the cost model

1. Input token reuse. Agents carry historical context into every step, so input tokens are often several times the output tokens. If the service supports context caching or reuse, the hit rate may affect total cost more than switching to a cheaper tier.
2. Thinking tokens and retries. Enabling extended thinking can improve accuracy on difficult tasks, but thinking content is often billed as output. The correct approach is to dynamically set the thinking budget based on task difficulty, not to keep it enabled everywhere by default.
3. Failure cost. Timeouts, retry costs caused by rate limits, and manual fallback effort must all be included to form the real total cost of ownership. In practice, maintain a cost dashboard that includes the “human intervention rate” alongside model unit price.

8. Launch and governance: turn selection into a process

• Shadow comparison: Run the old and new models on the same production traffic in parallel, using shadow mode where the new model only records results without affecting users. Compare success rate, P95 latency, and cost per successful task before switching.
• Regression gate: Add the L4 regression set to CI. Any model version upgrade, prompt change, or tool schema change must pass this gate first.
• Observability: Persist full traces to logging services and object storage. Label them by model identifier, task type, and step distribution. Set monitoring alerts for failure rate, invalid-call rate, and cost per successful task.
• Version policy: Model identifiers may differ across API versions. Production systems should explicitly pin model identifiers and snapshots to avoid behavior drift caused by default upgrades on the provider side.
• Safety boundaries: Irreversible operations, such as database writes, payments, deletions, and outbound sends, must require human confirmation or secondary validation. This is independent of model capability; it is an architectural constraint, not a model-capability question.

9. Common mistakes

• Replacing internal evaluation with public leaderboards. Leaderboards reflect general distributions; your business distribution is the decision basis. Leaderboards can be used to shortlist candidates.
• Making decisions based only on unit price. The only meaningful price metric is cost per successful task.
• Looking only at success rate without looking at step count. Step inflation directly increases latency and token consumption, especially for long-tail tasks.
• Ignoring the real constraints of context windows and maximum output per request. For long-document scenarios, measure middle-segment information utilization; nominal window size is not enough.
• Treating selection as a one-time decision. Model iteration cycles are measured in months. Quarterly re-evaluation should be built into the engineering rhythm.

10. Selection checklist

1. Has the task distribution been quantified in terms of steps, context length, and risk level?
2. Are the L1–L4 evaluation sets ready and automatically assertable?
3. Is the harness unified, with only the model adapter layer replaced?
4. Is cost per successful task calculated, rather than unit price alone?
5. Are there dedicated cases for error recovery and dangerous-operation blocking?
6. Have the cost impacts of context caching, batch processing, and thinking budgets been validated?
7. Are shadow traffic, gradual rollout, and rollback capabilities in place?
8. Is the regression set integrated into CI with monitoring alerts?

Following this process turns the selection conclusion from “a certain model feels stronger” into an engineering statement: Under task distribution X, model tier Y achieves W% task success rate at a cost of Z per successful task. This is a conclusion that can be written into a technical design and revisited in future re-evaluations.