Evaluating Agent-Optimized Models: Why General Benchmarks Fail

    Agent-optimized foundation models can excel in real automation workflows while barely moving on general Chinese-language leaderboards. This article explains the reason: a systematic mismatch between evaluation tools and optimization goals. It breaks the mismatch into three layers—task structure, metric definition, and measurement dimensions—then proposes six independently measurable capability areas, a shift from accuracy to task-level economics (including cost per successful task and Pass^k), a reproducible evaluation harness, five common interpretation pitfalls, and a four-stage roadmap from offline evaluation to online governance. It closes with a practical checklist for model selection.

    1. The Problem: Evaluation Goals and Optimization Goals Are Diverging

    In the past two years a new kind of foundation model has emerged: one whose core optimization direction is not “knowing more” but “getting the job done.” These models are typically described as versions deeply optimized for agentic scenarios, and their defining trait is that the data mix for the target scenario is factored in during late pre-training or mid-training—tool-call trajectories, multi-step task trajectories, and long-horizon interaction sequences with state—rather than only aligning format during post-training.

    This distinction looks subtle but actually determines whether your evaluation approach is valid. Post-training adaptation changes output style and format compliance; injecting the scenario during training changes the model’s prior over the task distribution. The former usually shows up on general benchmarks as small gains; the latter’s benefits concentrate precisely in the places general benchmarks do not cover.

    Hence a phenomenon that routinely confuses people doing technology selection: a model that performs excellently in a real automation pipeline may climb only one or two spots on a general Chinese-capability leaderboard, or move just a fraction of a percentage point. This does not necessarily mean the optimization failed. More often, there is a systematic mismatch between the evaluation tool and the optimization goal.

    2. Three Layers of Mismatch

    2.1 Task-Structure Mismatch

    General Chinese evaluation sets are mostly single-turn Q&A, single-turn reasoning, or shallow multi-turn dialogue. Each item carries complete context, and the answer can be produced in one shot. The typical agentic task, by contrast, is long-horizon dependent: the input to step 3 depends on the structured result returned by the tool at step 2; whether step 7 triggers a retry depends on the error type at step 5. The difficulty of such tasks is not point knowledge but state maintenance and error recovery—and these two are almost never sampled in general benchmarks.

    2.2 Metric Mismatch

    General benchmarks use “accuracy” as the primary metric, measuring end-state correctness. But the success rate of a long-chain task is a compound probability:

    1. end-to-end success rate ≈ Π (per-step accuracy) × (1 - error-propagation coefficient)
    2. if per-step accuracy = 0.95, chain length = 10 steps, no recovery mechanism:
    3. 0.95^10 ≈ 0.60

    This means a 2-percentage-point improvement in per-step capability is amplified along a long chain. Conversely, looking only at per-item accuracy treats a multiplication problem as an addition problem.

    2.3 Measurement-Dimension Mismatch

    An evaluation with accuracy but no latency and token consumption cannot support engineering decisions. At the same accuracy level, if the implementation path is “repeated trial and error + multiple rounds of self-correction,” its token consumption and end-to-end latency can be several times that of a direct path. In scaled scenarios billed per task, this difference directly determines whether a solution is viable.

    3. Redefining the Evaluation Target: Six Measurable Capabilities

    For agentic scenarios, it is worth splitting “model capability” into six dimensions that can be constructed and scored independently. Each dimension should come with a clear task-construction method and failure-mode annotation.

    1. Tool selection and argument construction

    • What it measures: given tool descriptions (name, purpose, parameter schema), whether the model can pick the right tool and generate valid arguments.
    • Key sub-items: type and enum constraints, completeness of required fields, correctness of nested structures, and whether it correctly refuses rather than fabricating a call when no tool fits.
    • Common failures: an argument name that is close but semantically wrong; treating optional parameters as required; forcing a call when no matching tool exists.

    2. Multi-step planning and state maintenance

    • What it measures: in tasks requiring 5+ steps, whether intermediate state is carried and updated correctly.
    • Key sub-items: key information not lost after context compression; completed steps not repeated; order dependencies not disrupted.
    • Common failures: forgetting early constraints in long context; repeating side-effecting operations that already took effect during a retry.

    3. Complex instruction following and constraint decomposition

    • What it measures: the full satisfaction rate when one instruction contains multiple parallel constraints (format, length, definition, veto items).
    • Key sub-items: the decay curve as the number of constraints grows with the task; trade-offs among conflicting constraints and explicit explanation of them.
    • Common failures: satisfying the main constraints while silently dropping secondary ones; being format-compliant but semantically incomplete.

    4. Scheduled and continuous tasks

    • What it measures: whether the task runs continuously on a schedule or by event rather than firing once.
    • Key sub-items: idempotency (repeated triggers produce no duplicate side effects), cross-session state persistence, adaptation to staggering and rate limits, and the ability to resume from an interruption point rather than starting over after an error.
    • Common failures: no idempotency key leading to duplicate submissions; compressing long-cycle state into a single-session context so it is lost on restart.

    5. Multi-agent collaboration

    • What it measures: the ability to decompose subtasks, assign them to different roles, and aggregate results.
    • Key sub-items: clear role boundaries (not overstepping to alter others’ outputs), effective context isolation, and whether the aggregation phase can detect and report subtask failures rather than silently filling them in.
    • Common failures: the lead agent blindly trusting sub-agent output; division of labor so fine-grained that coordination overhead exceeds the benefit.

    6. Long-context and high-concurrency stability

    • What it measures: throughput, quality decay under long inputs, and latency distribution under concurrency.
    • Key sub-items: the inflection point of the accuracy curve as input length grows; P95/P99 latency under high concurrency; observability of failed requests.

    4. Metric Design: From “Accuracy” to “Task-Level Economics”

    An evaluation report should contain at least the following metric groups, ideally presented as a table, with online measured values clearly distinguished from illustrative ones:

    Metric family Specific metrics Why needed
    Correctness end-to-end task success rate, Pass@1, Pass^k (all k runs succeed) distinguishes “can do it right once” from “reliably does it right”
    Robustness per-step accuracy, error-recovery rate, human-takeover rate locates where failures occur
    Latency time to first token, per-step round-trip time, end-to-end P50/P95/P99 long-chain scenarios are highly sensitive to tail latency
    Efficiency effective token ratio, average number of steps, redundant call count redundancy is the main source of cost
    Cost cost per successful task the denominator must be the number of successful tasks

    The last item is worth emphasizing. The common “cost per thousand calls” metric is misleading here: if a model’s success rate is low, it makes more calls, and looking at call cost alone makes it seem cheaper. The correct definition is:

    1. cost per successful task = (Σ input tokens × unit price + Σ output tokens × unit price + tool/sandbox overhead) / number of successful tasks

    This definition couples accuracy and cost together and truly reflects whether “spending 10% more tokens for 15% higher success rate” is worthwhile. Pass^k answers another key question: can this model enter an unattended automation pipeline, or must a human-confirmation step be retained?

    5. Evaluation Engineering: A Reproducible Harness Is a Prerequisite

    However rigorous the metric definitions are, the conclusions are invalid if the evaluation environment is not reproducible. A usable evaluation harness should satisfy four constraints:

    1. Mock all tools: version changes in real external APIs cause historical baselines to drift, so evaluation must use stubs with fixed contracts.
    2. Sandbox isolation: file system, network, and execution environment must be resettable to avoid side effects polluting subsequent cases.
    3. Controlled randomness: fix sampling parameters and random seeds, and record the hashes of the model version, prompt version, and scorer version.
    4. Separate the scoring function from the test cases: judge logic versioned independently, so the test set and the scorer can evolve separately.

    A minimal skeleton is shown below (illustrating structure only, not a specific platform interface):

    1. @dataclass
    2. class StepResult:
    3. ok: bool # whether this step met expectations (tool, args, side effects)
    4. recovered: bool # whether it recovered successfully after the previous step failed
    5. input_tokens: int
    6. output_tokens: int
    7. latency_ms: float
    8. def run_task(task: Task, agent: Agent, max_steps: int = 20) -> TaskResult:
    9. state, results = State.from_task(task), []
    10. for i in range(max_steps):
    11. if task.is_satisfied(state): # objective: state judgment, not text matching
    12. return TaskResult(success=True, steps=results, cost=sum(r.cost for r in results))
    13. if not state.budget_left(): # triple budget: tokens / steps / time
    14. break
    15. out = agent.step(state) # returns a tool call or a termination signal
    16. r = sandbox.execute(out) # dedupe by idempotency key to prevent duplicate side effects
    17. results.append(StepResult(
    18. ok=r.matches_expectation,
    19. recovered=r.ok and not results[-1].ok,
    20. input_tokens=r.usage.input, output_tokens=r.usage.output,
    21. latency_ms=r.elapsed))
    22. state = state.advance(out, r)
    23. return TaskResult(success=task.is_satisfied(state), steps=results)

    The key design is task.is_satisfied(state): the object being judged is the final state of the environment, not the text of the model’s output. This correctly handles cases where “the path differs but the result is correct” and avoids the scorer marking a valid solution as a failure.

    6. Five Common Pitfalls in Interpreting Results

    • Means hide the long tail: a model with an average latency of 2 seconds but a P99 of 40 seconds will cause noticeable pile-ups in scheduled batch jobs. Whenever scheduling is involved, always look at percentiles.
    • Task-distribution bias: the evaluation set comes from general corpora, while the business comes from real logs; the two differ greatly in tool variety, call depth, and failure-type distribution. A good practice is to sample 100–300 items directly from real logs, annotate success criteria, and use them as the primary evaluation set.
    • Benchmark contamination and overfitting: questions in public benchmarks may have entered training corpora. A mitigation is to build a private evaluation set and periodically cross-validate with newly constructed items.
    • Scorer noise: if the scorer itself has a 5% misjudgment rate while the gap between two models is only 1–2 percentage points, the conclusion does not hold. Confidence intervals must be reported, and when necessary, disagreement samples should undergo manual review sampling.
    • Single-cause attribution: score changes often come from the combined effect of prompts, context organization, retry strategy, and context-compaction scheme. In an A/B test, isolate variables one by one; otherwise the source of the gain cannot be determined.

    7. Implementation Roadmap: From Offline Evaluation to Online Governance

    Stage 1: Offline evaluation. Use the self-built task set to compare candidate models side by side, producing three tables—success rate, cost, and latency percentiles—and eliminating options that fail hard thresholds (such as a cap on human-takeover rate).

    Stage 2: Shadow run. The new model processes the same batch of real tasks in parallel with the incumbent model, recording only without executing side-effecting operations, and comparing divergent samples.

    Stage 3: Canary rollout. Route traffic by task type, and set automatic rollback thresholds (any one of: success-rate drop, P95 latency over limit, per-task cost over budget).

    Stage 4: Continuous governance. The capabilities needed online can be composed entirely from general-purpose cloud components: a unified model-service access layer for multi-model routing and fallback; an evaluation and experimentation platform to manage evaluation sets and baselines; log services and monitoring/alerting to observe success rate, retry rate, human-takeover rate, and per-task cost; and cost allocation to pinpoint the specific chains where token consumption is abnormal. Taking Baidu AI Cloud’s model and data service stack as an example, such platforms typically provide general-purpose capabilities such as model access, evaluation management, log monitoring, and cost analysis, so teams can turn evaluation from a one-off activity into a routine pipeline without building the entire infrastructure themselves.

    8. Conclusion and Checklist

    The value of scenario-enhanced foundation models is real, but their score changes on general Chinese-capability leaderboards do not themselves constitute a basis for selection—because that leaderboard is not measuring their strengths. When selecting a model, verify the following items one by one:

    1. Is there a self-built evaluation set that covers the real task distribution, with scoring based on the environment’s final state?
    2. Are end-to-end success rate and Pass^k reported, rather than accuracy alone?
    3. Are the three latency percentiles—P50/P95/P99—provided?
    4. Is the cost definition “cost per successful task”?
    5. Have scheduled and continuous tasks been verified for idempotency and resume-from-checkpoint?
    6. Have multi-agent solutions been verified for role boundaries and failure reporting?
    7. Are confidence intervals attached to conclusions, and are key differences manually reviewed?
    8. Does production have automatic rollback and cost alerts?

    Once the evaluation system aligns with the optimization goal, whether a model is “truly stronger” is no longer a matter of interpreting leaderboard rankings, but an engineering problem that can be measured, reproduced, and continuously monitored.