LLM Agent Evaluation and Model Selection: A Practical Guide

    This practical guide explains how to evaluate LLM agents and select models for production tasks. It defines agent capability as a multi-step closed loop, reviews major benchmark types, metrics such as Pass^k and cost per completed task, and provides scenario-based decision matrices, engineering recommendations, and a checklist for reproducible evaluation.

    Introduction: A New Selection Logic for the Agent Era

    Since 2026, competition among large language models has been shifting from answering questions to completing tasks. Major model vendors have entered a period of rapid iteration. Flagship models with different reasoning tiers, mid-tier models focused on cost-performance, and self-hostable open-source models have all appeared in succession.

    The adoption speed of applications such as intelligent coding tools, computer-use assistants, automated testing, and operations automation depends directly on whether the underlying model can reliably execute tasks in long-horizon, multi-step, and uncertain environments.

    At the same time, traditional selection methods are losing effectiveness. Three major problems stand out:

    • General leaderboards have limited reference value: Static evaluations such as knowledge Q&A and reading comprehension do not reflect how a model actually performs during the loop of “call tool → observe result → fix error”.
    • Large gap between demos and stable performance: A demo only needs to succeed once, but production systems require long-term stability. Agent tasks often involve dozens or even hundreds of steps, and reliability is amplified at every step.
    • Capability and cost are hard to balance: Different models vary significantly in pricing, reasoning modes, and response speed. Choosing only the highest-capability model or only the lowest-price model often fails to produce the best overall solution.

    This article first clarifies what agent capability means and which dimensions should be evaluated. It then reviews mainstream evaluation methods and key metrics, followed by selection decision matrices for different business scenarios and practical engineering recommendations. The goal is to help readers build a reproducible and verifiable model selection process.

    1. Understanding Agent Capability

    1.1 Definition: A Closed Loop from Goal Understanding to Result Delivery

    Agent capability refers to a large model’s ability to complete the full loop of “understand the goal → plan steps → call tools → check results → fix errors” autonomously, without continuous human intervention.

    The core of agent capability is not the quality of a single-step output. It is whether the model can still produce a correct result after multiple steps accumulate.

    1.2 Five Stages of the Capability Loop

    Stage What the model needs to do Typical failure modes
    Understand the goal Parse natural-language instructions, identify implicit constraints, acceptance criteria, and priorities Missing key constraints, drifting from the goal
    Plan steps Break the goal into an executable sequence of subtasks and adjust dynamically when necessary Missing steps, wrong order, or falling into loops
    Call tools Generate structurally correct and semantically reasonable calls based on tool descriptions and parameter schemas Wrong parameter types, missing required fields, hallucinating nonexistent tools
    Check results Read tool returns such as text, errors, and screenshots, and judge whether they meet expectations Ignoring errors, misjudging success, ending too early
    Fix errors Locate the cause and adjust the plan, including switching tools, changing parameters, rolling back, or asking for help Repeating ineffective attempts, going further down the wrong path

    1.3 Key Differences from Q&A

    Comparison dimension Traditional Q&A Agent tasks
    Output form Single-turn or multi-turn text answers A series of tool calls and state changes
    Interaction rounds Usually single-digit Tens to hundreds of steps, often across multiple context windows
    Environment dependency Mainly relies on knowledge stored in model parameters Depends on external environments such as file systems, terminals, browsers, and APIs
    Cost of errors A wrong answer can be clarified or corrected through follow-up A single-step mistake may damage environment state and cause overall failure
    Capability bottleneck Knowledge coverage and expression quality Planning, tool calling, error recovery, and long-context management
    Evaluation metrics Accuracy and relevance End-to-end success rate, reliability, steps, and cost

    1.4 Error Accumulation: Why a High Average Score May Still Be Unusable

    Assume the model’s per-step success rate is p, the steps are relatively independent, and the task requires n steps. The probability that the entire task succeeds is approximately p^n.

    • When p = 0.95 and n = 20, the success rate is approximately 36%.
    • When p = 0.99 and n = 100, the success rate is approximately 37%.

    This means that even 99% reliability at every step still allows only about one-third of 100-step tasks to complete successfully.

    In real tasks, steps often depend on one another. An error in an earlier step can contaminate later inputs, creating an “error snowball” that further reduces real-world performance.

    Therefore, agent selection is essentially about answering three questions:

    1. Can the model complete the task in a single run under the target scenario?
    2. Can it complete the task every time across multiple runs? This is reliability.
    3. How much time and cost does each completed task require?

    2. Six Evaluation Dimensions for Agent Capability

    2.1 Task Understanding and Planning

    This dimension measures whether the model can turn vague or lengthy requirements into a clear and executable plan.

    Key evaluation points include:

    • Whether the model identifies implicit constraints, such as “do not modify production configuration” or “all unit tests must pass”.
    • Whether the granularity of subtasks is reasonable.
    • Whether the plan can adjust as the environment provides feedback.

    Planning capability can be observed in two ways:

    • Ask the model to output a plan first and then execute it, while reviewers score the plan against a standard workflow.
    • Compare success rates between “allow pre-planning” and “force step-by-step decision-making” modes.

    2.2 Tool Calling and API Understanding

    This dimension measures whether the model can accurately understand tool names, purposes, parameter schemas, and return values, and then generate valid calls.

    The focus is on four aspects:

    • Whether the model calls only registered tools.
    • Whether required parameters are complete and types are correct.
    • Whether complex nested parameters can be organized correctly.
    • Whether the model can self-correct call formats based on error messages.

    This dimension is especially important for tool ecosystems with mature function-calling specifications and complex parameter structures.

    2.3 Environment Interaction and Observation Parsing

    When agents work in environments such as terminals, file systems, and operating-system interfaces, they must parse large amounts of unstructured information, including command output, error stacks, directory structures, and UI screenshots.

    The key capability is extracting meaningful signals from noise. For example:

    • Locating the actual error line in a long log.
    • Deciding whether to install a dependency or modify code based on an error message.

    2.4 Error Detection and Self-Recovery

    Agent tasks often fail not because the model lacks ability, but because it does not know what to do after making a mistake.

    This dimension evaluates whether the model can:

    • Detect anomalies, such as a command returning a nonzero exit code or a test failing.
    • Locate the root cause rather than repeatedly retrying ineffective actions.
    • Roll back, switch approaches, or ask a human for confirmation when necessary.

    2.5 Long-Context and State Management

    Long-horizon tasks cause context to expand continuously, including historical messages, tool outputs, file contents, and error logs.

    The model needs to:

    • Preserve critical information within a limited context window.
    • Compress secondary information.
    • Maintain task-goal coherence when the context approaches its limit.

    2.6 Safety Awareness and Boundary Judgment

    A model capable of autonomous execution also has the potential to cause damage. It is important to evaluate whether the model can:

    • Refuse clearly dangerous or unauthorized instructions.
    • Request confirmation before high-risk operations, such as deleting data, changing permissions, sending information externally, or incurring costs.
    • Resist malicious instructions embedded in external content, also known as indirect prompt injection, such as hidden text in a webpage or file that says “ignore previous instructions”.

    3. Mainstream Evaluation Methods: From General Benchmarks to Scenario-Specific Tests

    3.1 Overview of Evaluation Methods

    Evaluation method Main focus Strengths Limitations
    Head-to-head or arena-style evaluation Overall preference on real open-ended tasks Close to real experience; questions can be updated dynamically Depends on human reviewers or strong judge models; single-run variance can be high
    Terminal and command-line task benchmarks System operations, script execution, troubleshooting Controllable environment; results can be verified automatically Biased toward operations and development; limited task distribution
    OS and desktop operation benchmarks Cross-application GUI operation Covers real office scenarios Complex environment setup; UI changes affect stability
    Repository-level fix benchmarks Locating and fixing defects in real codebases Close to engineering practice; objective verification High task preparation cost; may overlap with training data
    Function-calling benchmarks Tool selection, parameter construction, multi-turn calls High automation; clear feedback Hard to reflect long-horizon execution
    Scenario-specific in-house evaluation sets Real tasks closely related to business Highly targeted; directly guide selection Requires continuous construction and maintenance

    3.2 Head-to-Head Evaluation: Measuring Overall Capability Through Preference Statistics

    Head-to-head evaluation gives the same task to two anonymous models and asks human reviewers or a judge model to vote based on whether the result is correct, whether the process is reasonable, and whether the behavior is safe. The results are then aggregated into an overall ranking.

    A core metric, net improvement rate, can be defined as follows:

    1. win_rate = (wins + 0.5 × ties) / total_matches
    2. net_improvement = (wins - losses) / total_matches
    3. Example (illustrative values):
    4. Model A plays Model B for 1,000 matches
    5. A wins 300, loses 200, ties 500
    6. A win_rate = (300 + 0.5 × 500) / 1000 = 55%
    7. A net_improvement = (300 - 200) / 1000 = 10%
    • A positive net improvement rate means the model has more wins than losses and is generally stronger than its opponent.
    • Net improvement rate only reflects direction: ties, including cases where both models fail, dilute the metric. If many matches end in “double failure”, a high net improvement rate does not necessarily mean strong task-completion capability.
    • It should be used with other metrics: it is better to examine win rate, tie rate, and the proportion of matches where both models fail at the same time.

    Some leaderboards also calculate composite scores from pairwise comparison data, such as Bradley-Terry-style models, and distinguish between “standard mode” and “reasoning/thinking mode”. Because enabling reasoning mode usually increases compute cost, comparisons should note which mode was used.

    3.3 Terminal and OS Benchmarks: Restoring Real Execution Environments

    Terminal benchmarks provide the model with a command-line environment inside a container or virtual machine. Tasks include:

    • Locating the cause of a service startup failure.
    • Repairing corrupted configuration files.
    • Batch-processing files.
    • Writing and debugging scripts.

    Verification is usually done by checking the final environment state or running a set of tests, so results are objective and automatable.

    Operating-system and desktop benchmarks extend this to graphical interfaces. Tasks include organizing files across applications, filling out forms, and completing configuration based on screenshots. The model needs to output mouse and keyboard actions and adjust step by step based on screenshot feedback.

    These benchmarks best expose execution-layer problems:

    • Incorrect command composition.
    • Ignoring exit codes.
    • Repeated execution causing side effects.
    • Overly aggressive operations, such as deleting an entire directory to “clean up” a problem.

    3.4 Repository-Level Fix Benchmarks: Close to Real Engineering Practice

    Repository-level fix benchmarks extract tasks from defect reports in real open-source projects. They provide the model with a historical version of the codebase and ask it to generate a patch that passes the corresponding tests.

    The result is judged by whether tests pass. Each task is classified as “solved” or “unsolved”, with no partial credit.

    This type of benchmark mainly evaluates:

    • Long-context understanding, such as reading cross-file dependencies.
    • Tool use, including searching, reading files, modifying code, and running tests.
    • Iterative repair capability.

    It is one of the most direct ways to reflect coding agent capability. However, older and highly public tasks may already be included in model training data, which can cause scores to be overestimated.

    3.5 Function-Calling Benchmarks: The Foundation of Agent Capability

    Function-calling benchmarks provide a set of tool definitions and conversational context. The model must select the correct tool at the correct time and fill in the parameters correctly.

    These benchmarks test two capabilities:

    • Single-turn selection.
    • Multi-turn chaining.

    Typical tasks include:

    • Selecting the correct tool among multiple distractors.
    • Chaining calls in dependency order, such as querying first and then placing an order.
    • Correcting a call after failure based on error messages.

    These benchmarks are highly automatable, but they only cover the “call” stage of the loop and cannot replace end-to-end evaluation.

    3.6 Scenario-Specific In-House Evaluation Sets: The Final Basis for Selection

    Public benchmarks help compare models broadly, but scenario-specific evaluation sets are the final basis for decision-making.

    The following steps are recommended:

    1. Collect real tasks: Select representative tasks from requirement tickets, operations tickets, defect repositories, and historical conversations.
    2. Stratified sampling: Stratify tasks by complexity (easy/medium/hard), tool type, and risk level, ensuring that the proportion of each task type matches online traffic.
    3. Design verifiers: Write automated scoring scripts for each task, such as checking final state or running tests. Tasks that cannot be automatically scored should be reviewed by humans.
    4. Control size and update cadence: Keep the initial evaluation set at dozens to hundreds of tasks for fast iteration. Update it quarterly to avoid overlap with training data or task leakage.
    5. Desensitize data: Remove personal information, account credentials, and business-sensitive content.

    4. Key Metrics: From “Can Do It Once” to “Can Do It Every Time”

    Metric category Specific metrics Meaning and use
    Correctness Pass@1 Success rate of a single run; reflects average performance
    Correctness Pass^k Proportion of k independent runs where all runs succeed; reflects reliability
    Process quality Average steps, redundant step ratio Measures execution efficiency and planning quality
    Process quality Invalid operation rate, loop detection count Exposes shortcomings in error recovery
    Efficiency First-step latency, per-step latency, task completion time Affects user experience and timeout rate
    Cost Average token consumption per task, cost per task Directly determines cost at scale
    Stability Timeout rate, context overflow rate, result variance Determines whether service-level requirements can be met
    Safety Unauthorized high-risk operation rate, prompt-injection resistance rate Controls production risk

    Among these, Pass^k is the most easily overlooked but most critical metric in agent scenarios.

    If the single-run success rate is p and runs are independent, the probability that all k runs succeed is approximately p^k:

    Single-run success rate k = 1 k = 3 k = 5 k = 10
    60% 60.0% 21.6% 7.8% 0.6%
    80% 80.0% 51.2% 32.8% 10.7%
    90% 90.0% 72.9% 59.0% 34.9%
    95% 95.0% 85.7% 77.4% 59.9%

    Small differences in single-run success rate are sharply amplified across multiple runs. For example, increasing success rate from 90% to 95% raises the all-success rate at k = 5 from 59.0% to 77.4%.

    For tasks that must run automatically multiple times per day, two models with similar Pass@1 may differ several times in actual usable proportion.

    Cost should be evaluated based on the average cost of completing a task, not on token price alone.

    A simplified formula is:

    1. effective cost per completed task = per-step token price × average tokens per task × (1 + retry rate × retry loss coefficient)

    If a model has a lower unit price but a much lower single-run success rate, it may require more retries. When retry overhead and per-run token consumption are comparable, the model with the lower unit price can end up with a higher effective cost per completed task.

    5. Building a Scientific and Reproducible Evaluation Plan

    5.1 Standardized Environments and Sandbox Isolation

    Agent evaluation must run in resettable and isolated sandboxes. It is recommended to provide an independent environment for each task using containerization.

    • Resettable: Restore to the initial snapshot before each run to avoid cross-task contamination.
    • Isolated: Limit file-system write scope and network access. Production credentials must never enter the evaluation environment.
    • Reproducible: Fix image versions, dependency versions, system time, and random seeds.

    5.2 Five Principles for Fair Comparison

    1. Use a unified execution framework: All models should use the same agent scaffold, including system prompt structure, tool definitions, and context-management strategy. Otherwise, the test measures the combined effect of “model + framework”.
    2. Use a unified resource budget: Set the same maximum steps, context limit, and timeout to avoid achieving higher success rates through unlimited retries.
    3. Align prompt adaptation: Different models respond differently to instruction styles. Reasonable prompt tuning is allowed for each model, but the tuning rules must be consistent across all candidate models.
    4. Fix decoding parameters: Specify temperature, random seed, and reasoning mode. If comparing different reasoning modes, provide both “same-mode” and “same-cost” comparison views.
    5. Use multiple repeated runs: Run each task at least 3–5 times and report mean values and confidence intervals, rather than a single best result.

    5.3 Example Evaluation Pipeline (Pseudocode)

    1. def evaluate(candidate, tasks, k=5, budget=None):
    2. results = []
    3. for task in tasks:
    4. outcomes = []
    5. for _ in range(k): # Multiple independent runs for Pass^k
    6. sandbox = create_sandbox(task.image) # Reset to the initial snapshot
    7. trace = run_agent(
    8. model=candidate,
    9. tools=task.tools, # All candidate models use the same tool definitions
    10. instruction=task.instruction,
    11. sandbox=sandbox,
    12. limits=budget, # Unified maximum steps and token budget
    13. )
    14. passed = task.verifier(sandbox) # The verifier checks final state or runs tests
    15. outcomes.append({
    16. "passed": passed,
    17. "steps": trace.step_count,
    18. "tokens": trace.total_tokens,
    19. "duration": trace.elapsed,
    20. "risk_actions": trace.high_risk_calls,
    21. })
    22. results.append({
    23. "task": task.id,
    24. "pass_at_1": mean(o["passed"] for o in outcomes),
    25. "pass_at_k": all(o["passed"] for o in outcomes),
    26. "cost": mean(o["tokens"] for o in outcomes),
    27. "latency_p95": percentile([o["duration"] for o in outcomes], 95),
    28. })
    29. return summarize(results) # Output per-dimension scores, confidence intervals, and capability-cost curve

    5.4 Result Analysis and Decision Outputs

    After evaluation, teams should not look only at a single total score. The output should include three parts:

    • Capability profile: A radar chart showing each candidate model’s performance in planning, tool calling, error recovery, and other dimensions, making weaknesses clear.
    • Capability-cost curve: Use average cost per task as the horizontal axis and task success rate as the vertical axis to identify the “cost-performance inflection point”. This is usually the best candidate for the default model.
    • Failure attribution: Classify failed samples by cause, such as planning errors, parameter errors, environment misunderstanding, or safety refusals. This helps determine whether the problem lies in the model, the prompt, or the tool design.

    6. Shared Patterns in Mainstream Model Capabilities in 2026

    Based on public evaluations and engineering practice, mainstream models show the following general patterns. Specific rankings and data should be based on official technical reports and the latest public evaluations at the time of each model’s release.

    6.1 Leading Closed-Source Models: Tiered Positioning by Task Complexity

    Leading models with adjustable reasoning tiers usually offer multiple tiers, from a “fast tier” to a “deep reasoning tier”.

    • Deep reasoning tier: Suitable for complex coding, long-horizon planning, and multi-step debugging. Task-completion rates under multi-round iteration are significantly higher, but latency and cost are also higher.
    • Standard or fast tier: Suitable for intent classification, information extraction, and simple tool calls. It responds quickly and costs less.

    The key selection point is: how large is the measured success-rate gap between the default tier and the deep reasoning tier, and how much extra cost is required to improve success rate by 10 percentage points? In scenarios where the gap is not obvious, the default tier should be preferred.

    6.2 Cost-Effective Mid-Tier Models: Handling High-Volume and Standardized Tasks

    Mid-tier models and models positioned for speed and cost-performance are stable at tasks such as structured output, single-turn or few-turn tool calls, document summarization, and rewriting.

    For tasks with fixed steps that can be constrained by workflows, the gap between mid-tier models and leading models is small, while cost can be lower by an order of magnitude.

    6.3 Open-Source Self-Hosted Models: For Data Compliance and Deep Customization

    Open-source models can be deployed in private environments and offer the following advantages:

    • Data stays within the organization’s domain, meeting compliance requirements in industries such as finance, healthcare, and government.
    • They can be fine-tuned on business data, and the inference framework and hardware can be chosen freely.
    • They avoid vendor lock-in, and costs can be predicted as scale changes.

    However, self-hosting also requires bearing hardware, operations, and model-upgrade costs. In low-concurrency or small-scale scenarios, these fixed costs may offset the unit-price advantage.

    6.4 Reasoning Mode and Cost-Performance Are Two Main Selection Lines

    “Stronger capability” and “lower cost” are not necessarily contradictory. Reasoning modes and cost-effective tiers can be combined to form a “high capability + low cost” option, while open-source self-hosting provides a “low cost + customizable” route.

    During selection, capability and cost should be weighed on the same coordinate system.

    7. Scenario-Based Selection Decision Matrix

    Scenario Task characteristics Key metrics Recommended strategy
    Complex coding and code generation Cross-file changes, debugging, multi-round testing Repository-level fix pass rate, Pass^k, average steps Prefer models with the strongest complex reasoning and stable long-context performance; enable deep reasoning mode when necessary; combine automated tests and human review
    Computer/terminal automation Long GUI or command-line operations Operation success rate, invalid operation rate, timeout rate Choose models with strong environment interaction and error recovery; restrict executable commands; require human confirmation for high-risk operations
    Customer service and business assistants Medium complexity, high concurrency, strong latency requirements First-step latency, intent recognition accuracy, per-turn cost Use cost-effective mid-tier models as the main option; add workflow constraints; fall back to humans or stronger models
    Batch data processing Fixed steps, separable, no state dependency Throughput, structured output compliance rate, cost per task Use fast low-cost models; enforce strict output-format constraints; send failed items to a retry queue
    R&D testing and operations inspection Script generation, log analysis, fault diagnosis Fault localization accuracy, false-positive rate Use open-source or cost-effective models for routine inspection; escalate difficult issues to higher-capability models
    Data-sensitive and private deployments Data cannot leave the domain; customization is required Deployment cost, response latency, fine-tuning effectiveness Choose self-hostable open-source models, or access cloud model services through platform-level data isolation solutions

    7.2 Layered Routing: Assigning Models by Difficulty

    A single model rarely achieves the strongest capability, the fastest speed, and the lowest cost at the same time. A more practical approach is to build layered routing, where simple tasks use low-cost paths and only complex tasks use high-capability models.

    Example code:

    1. def route(task, tiers):
    2. # tiers is a list of model tiers sorted from low to high capability
    3. # each item contains the model, capability score threshold, and budget
    4. tier = choose_tier_by_complexity(task, tiers) # Choose the initial tier based on task complexity and risk level
    5. result = run_with_budget(tier.model, task, tier.budget)
    6. if not result.passed and result.confidence < tier.escalate_threshold:
    7. # If the task fails or confidence is insufficient, escalate to a higher-capability tier
    8. # Log the routing decision for later optimization
    9. result = run_with_budget(tiers[-1].model, task, tiers[-1].budget)
    10. return result

    Three practical points are important:

    • Set an escalation limit: Escalate at most 1–2 times to prevent cost from spiraling out of control.
    • Routing strategy can be learned: Start with rule-based decisions. After accumulating enough routing logs, train a small classifier to predict task difficulty.
    • Continuously monitor routing quality: Regularly measure the task distribution and success rate handled by each tier to prevent many difficult tasks from being routed to lower-tier models by mistake.

    8. Engineering Systems That Support Agent Capability

    A model is only one component of an agent system. The same model can perform very differently under different engineering frameworks. The following engineering factors are as important as model capability.

    8.1 Tool Layer: Discoverable, Constrainable, and Auditable

    • Use standardized tool descriptions, including name, description, parameter schema, and return format. Clearly state “when to use” and “when not to use”.
    • Keep the number of tools within a reasonable range and group them by business domain to avoid selection difficulty.
    • Design idempotency keys and precondition checks for write operations to prevent side effects from repeated execution.
    • Log all tool calls in a unified audit trail, including caller, parameter summary, result, and latency.

    8.2 Context Management Layer: Preserving Critical Information Within a Limited Window

    • Summarization and compression: Periodically generate structured summaries of historical messages, preserving the goal, completed steps, and pending tasks.
    • Externalized state: Write task progress and intermediate outputs to workspace files so the task can be recovered from files when context is lost.
    • Layered memory: Put stable business knowledge into a retrieval system and load it on demand, rather than stuffing everything into the system prompt.
    • Output trimming: Summarize or paginate overly long logs and file contents to prevent a single observation from consuming too much context.

    8.3 Security and Sandbox Layer: Least Privilege by Default

    • By default, an agent should have only the minimum permissions needed to complete the task.
    • Limit file writes to the working directory and allow network access through domain allowlists.
    • Require secondary confirmation for high-risk operations such as deletion, overwriting, sending information externally, and payments.
    • Mark the source of external content from webpages, files, and tool returns to reduce the risk of indirect prompt injection.

    8.4 Observability and Continuous Optimization Layer

    It is recommended to use cloud-platform capabilities to build the operations system. The following are general capability categories; specific products depend on the cloud services actually used:

    • Use log services to centrally collect agent traces, including input, output, latency, and state changes for each step.
    • Use monitoring and alerting to set threshold alerts for metrics such as success rate, P95 latency, cost per task, and timeout rate.
    • Use object storage to archive evaluation datasets, run traces, and failure samples.
    • Use container platforms to elastically scale sandbox and model inference resources.
    • Cloud platforms such as Baidu AI Cloud can also be used to unify access to multiple models, supporting unified invocation, switching, and cost statistics where applicable.

    When a model version is upgraded, first complete regression testing on the offline evaluation set, then validate with small-traffic canary releases, and gradually ramp up traffic. Each upgrade should retain a complete comparison report, preventing “the new version is better” from remaining a subjective judgment.

    9. Common Pitfalls and How to Avoid Them

    1. Looking only at overall leaderboards: Overall rankings cannot reflect scenario-specific performance. Use per-dimension scores and in-house evaluation sets instead.
    2. Running only a single evaluation: Single-run results are highly variable. Run each task at least 3–5 times and report Pass^k.
    3. Ignoring the execution framework: Fix the agent scaffold before comparing models. Otherwise, it is impossible to tell whether improvement comes from the model or prompt optimization.
    4. Underestimating long-horizon task difficulty: A task that succeeds in a demo may show a large success-rate drop after dozens of runs. Always run long-horizon stress tests before launch.
    5. Ignoring security risks: The stronger the capability, the greater the potential impact of unauthorized or destructive behavior. Permission controls and confirmation mechanisms should be established before capability expansion.
    6. Ignoring hidden costs: Retries, context expansion, human takeover, and operations effort should all be included in total cost.
    7. Building the evaluation set once and never updating it: Model iterations and business changes will make the evaluation set obsolete. It needs regular maintenance and new hard cases.

    10. Summary and Selection Checklist

    The core of agent capability evaluation is to use reproducible methods to answer one question: Which model can reliably complete tasks in my scenario at an acceptable cost?

    Selection is not about choosing the single “strongest model”. It is about matching different capability tiers to different tasks and using engineering methods to compensate for model weaknesses.

    Before implementation, teams can check the following items one by one:

    • Have we defined core scenarios and quantitative acceptance criteria, such as success rate, reliability, latency, and cost?
    • Have we built an in-house evaluation set covering easy, medium, and hard tasks, and designed automated verifiers?
    • Have we completed fair comparisons under unified sandboxes, frameworks, and budgets, and reported Pass^k and confidence intervals?
    • Have we produced a capability-cost curve and identified default models and escalation models?
    • Have we designed layered routing, retry limits, and failure fallback processes?
    • Have we made targeted designs for the tool layer, context management, and security sandbox?
    • Have we established logging, monitoring, and canary-release mechanisms to support continuous model iteration?

    By following the path of “define capability → build evaluation → analyze metrics → match scenarios → engineer safeguards → continuous regression”, teams can continue making data-driven model selection decisions even in an environment of rapid model iteration.