LLM Agent Model Selection: Four Evaluation Dimensions and a Production Workflow

    This article explains what agent capability means in practice and how to evaluate foundation models for agent workloads. It covers four core dimensions—task planning, tool calling, result validation and self-checking, and error recovery—and then describes how to build a layered test set, define quantitative metrics, structure test cases, automate evaluation, and run a model selection process. It also outlines practical criteria for assessing cost, latency, engineering support, managed services, and continuous regression testing.

    1. What Do Agent Capabilities Actually Mean?

    Agent capability refers to a model’s ability to autonomously complete the full closed loop of understanding the goal → planning steps → calling tools → checking results → correcting errors without step-by-step human intervention.

    This differs from single-turn question answering in three important ways:

    • Longer task horizons: Agent tasks often require dozens to hundreds of steps. If any single tool call fails, the entire task may fail, and the failure rate rises quickly as the number of steps increases.
    • More complex state management: The model must remember completed operations in a long context, distinguish completed from unfinished subgoals, and avoid executing the same operation repeatedly.
    • Feedback-driven decisions: The result of each step is returned to the model as new input. The model must adjust its plan based on actual results instead of mechanically following a prearranged workflow.

    Therefore, deciding whether a model is suitable for agent work cannot rely only on conversational fluency or knowledge question-answering accuracy. Evaluation must be designed around each part of the closed loop.

    Note that evaluation conclusions depend on the tool set, prompts, and runtime environment. If any of these changes, the regression tests should be run again.

    2. Four Evaluation Dimensions and Pass Criteria

    1. Task Planning

    Planning is effectively the agent’s brain. Evaluation should focus on three areas:

    • Reasonable goal decomposition: Given a complex goal such as “analyze a dataset and generate a report,” can the model break it into subtasks that are correctly ordered and appropriately scoped?
    • Foresight: A good model anticipates risks. For example, it should check whether dependencies exist before installation rather than executing first and remediating after a failure.
    • Dynamic replanning: When intermediate results differ from expectations, can the model adjust subsequent steps in time instead of insisting on the original plan?

    2. Tool Calling

    Tool calling is central to putting agents into production. Evaluation should cover:

    • Tool selection accuracy: When multiple tools have similar functions, can the model select the most appropriate one?
    • Parameter construction correctness: Parameter types, required fields, and format constraints must match the tool definition. In practice, many failures come from parameter hallucination, such as writing a numeric parameter as a string or omitting a required field.
    • Concurrency and sequencing decisions: Can the model identify which calls have no dependencies on each other and can run in parallel, and which calls must run sequentially after the previous step finishes?

    Below is an example tool definition and a corresponding parameter validation function. The validator checks whether the tool name exists, whether required parameters are present, whether types are correct, and whether string formats match their regular expressions.

    1. import re
    2. TYPE_MAP = {"string": str, "integer": int, "number": (int, float), "boolean": bool}
    3. tools = [{
    4. "name": "query_order",
    5. "description": "Query order details by order ID",
    6. "parameters": {
    7. "type": "object",
    8. "properties": {
    9. "order_id": {"type": "string", "pattern": r"^ORD\d{10}$"}
    10. },
    11. "required": ["order_id"]
    12. }
    13. }]
    14. def validate_call(call, tools_schema):
    15. """Return (passed, reason). call is of the form {"name": ..., "arguments": {...}}."""
    16. tool = next((t for t in tools_schema if t["name"] == call.get("name")), None)
    17. if tool is None:
    18. return False, f"Unknown tool: {call.get('name')}"
    19. schema = tool["parameters"]
    20. args = call.get("arguments", {})
    21. for key in schema.get("required", []):
    22. if key not in args:
    23. return False, f"Missing required parameter: {key}"
    24. for key, value in args.items():
    25. spec = schema["properties"].get(key)
    26. if spec is None:
    27. return False, f"Undefined parameter: {key}"
    28. expected_type = spec["type"]
    29. # bool is a subclass of int, so it must be excluded separately;
    30. # otherwise True would pass as an integer.
    31. if isinstance(value, bool) and expected_type != "boolean":
    32. return False, f"Invalid parameter type: {key}"
    33. if not isinstance(value, TYPE_MAP[expected_type]):
    34. return False, f"Invalid parameter type: {key}"
    35. if "pattern" in spec and isinstance(value, str):
    36. if not re.fullmatch(spec["pattern"], value):
    37. return False, f"Format does not match requirement: {key}"
    38. return True, "ok"
    39. # Examples
    40. print(validate_call({"name": "query_order", "arguments": {"order_id": "ORD0000000001"}}, tools))
    41. # (True, 'ok')
    42. print(validate_call({"name": "query_order", "arguments": {"order_id": 1}}, tools))
    43. # (False, 'Invalid parameter type: order_id')

    3. Result Validation and Self-Checking

    After receiving a result from a tool, the model should be able to:

    • Identify abnormal responses such as empty results, error codes, and timeout messages, and handle them according to their type.
    • Confirm suspicious results again instead of accepting them directly.
    • Accurately determine that the task is complete when results meet expectations, avoiding redundant repeated calls.

    4. Error Recovery

    Error recovery is a key indicator that distinguishes model maturity. Evaluation should actively inject faults and observe the following behaviors:

    • After a tool error, does the model retry blindly, switch to a fallback option, or give up immediately?
    • On retry, does the model correct the parameters, such as fixing malformed input, or submit the same request unchanged?
    • After repeated failures, can the model clearly explain the cause of the failure? If it fabricates a “successful” result, it should fail the evaluation directly.

    3. Building Your Own Evaluation System

    1. Layered Test Set

    It is recommended to divide test cases into three layers:

    • Basic capability layer: Single-step tool calls and simple parameter construction. Use this layer to quickly screen out clearly unqualified models.
    • Task-chain layer: Multi-step tasks with 5 to 20 steps, used to assess planning capability and state maintenance.
    • Adversarial layer: Inject ambiguous instructions, missing dependencies, or malicious tool outputs to assess robustness and security boundaries.

    2. Quantitative Metrics and Measurement Definitions

    Every metric must have a clearly defined calculation method; otherwise, results from different evaluators cannot be compared.

    • End-to-end success rate: The proportion of tasks ultimately completed. This is the most important metric. The success criterion should be defined in advance by the business side and written as rules that can be judged automatically.
    • Average steps and token consumption: Reflect efficiency and cost. A clearly excessive number of steps usually indicates inefficient planning or a loop.
    • Tool-call accuracy: The proportion of calls in which the tool selection is correct and the parameters are valid.
    • Recovery success rate: The proportion of tasks that can still be completed after fault injection. The fault injection ratio should be fixed; for example, randomly inject one fault per test case, and fix the random seed for reproducibility.

    The same test case should be run at least three times. Use the average and record the variance to reduce errors caused by model output randomness.

    3. Test Case Checklist

    Each test case should contain at least the following fields:

    Case ID Task Goal Tool Set Injected Fault Success Criteria Metric Definition
    QRY-001 Query a specified order and report its logistics status query_order, query_logistics The first call returns a timeout The final answer contains the correct logistics status, and the order ID matches the input End-to-end success, steps, recovery success
    PLAN-004 Summarize three reports and generate a weekly report read_file, summarize, write_report One of the files does not exist The report notes the missing file and does not fabricate data End-to-end success, replanning correctness

    4. Automated Evaluation Framework

    The framework below relies on two external interfaces: model_client, which handles interaction with the model, and tool_env, which executes tools and validates results. Implement these interfaces according to your actual environment.

    1. from statistics import mean
    2. class AgentEvaluator:
    3. def __init__(self, model_client, tool_env):
    4. self.client = model_client
    5. self.env = tool_env
    6. def run_case(self, task, max_steps=50):
    7. messages = [{"role": "user", "content": task}]
    8. for step in range(1, max_steps + 1):
    9. resp = self.client.chat(messages, tools=self.env.tools)
    10. if resp.is_final_answer:
    11. ok = self.env.verify(resp.answer)
    12. return {
    13. "success": ok,
    14. "steps": step,
    15. "answer": resp.answer,
    16. "reason": "" if ok else "Final answer failed verification",
    17. }
    18. result = self.env.execute(resp.tool_call)
    19. messages.append(resp.to_message())
    20. messages.append({"role": "tool", "content": result})
    21. return {
    22. "success": False,
    23. "steps": max_steps,
    24. "answer": None,
    25. "reason": "Maximum step limit exceeded",
    26. }
    27. def evaluate(self, cases):
    28. stats = [self.run_case(c) for c in cases]
    29. return {
    30. "success_rate": mean(s["success"] for s in stats),
    31. "avg_steps": mean(s["steps"] for s in stats),
    32. }

    Whether a case succeeds or fails, the return value includes the success, steps, and answer fields so that evaluate can aggregate results consistently. When the step limit is exceeded, steps is recorded as the limit so that failed cases are not omitted from step statistics.

    4. Selection Workflow

    1. Define the scenario before choosing the model. High-frequency, low-complexity automation tasks such as log summarization and scheduled inspection have relatively low requirements for model capability, so cost and latency can be prioritized. Long-chain tasks that depend heavily on tools, such as code engineering and data analysis, should focus evaluation on error recovery and long-context consistency.

    2. Evaluate with your own data instead of relying on public leaderboards. Public benchmarks reflect general capabilities. The tool sets, instruction styles, and domain terminology in a business can differ significantly from benchmark data distributions. It is recommended to construct 100 to 200 high-quality test cases from real business logs; their conclusions are usually more reliable than leaderboard rankings.

    3. Focus on engineering support, not only the model. The model is only one part of an agent system. Tool definition quality, context management strategy, and retry and fallback mechanisms also determine whether the system can run stably.

    4. Use explicit criteria when evaluating managed services. If a team lacks the ability to build its own evaluation and deployment, it can evaluate model services provided by major cloud platforms. The assessment should consider:

      • Whether structured output and JSON Schema constraints are supported.
      • Whether streaming output and parallel tool calling are supported.
      • Whether call logs, monitoring and alerting, and usage statistics are provided.
      • Whether data storage location and private deployment options meet compliance requirements.
      • Whether multiple foundation models can be conveniently compared side by side on the same infrastructure.
    5. Roll out gradually and keep regressing continuously. Model versions iterate frequently, and any selection conclusion has a limited shelf life. It is recommended to include the evaluation set in the CI process. Before a new model goes live, first run a regression on shadow traffic to confirm that key metrics have not degraded, and then increase traffic gradually.

    Below are reference priorities by scenario. Specific values should be adjusted according to the business:

    Scenario Error Recovery Tool-Call Accuracy Cost and Latency Long-Context Consistency
    Log summarization, scheduled inspection Low Medium High Low
    Automated customer support ticket handling High High Medium Medium
    Code engineering, data analysis High High Low High

    5. Summary

    The core of agent capability evaluation is to break down “whether the model can reliably complete the closed loop” into quantifiable, reproducible engineering metrics. Task planning, tool calling, result validation, and error recovery are all essential. Selection conclusions should be based on an evaluation system built from your own business data, while cost, latency, and engineering support should be considered together. By building an evaluation pipeline according to the method described here, teams can continuously select suitable and reliable foundation models for agent workloads as models iterate rapidly.