LLM Tool Calling and Coding Evaluation: A Practical Guide

1. Bottom Line: Evaluate Whether the Model Can Work Reliably in Your Scenario

Evaluating the real-world performance of large language models in tool calling and coding requires a quantifiable, repeatable, and regression-ready evaluation method. This method supports model selection, test-set construction, and production validation. Instead of relying only on a leaderboard percentage, answer three questions first:

• Can the model select the right tool and fill in the correct parameters?
• Can the generated code actually run in an isolated environment and pass tests?
• Does capability regress after a model or prompt update?

Key points:

• Tool calling evaluation focuses on tool selection, argument filling, multi-step orchestration, error recovery, and end-to-end success rate.
• Coding capability evaluation focuses on code generation, code repair, terminal task execution, scientific computing implementation, and whether the output passes tests in a sandbox.
• Before evaluation, fix the model version, system prompt, temperature, top_p, sampling count, tool definitions, and execution environment; otherwise results are not reproducible.
• The test set must be isolated from training data, retrieval libraries, and example libraries to avoid leakage. Sampling from real business data must be de-identified.
• Code and tool calls generated by the model must run in an isolated sandbox. For destructive or billing-related tools, use mock stand-ins.

2. Tool Calling Evaluation: From Selecting the Right Tool to Closing the Loop

2.1 What Is Actually Being Evaluated?

Tool calling capability is not merely whether the model outputs a function name. It measures whether the model can complete the full chain after receiving a set of external tool or API definitions: understand the task, select a tool, generate a correct call, parse the result, and continue reasoning or correct errors.

Common evaluation dimensions include:

• Tool selection accuracy: whether the model chooses the target tool from multiple candidates.
• Argument filling accuracy: whether required arguments are complete and optional argument values are valid.
• Multi-step orchestration: whether the model can order multiple tool calls correctly and pass intermediate results.
• Error recovery: whether the model can retry, switch to another tool, or provide a reasonable fallback when a tool returns an exception, times out, or returns empty results.
• End-to-end task success rate: whether the final business goal is achieved, not just whether one call is correct.

Dimension	Key question	Common verification method
Tool selection accuracy	Whether the expected tool is selected	Compare function name or action name
Argument filling accuracy	Whether arguments are complete, correctly typed, and valid against enums	Field assertions, schema validation
Multi-step orchestration	Whether step order and intermediate results are correct	Call-chain assertions, state machine validation
Error recovery	Whether the goal can still be completed after an exception	Inject exceptions, observe retries or fallbacks
End-to-end task success rate	Whether the user goal is completed	Final result assertion or manual spot check

2.2 Define Evaluation Conditions Before Starting

Tool calling evaluation often produces different results under the same question because evaluation criteria differ significantly. Before evaluation, at least fix the following conditions:

1. Distinguish native function calling from ReAct/JSON-mode prompts.
   Native function calling usually relies on the model API to output structured tool calls. ReAct or JSON-mode prompts require the model to output actions, reasoning traces, or JSON structures through prompt instructions, followed by additional parsing. Their failure points differ, so they should not be compared directly on the same leaderboard.

2. Define single-turn evaluation versus multi-turn session evaluation.
   Single-turn evaluation only checks whether one model output is correct. Multi-turn evaluation feeds tool results back to the model and evaluates follow-up reasoning, retries, error correction, and closed-loop completion. Strong single-turn performance does not necessarily mean strong multi-turn closed-loop performance.

3. Fix model version, prompts, temperature parameters, and sampling count.
   Model version, system prompt, temperature, top_p, maximum output length, and tool descriptions all affect results. If multiple samples are used, specify whether the metric is pass@1, pass@3, or pass@k, and report the statistical method.

4. Prevent test-set leakage.
   The test set must be isolated from training data, retrieval libraries, and few-shot example libraries. If test samples have been used in training, retrieval, or prompt examples, pass rates can be artificially inflated.

2.3 Minimal Tool Definition Example

Using a common function-calling convention, tool capability descriptions are usually structured metadata:

{
  "name": "query_weather",
  "description": "Query real-time weather for a city",
  "parameters": {
    "type": "object",
    "properties": {
      "city": {
        "type": "string",
        "description": "City name, for example, Beijing"
      },
      "unit": {
        "type": "string",
        "enum": ["celsius", "fahrenheit"]
      }
    },
    "required": ["city"]
  }
}

For evaluation, give the model a natural-language instruction, for example, “What is the temperature in Beijing today? Use Celsius.” Then verify whether the model selects query_weather, whether city is extracted correctly, and whether unit is omitted or within the valid enum. Automating and batch-running this process produces quantitative scores.

2.4 Clarify Metric Direction

Different metrics point in different directions and must be explained in advance:

• Tool selection accuracy: higher is better.
• Argument filling accuracy: higher is better.
• End-to-end task success rate: higher is better.
• Average number of calls: usually lower is better, but state whether failed retry steps are included.
• Average tool-call latency: lower is better.
• End-to-end latency: lower is better; it includes model inference, tool execution, and result parsing.

Do not look only at the overall pass rate. One model may select tools accurately but often extract arguments incorrectly; another may be stronger in multi-step orchestration but weaker in error recovery. Only by breaking metrics down according to the business scenario can real weaknesses be identified.

3. Coding Capability Evaluation: Generation, Repair, Terminal Execution, and Scientific Computing

3.1 Coding Capability Is Not a Single Metric

Coding capability can be divided into at least three categories:

• Terminal/command-line tasks: evaluate whether the model can complete builds, debugging, script writing, log analysis, and environment repair in a real shell environment through commands.
• Code generation and repair: give the model a requirement, a defect, or a repository issue, and let it produce runnable code or directly fix the defect.
• Scientific computing/numerical programming: cover mathematical modeling, data processing, algorithm implementation, and scientific library usage, evaluating the model’s understanding of domain formulas, common libraries, and numerical methods.

3.2 Public Benchmarks and Leaderboards to Consider

The following lists names and focus areas only, without comparing scores, to help establish comparison coordinates during model selection:

• Terminal/command-line tasks

  • Terminal-Bench: evaluates terminal task execution, command orchestration, environment troubleshooting, and script-based processing.
  • SWE-bench: evaluates real repository issue repair; it often involves code localization, patch generation, and test verification, and some tasks require command-line operations.

• Code generation and repair

  • HumanEval: evaluates function-level code generation and unit test pass rate.
  • MBPP: evaluates basic programming problems and function-level implementation.
  • BigCodeBench: evaluates complex library usage, multi-step code generation, and engineering constraints.
  • SWE-bench: evaluates repository-level defect repair and regression validation.

• Scientific computing/numerical programming

  • SciCode: evaluates scientific computing, numerical algorithms, and scientific programming tasks.
  • DS-1000: evaluates data science, numerical processing, library usage, and analysis script writing.

These benchmarks emphasize different capability areas. Directly comparing “which model is stronger” across leaderboards has limited value. It is more useful to focus on “which model is stronger on the task types relevant to you.”

3.3 Why Different Coding Leaderboards Can Reach Different Conclusions

Even when all are called “coding capability,” different benchmarks may produce opposite conclusions. Common reasons include:

• Different task distributions: command-line tasks emphasize engineering workflows, while scientific computing emphasizes mathematics and domain knowledge.
• Different scoring criteria: some benchmarks check whether code runs, some check whether all unit tests pass, and some also consider code quality and maintainability.
• Different prompts and tool environments: whether the model is allowed multiple trial-and-error rounds, whether execution feedback is provided, and whether search or file reading is allowed can significantly affect results.
• Different execution methods: some evaluations only inspect static generated results, some require actual code execution, and some require terminal operations inside an isolated sandbox.

Therefore, during model selection, focus on whether the benchmark closely matches your own tasks rather than simply citing an overall score.

4. Build Your Own Evaluation System: A Four-Step Approach

Rather than relying entirely on external leaderboards, build a lightweight, reproducible, and regression-ready evaluation system for your own business. A four-step approach is recommended:

4.1 Step 1: Define Tasks and Metrics

First, break the business into typical tasks, for example:

• Locate anomalies from logs.
• Generate data-processing scripts from requirements.
• Select and call tools according to API documentation.
• Fix a defect in a code repository.
• Create files, run tests, and analyze errors in a terminal.

Set automatically verifiable metrics for each task type, for example:

• Tool calling: function name match rate, argument accuracy, end-to-end success rate, average number of calls, average latency.
• Code generation: unit test pass rate, syntax pass rate, repair success rate, timeout rate.
• Terminal tasks: command execution success rate, file assertion pass rate, in-sandbox task completion rate, error recovery rate.

4.2 Step 2: Build the Test Set

Do not simply write “sample from real requirements.” The test set should cover common scenarios, boundary scenarios, and exception scenarios, with fixed inputs for horizontal comparison.

Below are three actionable test-case examples.

Example 1: Tool Calling Test Case

{
  "id": "tool-weather-001",
  "prompt": "What is the temperature in Beijing today? Use Celsius.",
  "tools": [
    {
      "name": "query_weather",
      "description": "Query real-time weather for a city",
      "parameters": {
        "type": "object",
        "properties": {
          "city": { "type": "string" },
          "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] }
        },
        "required": ["city"]
      }
    }
  ],
  "expected": {
    "type": "tool_call",
    "function_name": "query_weather",
    "required_args": {
      "city": "Beijing"
    },
    "optional_args_enum": {
      "unit": ["celsius", "fahrenheit"]
    },
    "rule": "The function name must match; city must be present and valid; unit may be missing, but if provided it must be within the enum."
  }
}

The main judgment is tool-call assertion: function name, required arguments, and optional argument enums.

Example 2: Code Repair Test Case

id: code-fix-parser-001
prompt: Fix parse_date in utils/parser.py so that it passes tests
repo_context: |
  utils/parser.py
  tests/test_parser.py
expected:
  type: code_execution
  sandbox_command: pytest tests/test_parser.py
  timeout_seconds: 60
  rule: "Run pytest inside the sandbox; the exit code must be 0 and all tests must pass."

The main judgment is code execution assertion: run tests in an isolated sandbox, enforce a timeout, and check whether they pass.

Example 3: Terminal Task Test Case

id: terminal-create-report-001
prompt: Create report.txt in /workspace, with content containing "ERROR: timeout"
expected:
  type: terminal_execution
  sandbox_enabled: true
  checks:
    - file_exists: /workspace/report.txt
    - file_contains: "ERROR: timeout"
  timeout_seconds: 30
  rule: "Execute the command in an isolated sandbox; after completion, check that the file exists and contains the specified text."

Terminal tasks must also run in an isolated sandbox. Do not run model-generated commands directly on production machines or development hosts.

Sample size and variance also matter. If the test set has only a dozen cases, whether a single case passes will noticeably affect the overall score. Generally, prepare at least 50 to 100 samples per task type; critical business tasks can be expanded further. If the sample size is small, report the variance range, standard deviation, or multi-run average, and do not draw conclusions from a single run.

Security and compliance should also be considered when designing the test set:

• Execute model-generated code and tool calls only in isolated sandboxes.
• Apply least-privilege access to tool credentials.
• Use mock stand-ins for destructive, write, or billing-related tools.
• De-identify logs and returned results.
• When sampling from real requirements or production logs, remove personal information, sensitive parameters, and internal keys.

4.3 Step 3: Automate Execution and Verification

The evaluation executor should unify “invoke model -> execute/verify -> record results.” The key point is that verification cannot be described only as executor.verify; the verification method must be explicit.

Verification should be divided into two types:

1. Tool-calling assertions
   Check function names, required arguments, optional argument enums, multi-step call order, and whether intermediate results are passed correctly.

2. Code execution assertions
   Write files, run commands, execute unit tests, check output, and enforce timeout and resource usage inside an isolated sandbox.

A reusable evaluation skeleton is shown below:

from dataclasses import dataclass
from typing import Any
@dataclass
class EvalCase:
    id: str
    prompt: str
    tools: list[dict[str, Any]]
    expected: dict[str, Any]
def run_eval(cases: list[EvalCase], model_client, executor):
    if not cases:
        raise ValueError("cases must not be empty to avoid division by zero or empty-set misjudgment")
    results = []
    for case in cases:
        # The case fields include id, prompt, tools, and expected.
        # For tool-calling tasks, tools is the list of tool definitions.
        # For code-generation or terminal tasks, tools can be an empty list, but expected must still be passed.
        resp = model_client.invoke(case.prompt, tools=case.tools)
        # Verification methods:
        # 1. Tool-calling assertions: function name, required arguments, optional argument enums, and call order.
        # 2. Code execution assertions: write files, run tests, and enforce timeout and resource limits inside an isolated sandbox.
        # 3. Terminal task assertions: execute commands inside an isolated sandbox and check files, output, or exit codes.
        passed = executor.verify(resp, case.expected)
        results.append({
            "id": case.id,
            "passed": passed,
            "steps": resp.get("steps", 1),
            "latency_ms": resp.get("latency_ms"),
        })
    pass_rate = sum(r["passed"] for r in results) / len(results)
    return {
        "pass_rate": pass_rate,
        "details": results,
    }

In practice, executor.verify should include at least:

• Tool-calling schema validation.
• Function name and argument assertions.
• Multi-step call-chain order validation.
• Sandboxed execution of code or terminal commands.
• Timeout limits, memory limits, and network restrictions.
• Retrieval of test reports or execution logs.
• Recording failure reasons for manual spot checks.

4.4 Step 4: Continuous Regression

After every update to the model, prompt, tool definitions, retrieval content, or execution environment, rerun the evaluation. Continuous regression prevents capability regression, such as a tool call that previously passed becoming invalid after a prompt change, or a code repair that previously succeeded failing because the context format changed.

It is recommended to retain version identifiers for each evaluation, for example:

• Model version.
• Prompt version.
• Test-set version.
• Tool definition version.
• Execution environment version.
• Evaluation script version.

Only when versions are traceable are evaluation results comparable.

4.5 Common Evaluation Mistakes and How to Avoid Them

• Looking only at the overall score: overall scores can hide weaknesses in individual items. Break results down by tool selection, argument filling, code pass rate, and error recovery rate.
• Test-set leakage: if test samples enter training, retrieval, or example libraries, scores can be inflated. Isolate and deduplicate them.
• Running only once: a single run can vary widely. Use multiple samples and report mean, standard deviation, or pass@k.
• Ignoring cost and latency: a model may meet capability requirements but still be unsuitable for production if it is too expensive or too slow.
• Using pass@1 when pass@k is required: if the business allows retries or multiple rounds, clarify the difference between pass@1 and pass@k.
• Unclear metric direction: for example, “average number of calls” is usually better when lower, but specify whether failed retries are included.
• No security isolation: running model-generated code directly in a production environment creates risks of data leakage, privilege abuse, and billing exposure.

5. Engineering Implementation: Focus on Safety, Isolation, and Regression

Once the evaluation system is built, the most easily overlooked part of implementation is the safety and isolation of the execution environment. Both tool calling and code execution must assume that model output cannot be fully trusted, so verification must happen in a controlled environment.

During implementation, pay attention to the following:

• List scenarios before model selection: whether the application is mainly single-tool calls or complex orchestration; whether it involves business scripts or scientific computing; whether it requires code generation or terminal repair.
• Focus on end-to-end behavior rather than a single point: for tool-calling applications, include execution feedback loops in evaluation; for coding applications, actually run the code for validation.
• Retain the test set and every run result by version: record test-set version, model version, prompt version, execution logs, and failure cases.
• Connect metrics to a monitoring dashboard: visualize pass rate, latency, failure rate, timeout rate, and number of calls.
• Set regression alert thresholds for critical metrics: for example, trigger an alert when end-to-end pass rate drops beyond a predefined threshold, preventing silent capability regression after model or prompt updates.
• Keep manual spot checks: after automated evaluation passes, humans should still inspect error cases to avoid inflated metrics or misjudgment.

Cost and latency should also be evaluated on the premise that capability is sufficient. Unit price, concurrency capacity, response latency, and retry cost often determine whether a model can enter production. During evaluation, record token consumption, number of tool calls, average latency, and retry costs for failures.

6. Conclusion

Tool calling and coding capabilities are two touchstones for whether a large language model can do real work. Building a multi-dimensional, reproducible, and regression-ready evaluation system helps find the right balance between capability, cost, and latency for your own scenario.