LLM Agent Capability Evaluation and Engineering Practice
This article explains how to evaluate LLM agents for engineering use. It covers why leaderboard rankings are not enough for model selection, how to break agent capability into planning, tool use, state maintenance, and error recovery, and how to build a minimal evaluation loop. It also covers per-task cost estimation with cached input pricing, adapter layers for switching models, tool definition conventions, training-inference template consistency, and security practices such as least-privilege access and human approval for irreversible actions.
1. Why Model Selection Should Not Rely Only on Leaderboard Rankings
Model versions change quickly. After each new release, the community usually discusses composite scores, code generation rankings, and personal impressions. This information has reference value, but using it directly for engineering decisions can easily lead to errors.
There are three main reasons.
First, a composite score is a weighted result of several subtasks. The evaluator sets the weights, and they may not match your business scenario. For example, if your system relies mainly on multi-turn tool calling, a high score on code completion subtasks tells you little.
Second, different evaluations use different settings. Prompt templates, decoding parameters, reasoning effort levels, and context length limits all affect results. Scores from two leaderboards may look close, but they may not be directly comparable.
Third, the task distribution of a public evaluation set may differ greatly from your team’s real data. A model that performs well on a public set may behave very differently on internal documents, private codebases, or specific business workflows.
A more practical approach is to treat external evaluations as initial screening, then validate with your own task set. Initial screening narrows the candidate pool. Validation shows how well a model works in your specific scenario.
2. Breaking Down and Designing Evaluation Dimensions
2.1 General Capability
General capability metrics show the overall level a model reaches. These metrics usually cover reasoning, knowledge question answering, mathematics, and coding. When you use them, check two things: the weight of each subtask, and whether the evaluation version and inference settings match the configuration you plan to use.
2.2 Agent Capability
Agent capability measures how a model performs in multi-step tasks. It includes the following areas.
- Planning: whether the model can break a vague goal into executable steps.
- Tool use: whether the model selects the correct tool and generates a call request that matches the parameter definition.
- State maintenance: whether the model remembers earlier constraints and intermediate results across multiple turns.
- Error recovery: when a tool returns an error or an unexpected result, whether the model adjusts its strategy instead of repeating the same action.
A single score cannot summarize these capabilities well. It is better to design end-to-end tasks and record every tool call.
2.3 Code and Frontend Generation
Code generation quality depends strongly on reasoning effort. The same model can show very different latency and output length at different effort levels. When comparing models, fix the effort setting. Otherwise, the results do not mean much. For frontend generation tasks, add manual review to check whether pages run, whether interactions are complete, and whether the visual design is consistent.
2.4 Example Evaluation Framework
The following example shows a minimal evaluation loop. The core idea is to fix the task set, record unified metrics, and allow the scoring function to be replaced.
def run_eval(model_client, tasks, scorer):"""Run the model on the same task set and aggregate unified metrics"""records = []for task in tasks:reply = model_client.chat(messages=task["messages"],tools=task.get("tools"),)records.append({"task_id": task["id"],"score": scorer(task, reply),"tool_call_count": len(reply.get("tool_calls", [])),"output_tokens": reply["usage"]["output_tokens"],})return summarize(records)
For every experiment, record the model version, decoding parameters, reasoning effort, prompt template hash, and task set version. Without these records, reproducing the experiment later is very difficult.
3. From Unit Price to Per-Task Cost
3.1 Pricing Structure
Most LLM services charge by the number of input and output tokens. Some services also offer cache pricing for repeated input prefixes. The price for cache-hit tokens is usually much lower than the price for uncached tokens. Therefore, calculate per-task cost separately for three token types: uncached input, cache-hit input, and output.
The cost model can be written as follows:
Per-task cost = uncached input tokens × input price + cache-hit input tokens × cache price + output tokens × output price
3.2 Cost Estimation Code
def estimate_task_cost(input_tokens, cached_tokens, output_tokens,price_in, price_cached, price_out):"""Price unit: CNY per million tokensReturns: estimated cost of a single task in CNY"""uncached = input_tokens - cached_tokenstotal = (uncached * price_in+ cached_tokens * price_cached+ output_tokens * price_out)return total / 1_000_000
When comparing models, use the average token usage across the same task set, not individual samples. Pay close attention to differences in output length. Two models with the same unit price can have different actual costs if one produces much longer output.
3.3 Joint Evaluation of Cost and Quality
Comparing cost alone is not meaningful. You need to view cost and quality together. A common method is to calculate the cost per unit of quality, which means comparing per-task cost at the same pass rate. If a model has a lower unit price but needs more turns or retries to finish tasks, its actual cost may not be lower.
4. Interface Adaptation and Training-Inference Consistency
4.1 Adapter Layer Design
Services use different request and response formats. If business code handles these differences directly, switching models requires many changes. An adapter layer converts external formats into a unified internal format.
class ModelAdapter:def __init__(self, backend):self.backend = backenddef chat(self, messages, tools=None):payload = self._to_wire_format(messages, tools)raw = self.backend.send(payload)return self._normalize(raw)def _to_wire_format(self, messages, tools):# Assemble the request body in the format required by the target service...def _normalize(self, raw):# Normalize to three fields: text, tool_calls, and usage...
The adapter layer offers two main benefits. First, when you switch models or services, you only replace the adapter implementation. Second, the evaluation framework and business logic depend only on a unified structure, which makes side-by-side comparison easier.
4.2 Standardizing Tool Definitions
Agents rely on tool definitions to understand what they can do. A clear description and strict parameter constraints raise the chance that the model generates valid calls. The following is a general tool definition example:
{"name": "read_file","description": "Read the text content of a specified file within the project directory","parameters": {"type": "object","properties": {"path": {"type": "string","description": "File path relative to the project root directory"}},"required": ["path"]}}
The description should state the scope and limits of the tool. For example, “can only read files within the project directory” helps the model avoid out-of-scope requests during planning.
4.3 Training and Inference Consistency
Results are usually more stable when the dialogue templates, tool call formats, and end markers used at inference match those used during training. If the inference template differs from the training template, the model may produce format errors, omit tool parameters, or stop early.
For self-built systems, there are two recommendations. First, use the standard templates provided by the model provider, and do not modify them casually. Second, add a format compliance metric to your evaluation, such as the JSON parsing success rate of tool calls, so you can catch adaptation problems early.
5. Integration Essentials for Programming Scenarios
When you connect a model to a code assistant or automated toolchain, engineering security matters as much as capability and cost.
5.1 Permissions and Sandboxing
The more operations an agent can perform, the higher the risk. Follow the principle of least privilege and enable only the tools the task requires. Restrict file reads and writes to specified directories, and run commands in an isolated environment.
import_safe_root = "/workspace/project"def safe_resolve(path, root=import_safe_root):"""Resolve a path provided by the user or model into an absolute path within the root directory"""candidate = normalize_path(join_path(root, path))if not candidate.startswith(root + "/"):raise PermissionError("Path is outside the allowed scope")return candidate
Replace normalize_path and join_path with the normalization functions your platform provides, and make sure they handle cases such as symbolic links.
5.2 Secrets Management
Inject access credentials through environment variables or a secrets management service. Do not write them into code, prompts, or logs. If evaluation logs record complete requests, redact sensitive values first.
5.3 Human Confirmation
Irreversible operations, such as deleting files, pushing code, or publishing a service, should require human confirmation. The agent can propose a plan, but it needs explicit approval before executing it.
6. Selection Process and Implementation Recommendations
Based on the above, you can use the following process for model selection.
- Define the business scenario and key metrics, such as task pass rate, per-task cost, and average latency.
- Use public evaluations for initial screening, and record the versions and settings.
- Build an internal task set of at least several dozen items that covers both normal and exception workflows.
- Connect candidate models through the adapter layer, fix the inference parameters, and run them under the same conditions.
- Compile pass rate, tool call success rate, per-task cost, and manual ratings into a comparison table.
- Run a small-traffic canary rollout for a period to observe stability on real data.
During implementation, also keep the following points in mind.
- Community rumors and personal experiences can point you toward useful leads, but they are not conclusions. Verify them with reproducible data.
- After a model version update, existing conclusions may no longer hold. Rerun evaluations regularly.
- Cost optimization should not reduce quality on critical tasks, especially in scenarios involving security or funds.
- The evaluation framework needs ongoing maintenance, and the
