Mid-Sized Dense Models Agent Capability Evaluation: A Practical Guide
This guide explains how to evaluate the agent capabilities of mid-sized dense language models. It covers why general benchmark scores alone are not enough, four evaluation dimensions (planning stability, tool call accuracy, handling of environment feedback, and consistency over long task chains), a reusable evaluation loop with core metrics, common failure modes, cost per successful task as a selection metric, and considerations for production rollout.
1. Why Mid-Sized Dense Models Are Getting Attention Again
For a long time, teams evaluating language models assumed that more parameters meant stronger capabilities. As inference cost, GPU memory usage, and response latency became real operational constraints, more teams started to focus on capability per parameter, meaning how much usable capability a model reaches at a smaller scale.
A dense model activates all of its parameters during every forward pass. A Mixture-of-Experts (MoE) model activates only a subset of its expert subnetworks. This difference has three main engineering effects:
- Deployment threshold: Mid-sized dense models can usually run on a single GPU or a few GPUs, which keeps operational complexity low.
- Latency stability: The computation path is fixed, so latency for individual requests varies less. This makes it easier to set timeout and retry policies.
- Behavioral predictability: All parameters take part in every computation, so output style and failure patterns are easier to characterize across large numbers of samples.
Therefore, judging whether a mid-sized dense model is worth using cannot rely on single exam scores alone. Teams also need to see how the model performs across real task chains.
2. General Capability and Agent Capability Are Different Things
General capability evaluations typically cover static tasks such as knowledge question answering, mathematical reasoning, code generation, and scientific questions. The model gives an answer in one pass, so the evaluation measures what the model knows and what it can derive.
Agent capability focuses on how a model behaves in a closed loop. It includes:
- Breaking a vague goal into executable steps;
- Selecting the appropriate tool and filling in its arguments in the required format;
- Reading the tool’s output, determining whether the call succeeded or failed, and adjusting the next step;
- Keeping the goal and state consistent across a chain of a dozen or more steps.
The two capabilities are related but not equivalent. A model that performs well on static questions might use the wrong parameter name at the fifth tool call, or repeat the same action after receiving an error message. The table below summarizes the main differences between the two types of evaluation.
| Dimension | General Capability Evaluation | Agent Capability Evaluation |
|---|---|---|
| Input format | Single-turn question | Goal description plus callable tools |
| Output format | One answer | A sequence of actions and a final result |
| Key metrics | Accuracy, solve rate | Task completion rate, step efficiency, tool error rate |
| Main risks | Missing knowledge, reasoning errors | Planning drift, parameter errors, getting stuck in loops |
3. Four Dimensions of Agent Capability Evaluation
3.1 Task Planning Stability
Planning stability refers to whether a model produces plans with similar structure and consistent direction when it runs the same task multiple times. To evaluate it, run the same goal several times and observe the following:
- Whether the first step is reasonable, and whether the model gathers information before making changes;
- Whether the number of subtasks stays stable, without unnecessary extra or missing steps;
- Whether the model recognizes when a precondition is not met and adjusts its plan.
A typical sign of unstable planning is that, for the same task, the model queries first on one run, modifies directly on another, and skips verification on a third. Differences like these are hard to spot in a single demonstration. Repeated runs are needed to expose them.
3.2 Tool Call Accuracy
Tool calls are where agents most often make basic errors. The evaluation should check at least three types of problems:
- Structural errors: The output does not match the call format. For example, a required field is missing or the JSON cannot be parsed.
- Semantic errors: The parameter name is correct, but the value does not fit the task. For example, a time range is written as a single date.
- Selection errors: A suitable tool is available, but the model chooses an unrelated one.
Below is a simplified argument validation example. It illustrates the checking logic within an evaluation framework and does not represent any specific platform’s interface.
def validate_arguments(schema: dict, args: dict) -> tuple[bool, str]:"""Check whether tool call arguments meet the required field and type requirements (simplified example)."""for field, spec in schema["required"].items():if field not in args:return False, f"Missing required argument: {field}"if not isinstance(args[field], spec["type"]):return False, f"Incorrect argument type: {field}"return True, "ok"
3.3 Handling Environment Feedback
Real tools return many kinds of feedback, including successful results, empty results, timeouts, and permission errors. A capable agent should respond to each differently:
- On a permission error, stop retrying and ask the user for confirmation.
- On an empty result, check whether the query conditions were too narrow before deciding whether to loosen them.
- On a timeout, retry a limited number of times and record the reason for the failure.
Evaluation needs to construct abnormal scenarios on purpose and count how the model handles each one. Testing only the normal path cannot reveal how resilient the model really is.
3.4 Consistency Across Long Chains
The longer the task chain, the more likely the model is to lose track of its constraints. For example, suppose the user requires “do not modify the original configuration file,” but at step twelve the model overwrites that file. This kind of problem usually appears when the context is long and there are many intermediate results.
Evaluation methods include inserting several hard constraints at the start of a task and checking each one at the end. They also include recording a state summary at each step and observing whether the state drifts partway through.
4. Building a Reusable Evaluation Process
The core of an evaluation framework is a unified execution loop with quantifiable metrics. The example below shows the basic structure of one task run. All functions and fields are designed for illustration only.
from dataclasses import dataclass, field@dataclassclass Task:instruction: strtools: listenv: object # Simulated environment that executes tools and returns resultscheck: callable # Checks whether the final result meets the goaldef run_episode(model, task: Task, max_steps: int = 20) -> dict:history = [{"role": "user", "content": task.instruction}]stats = {"steps": 0, "tool_calls": 0, "tool_errors": 0, "completed": False}for _ in range(max_steps):action = model.next_action(history, tools=task.tools)stats["steps"] += 1if action.kind == "finish":stats["completed"] = task.check(action.content)breakstats["tool_calls"] += 1observation = task.env.execute(action.name, action.arguments)if observation.is_error:stats["tool_errors"] += 1history.append({"role": "tool", "content": observation.text})return stats
After running the full task set, you can calculate the following metrics:
- Task completion rate = Completed tasks / Total tasks
- Tool error rate = Tool errors / Tool calls
- Average steps per task = Total steps / Total tasks
Organize the task set by difficulty. It should include at least single-step tasks, multi-step tasks, and tasks with abnormal feedback. Run each category with a fixed seed or fixed samples, and repeat the runs to reduce the effect of random variation on the conclusions.
5. Common Failure Modes and Troubleshooting
Failure cases collected during evaluation are often more valuable than a single score. The following are common problem types:
- Parameter hallucination: The model invents parameters that a tool does not have, or uses IDs that never appeared in the context. To troubleshoot, validate arguments against an allowlist before execution.
- Wrong tool selection: Tools with similar descriptions are easy to confuse. Improve tool names and descriptions, and add contrasting examples.
- Looping calls: The same action is repeated. Record call history at the framework level and set a maximum number of repeats for identical actions.
- Premature completion: The model declares the task finished before completing all subtasks. Add an explicit completion check before the task can end.
- Constraint forgetting: After a long chain, the model ignores its initial constraints. Re-inject key constraints in every prompt turn.
Each type of problem should be tied to specific log excerpts that record the model input, output, tool return, and final judgment. This makes issues easier to reproduce and fix.
6. Selecting Models by Cost per Successful Task
When comparing models of different sizes, neither unit price nor parameter count tells the full story. A more useful metric is cost per successful task:
Cost per successful task = Total inference cost / Number of successfully completed tasks
This metric accounts for the price of each call, the retries caused by failures, and the number of steps per task. A model with a low unit price but a low completion rate can have a much higher cost per successful task than a more expensive model that completes tasks reliably.
Suppose that in one evaluation, two models each run 100 tasks. Model A has a total cost of 10 and completes 70 tasks. Model B has a total cost of 18 and completes 92 tasks. Model A’s cost per successful task is about 0.14, and Model B’s is about 0.20. In this hypothetical example, the cheaper model is not necessarily the better value. In practice, selection should be based on calculations using real business tasks.
7. Recommendations for Production Rollout
Before connecting an agent to a production environment, follow these steps:
- Shadow running: The model runs tasks in the background and only records results without submitting actual changes. This lets you observe its behavior over a period of time.
- Tiered permissions: Read operations can run automatically. Irreversible operations such as writing, deleting, or sending must go through manual confirmation.
- Guardrails: Limit the maximum number of steps, the timeout for each call, and the resources the agent can access.
- Continuous evaluation: Whenever the model version, prompt template, or tool definition changes, rerun the evaluation task set.
Agent capability evaluation has no one-time answer. Teams need to turn evaluation into an ongoing engineering process. Reproducible tasks, quantifiable metrics, and traceable logs help them judge whether a small-scale dense model is reliable enough for a specific scenario.
