How to Choose Lightweight LLMs for Agent Tasks: An Evaluation Methodology
This article explains why fast-response models should be evaluated by task completion rather than answer speed alone in agentic settings. It covers three core capabilities—coding agents, tool calling, and long-task execution—and proposes five evaluation dimensions: task success rate, tool-call accuracy, step efficiency, recovery ability, and cost/latency. It also outlines how to build a reliable evaluation set, provides a minimal Python evaluation framework, and discusses how to interpret results, make model-selection decisions, and address engineering concerns such as permissions, sandboxing, timeouts, log sanitization, and regression testing.
1. Why Fast-Response Models Need a New Evaluation Lens
In early model selection, teams often focused on response speed and single-turn answer quality. As agent applications become more common, models must read context, call external tools, modify files, and verify results across multi-step tasks. These tasks impose different requirements from single-turn Q&A. A model that answers quickly may still fail the entire task if it passes incorrect parameters when calling a tool at step five. The evaluation focus should therefore shift from “how fast does it answer?” to “can it finish the task?”
Fast-response models usually have advantages in latency and cost, making them suitable for high-frequency invocation scenarios. Whether that value can be realized depends on their completion rate within specific task workflows.
2. Three Core Capabilities for Agent Scenarios
2.1 Coding Agents
A coding agent needs to understand repository structure, locate problematic files, generate patches, run tests, and iterate based on failure information. Key capabilities include:
- Cross-file localization: whether it can find the relevant module from an error stack or requirement description;
- Patch quality: whether changes are minimal and do not break existing interfaces;
- Use of test feedback: whether it can understand failure logs and correct its direction.
2.2 Tool Calling
Tool calling requires the model to output invocation requests according to agreed parameter structures and correctly interpret returned results. Common failure modes include incorrect parameter types, missing required fields, assuming a call succeeded when a tool returns an error, and repeatedly calling the same tool in a loop.
2.3 Long-Task Execution
Long tasks often involve a dozen or more operations. The model needs to maintain goal consistency, remember completed steps, and correct deviations in intermediate states in a timely manner. Context utilization efficiency and state summarization are especially important here.
3. Designing Evaluation Dimensions
A single composite score is rarely sufficient to guide decisions. We recommend splitting evaluation into the following five dimensions:
- Task success rate: whether the task ultimately meets acceptance criteria in an isolated environment;
- Tool-call accuracy: the correctness of call counts, parameter validity, and result interpretation;
- Step efficiency: the average number of steps required to complete a task, and the proportion of invalid operations;
- Recovery ability: whether the model can adjust autonomously and complete the task after errors or timeouts;
- Cost and latency: token consumption per task, end-to-end duration, and failure-retry overhead.
Task success rate is the final metric, while the other dimensions help explain why tasks succeed or fail.
4. Building an Evaluation Set
The quality of the evaluation set determines the reliability of the conclusions. The following principles can be followed when building one.
Task sources should be close to real business operations. Prefer real internal tickets, defects, and operations that have occurred within the team, with sensitive data removed.
Samples should be stratified. Divide tasks by difficulty into simple, medium, and complex tiers, and retain enough tasks in each tier to avoid conclusions being inflated only by simple tasks.
Acceptance criteria should be automatically determinable. For coding tasks, use unit test passes as the criterion; for tool-based tasks, use target-state checks. Avoid relying on subjective human scoring whenever possible; when human judgment is necessary, keep records that can be reviewed.
Environments must be isolated. Use a clean sandbox for each run and prohibit access to production data and real accounts.
Sample size should be sufficient. The success or failure of a single task is stochastic. Run the same task multiple times and report both the mean success rate and its variation range.
5. Example Evaluation Framework
The following code provides a minimal evaluation skeleton that demonstrates the basic structure for task execution, recording, and metric aggregation. The model client and sandbox need to be implemented by the team according to its own environment.
import timefrom dataclasses import dataclass, fieldfrom statistics import mean@dataclassclass TaskResult:task_id: strsuccess: boolsteps: inttool_errors: inttokens: intlatency_s: float@dataclassclass Report:results: list = field(default_factory=list)def summary(self):n = len(self.results)if n == 0:return {}total_steps = sum(r.steps for r in self.results)return {"success_rate": sum(r.success for r in self.results) / n,"avg_steps": mean(r.steps for r in self.results),"tool_error_rate": sum(r.tool_errors for r in self.results) / max(1, total_steps),"avg_tokens": mean(r.tokens for r in self.results),"avg_latency_s": mean(r.latency_s for r in self.results),}def run_task(model_client, sandbox, task, max_steps=30):"""Execute a single task in an isolated sandbox and return a TaskResult."""state = task.initial_state()steps, tool_errors, tokens = 0, 0, 0start = time.monotonic()while steps < max_steps:reply = model_client.next_action(state) # Return a tool call or final answertokens += reply.usage_tokenssteps += 1if reply.is_final:breakresult = sandbox.execute(reply.tool_name, reply.arguments)if not result.ok:tool_errors += 1state = state.append(reply, result)success = task.check(sandbox) # Judge based on the target statereturn TaskResult(task_id=task.id,success=success,steps=steps,tool_errors=tool_errors,tokens=tokens,latency_s=time.monotonic() - start,)
At runtime, execute each model against each task multiple times, aggregate the results into Report, and then output summary tables stratified by difficulty.
6. Interpreting Results and Making Selection Decisions
- Do not look only at averages. The success rate for the same task can vary widely across runs, so compare the variation range as well.
- Review results by stratum. If a model leads on simple tasks but has a higher failure rate on complex tasks, it may not be suitable for long-task scenarios.
- Convert cost into “cost per successful task.” Dividing total spend by the number of successful tasks reflects real overhead better than per-call unit price.
- Consider combined approaches. Route high-frequency, low-complexity steps to fast models, and key decisions and complex reasoning to stronger models. This tiered routing pattern is common in engineering and can control both cost and success rate.
7. Engineering Implementation Notes
- Least privilege: agents should only access the tools and data required to complete the task, and write operations should require human confirmation by default.
- Sandboxing and rollback: file modifications and external requests should be executed in isolated environments, with snapshots retained for rollback.
- Loops and timeouts: set maximum steps, per-step timeouts, and duplicate-call detection to prevent invalid loops from consuming resources.
- Log sanitization: record complete execution traces for retrospective analysis, but filter keys, personal information, and internal addresses.
- Continuous regression: after model version updates, tool interface changes, or prompt adjustments, rerun the evaluation set to avoid overlooking capability regressions.
