How to Choose a Lightweight LLM: A Practical Framework for Evaluating Task Execution
A practical guide to evaluating lightweight language models for agent-based work. It explains how to assess task decomposition, tool usage, code changes, testing loops, long-task reliability, and recovery from errors, then provides a lightweight internal benchmark design and a Python evaluation framework.
1. Conversational Fluency Does Not Equal Task Execution
When evaluating a large language model, many people first look at whether its responses are fluent and well phrased. These metrics are useful for question answering and writing, but they are limited when judging AI agents.
Agent-based tasks require a model to break down a goal, select tools, perform actions, observe results, and adjust its approach based on feedback until the task is complete.
The focus should therefore shift from “How well does it answer?” to “Can it finish the job?” A model that produces polished answers may stop after the first step, providing a code snippet without verifying it. A less polished model may inspect the project structure, modify the necessary files, and run the tests successfully. The difference may be small in everyday conversation, but it is significant in task execution.
2. Four Core Capabilities for Agent-Based Tasks
1. Requirement Understanding and Task Decomposition
A model often receives a requirement in a single sentence, such as “Refactor the login module and add tests.” To complete it, the model must convert the goal into executable steps: locate the relevant files, understand the existing interfaces, identify behavior that must not change, and determine the order of modifications.
The quality of this decomposition directly affects every subsequent step.
How to evaluate it: Check whether the model creates a plan before taking action, whether the plan includes checkpoints, and whether it accounts for existing tests and dependencies between modules.
2. Accurate Tool Use
Agents interact with external environments through tools such as file readers, code search, command execution, and API calls. Tool-use failures generally appear in three forms:
- Incorrect parameter formats
- Poor timing, such as modifying a file before reading it
- Misinterpreting returned results
During evaluation, record the number of tool calls, failed calls, and retries for each task. The ability to adjust parameters based on an error message can reveal more about a model’s capability than getting every call correct on the first attempt.
3. The Code Modification and Testing Loop
For programming tasks, the key capability is completing a loop:
- Modify the code
- Run the relevant commands or tests
- Inspect the results
- Make further changes if necessary
Generating code without running it leaves the verification work to the user.
A reliable loop should meet the following conditions:
- The scope of changes matches the requirement and contains no unrelated modifications.
- The test commands are actually executed.
- Failure messages are read and used to guide the next round of changes.
- The final tests pass without deleting or weakening existing assertions.
The last condition is particularly important. Some models may make tests pass by deleting failing cases. Results of this kind require manual review.
4. State Retention and Error Recovery in Long Tasks
As a task grows, the context continues to expand. The model must remember completed steps, discovered issues, and pending work. A large context window is only a basic requirement. More important is whether the model can stay focused on the goal throughout a long interaction, avoid drifting, and refrain from repeating work that has already been completed.
To evaluate this capability, design a task that requires more than 20 steps and record the point at which the model begins to repeat actions or deviate from the objective.
3. Build an Internal Evaluation Set
Public leaderboards can provide useful reference points, but their task distributions may not match your business needs. A more reliable approach is to build a small internal evaluation set. Twenty to thirty tasks are usually enough to cover the most common types of work in your environment.
Task Selection Principles
- Use real sources: Select tasks from actual tickets or requirements handled during the previous quarter rather than creating idealized examples.
- Use difficulty levels: Include a mix of simple, intermediate, and complex tasks to identify capability boundaries.
- Make results verifiable: Each task should have clear acceptance criteria, such as passing tests, specific file contents, or an expected API response.
- Isolate the environment: Run every task in a clean copy so that results from one run do not affect another.
Evaluation Dimensions
| Dimension | Evaluation question | Recording method |
|---|---|---|
| Task completion | Are all acceptance criteria satisfied? | Pass / fail |
| Step efficiency | How many tool calls are required? | Numeric value |
| Error recovery | Can the model correct itself after a failure? | Number of recoveries |
| Change scope | Are there unrelated modifications? | Number of affected files |
| Test integrity | Were existing assertions deleted or weakened? | Yes / no |
Completion should be the highest-priority metric. A model that requires many steps but eventually completes the task is generally more useful than one that uses few steps but gives up halfway, unless cost is a strict constraint.
4. Example: An Evaluation Execution Framework
The following code demonstrates a simplified evaluation workflow. It separates task definitions, execution, and acceptance checks. The acceptance command is fixed by configuration rather than generated by the model, preventing the model from changing the evaluation criteria.
import subprocessfrom dataclasses import dataclass, field@dataclassclass TaskCase:name: strprompt: strworkdir: stracceptance_cmd: list[str] # Fixed by configuration; model output is not acceptedmax_tool_calls: int = 40@dataclassclass RunReport:name: strpassed: booltool_calls: intnotes: list[str] = field(default_factory=list)def check_acceptance(case: TaskCase) -> bool:"""Pass the command as a list without using a shell to avoid command injection."""proc = subprocess.run(case.acceptance_cmd,cwd=case.workdir,capture_output=True,text=True,timeout=600,)return proc.returncode == 0def evaluate(case: TaskCase, agent_run) -> RunReport:"""agent_run is the execution function for the model under test."""calls = agent_run(case.prompt, case.workdir, case.max_tool_calls)passed = check_acceptance(case)report = RunReport(name=case.name, passed=passed, tool_calls=calls)if calls >= case.max_tool_calls and not passed:report.notes.append("Call limit reached without passing")return report
The important point is the structure. The acceptance logic is independent of the model being evaluated, so every run produces comparable, structured results.
In practice, you can connect agent_run to the agent framework you use and write the results to a table for side-by-side comparison.
Acceptance commands should not contain unchecked user input or model output. If a task involves external addresses or credentials, read them from environment variables or a secrets management service. Do not place them in source code or logs.
5. Interpreting Results and Choosing a Model
After the evaluation, do not look only at the overall pass rate. Analyze the results from several angles:
- Break down pass rates by difficulty. If a model passes all simple tasks but fails most complex ones, it may be suitable for well-defined subtasks, while task decomposition still needs to be handled by a human.
- Analyze failure types. Requirement misunderstandings, tool-call errors, and defects that remain after tests pass each require different improvements.
- Consider cost and latency. When pass rates are similar, a solution that uses fewer tool calls and responds faster is usually more economical for batch workloads.
- Measure stability. Run the same task three to five times and compare the results. A model that succeeds only occasionally is not suitable for an automated workflow.
Model selection should be based on comparisons within a specific scenario rather than on a single overall ranking. Frontend prototyping, data scripting, document generation, and code review require different capabilities, so the best model may differ across use cases.
6. Common Mistakes
- Treating leaderboard scores as business conclusions: Leaderboard tasks may not match your codebase, toolchain, or coding standards.
- Evaluating only the first response: The value of an agent lies in repeated correction. Looking only at the initial output underestimates its capabilities.
- Ignoring permission boundaries: When an agent can execute commands, restrict the directories it can access, prohibit high-risk operations, and retain a human approval step.
- Treating passing tests as proof of completion: Also check whether tests were modified and whether the changes exceed the scope of the requirement.
7. Practical Adoption Advice
Start with a small set of repetitive tasks, such as adding unit tests to an existing module. Record the outcome of every run and the amount of manual correction required. After collecting two or three weeks of data, decide whether to expand the model’s use.
