Agent Model Selection: An Engineering Evaluation Across Five Task Categories
Selecting a model for an AI agent is not a matter of ranking single-turn answers. This article presents a reproducible evaluation method based on controlled variables, execution traces, task success, tool use, safety, cost, stability, and maintainability. It covers five representative workflows—skill packaging, web development, presentation creation, knowledge base management, and browser automation—along with failure patterns, cost governance, scoring criteria, and a layered architecture for production-ready agent evaluation.
Agent Model Selection: An Engineering Evaluation Across Five Task Categories
1. Why Agent Model Selection Is Not a Single-Turn Ranking
Once a large language model is connected to an agent framework, its capabilities can be amplified or constrained by memory, skill packages, tool gateways, system prompts, permission policies, and the execution environment.
Two models that perform similarly in single-turn conversations may behave very differently during long-running tasks. One may frequently select the wrong tool, forget constraints after several execution steps, skip risk confirmation before a critical write operation, or produce a usable result at a rapidly increasing cost because of repeated context reads, ineffective retries, and unnecessarily verbose tool calls.
Model selection for agent scenarios should therefore not focus only on which model produces the most polished answer. The more important question is which model can complete a task consistently under a defined scaffold, permission set, and budget.
A relatively rigorous approach is to control the variables. Build multiple identical agent instances, keep the system prompt, memory snapshot, skill registry, tool list, timeout policy, retry policy, and security rules unchanged, and replace only the underlying model. Then run a task suite that represents real workflows multiple times and record the complete execution traces.
A simplified process looks like this:
for model in candidate_models:agent = build_agent(system_prompt = shared_system_prompt,memory_snapshot = shared_memory,skill_registry = shared_skills,tool_gateway = shared_tools,safety_policy = shared_safety_policy,budget_policy = shared_budget_policy)traces = run_tasks(agent, task_suite, repeats = 3)report = score(traces)
For every request, the evaluation should record an input summary, tool selection, parameters, returned results, retry count, error type, points of human intervention, final artifacts, context consumption, generation consumption, and end-to-end latency.
For safety-related behavior, also record whether the agent requested confirmation, followed the principle of least privilege, and accessed resources outside the task scope.
This article describes general evaluation methods and engineering practices. It does not evaluate or recommend any particular model or platform. The reusable value lies in a selection method that is reproducible, observable, and suitable for regression testing.
2. Evaluation Metrics: Breaking “Usability” into Measurable Dimensions
The overall performance of an agent model should be divided into multiple dimensions:
| Dimension | Key question | Primary evidence |
|---|---|---|
| Task success rate | Does the final artifact meet the acceptance criteria? | Runnable artifacts, documents, test results, and human acceptance records |
| Instruction following | Does the model follow formatting requirements, boundaries, prohibitions, and output conventions? | Artifact inspection, rule-hit rates, and records of out-of-scope behavior |
| Tool-use accuracy | Does it select the correct tool, provide valid parameters, and use a reasonable call sequence? | Tool traces, parameter validation results, and failure reasons |
| Error recovery | Can it correct itself after a tool failure or abnormal intermediate result? | Retry strategies, fallback paths, and the number of human interventions |
| Memory and context consistency | Can it retain goals, constraints, and previous conclusions during a long task? | Multi-turn consistency checks and cross-stage reference accuracy |
| Safety and compliance | Does it confirm high-risk actions, restrict permissions, and isolate sensitive data? | Policy-engine logs, confirmation records, and permission audits |
| Cost efficiency | How much context, generation, and retry overhead is required for one successful task? | Total cost per successful task, cost variance, and budget violations |
| Stability and latency | Is the model reliable in long sessions, and does it meet business response-time requirements? | Timeout rate, retry rate, and continuous-run performance |
| Maintainability | Are the resulting artifacts clear, readable, testable, and reversible? | Code structure, skill-package structure, and documentation completeness |
| Observability | Can failures be attributed to the model, tools, permissions, or context? | Logs, traces, artifact snapshots, and metric dashboards |
“Cost per successful task” deserves particular attention. It is not simply the model price multiplied by the number of calls. It should also include failed retries, repeated context injection, replayed tool results, repeated reading of long documents, ineffective exploration, and human remediation.
A model with a low average response cost but a high rework rate may have a higher real-world cost.
3. Five Task Categories for Realistic Agent Evaluation
To prevent evaluation from becoming a collection of mechanical question-answering tests, tasks can be designed as a sequence of challenges. Each stage should produce an inspectable artifact and preserve the execution trace.
Task 1: Skill Packaging—from Experience Documents to Installable Capabilities
The first task asks the model to read a relatively long design prompt or operating specification and turn it into a publishable, installable, and reusable skill package.
The goal is not to summarize the source material. It is to convert implicit experience into explicit capability by defining:
- Trigger conditions
- Input parameters
- Execution steps
- Required tools
- Output formats
- Boundary conditions
- Failure handling
- Self-check procedures
A qualified skill package should include:
- A clear purpose statement and applicable scenarios;
- Structured parameter definitions and default values;
- Mandatory constraints and prohibited actions;
- Normal examples, edge cases, and counterexamples;
- Required resources and permissions;
- Output acceptance criteria and a self-checklist;
- A version number, change log, and rollback strategy;
- Known failure modes and recommended fixes.
Acceptance should focus on four questions:
- Can another user operate the skill independently?
- Does the model trigger it in the correct scenarios?
- Are high-risk operations restricted by default?
- Can the self-check detect common errors?
Common risks include a skill not being recognized by the registry, an overly broad trigger description causing misuse, missing parameter validation, insufficient examples, and a one-time prompt being incorrectly packaged as a general-purpose capability.
Engineering controls should not rely solely on the model’s judgment. Add registry validation, parameter-schema checks, least-privilege permissions, and an external policy engine.
Task 2: Web Development—from Requirements to a Runnable Artifact
The second task asks the model to produce a runnable web application or page module from a set of requirements.
Evaluation should go beyond whether the model can generate code. It should also examine requirement decomposition, component structure, state handling, error states, accessibility, input validation, dependency management, and maintainability.
Recommended acceptance checks include:
- Can the page run as specified?
- Are key interactions complete?
- Are empty, error, and loading states handled?
- Does the code contain hard-coded sensitive information?
- Are dependencies clear and is the build process reproducible?
- Is the code easy to locate and modify?
- Are basic tests or verification steps provided?
During development tasks, agents can easily enter a loop of “generate, encounter an error, and rewrite a large section.” The environment should allow the model to read necessary context while requiring incremental changes and limiting ineffective retries. Save snapshots at key points.
High-risk writes—such as modifying production configuration, deleting files, or deploying to production—must be intercepted by an external policy and require human confirmation.
Task 3: Presentation Design and Copywriting
The third task asks the model to design a presentation structure, write slide copy, and organize the content.
Evaluation should focus on whether the topic remains focused, the argument is logically structured, data and conclusions are consistent, copy is concise, the visual hierarchy is clear, and the material fits the target audience.
Possible checks include:
- Does the outline cover the core arguments?
- Does each slide carry one primary message?
- Are there unsupported absolute claims?
- Are chart titles, units, and definitions consistent?
- Has long-form text been compressed into readable points?
- Are speaker notes or a script provided to support delivery?
- Are there risks of overflow, overlap, or garbled text after export?
This task is useful for observing planning and self-review capabilities. High-quality output requires more than content generation. The model should also identify uncertain information and mark data that requires human confirmation.
Brand guidelines, fonts, colors, and templates should be provided through structured configuration rather than scattered across a long prompt.
Task 4: Knowledge Base Management
The fourth task asks the model to participate in knowledge base ingestion, cleaning, chunking, labeling, retrieval, citation, updating, and deletion.
The difficult part is not question answering. It is knowledge lifecycle management:
- How should documents be chunked?
- How should metadata be designed?
- How should permissions be inherited?
- How should outdated content be retired?
- How should conflicting information be resolved?
- How should answers provide traceable citations?
Acceptance can focus on whether:
- Answers are supported by retrievable evidence;
- Citations point precisely to a paragraph or record;
- The model clearly states when the knowledge base is insufficient;
- Document permissions and field-level restrictions are respected;
- Old content is no longer retrieved after updates or deletion;
- Conflicting versions follow a defined priority rule;
- Retrieved results are not excessive, fragmented, or outdated.
In production, permission checks should be handled by the retrieval service or policy layer rather than left to the model. The model should interpret the question and organize the answer, while the retrieval system performs permission filtering, version filtering, and citation injection.
This reduces unauthorized-access risk and improves auditability.
Task 5: Browser Automation
The fifth task is browser automation. It is often highly context-intensive because page structures, screenshots, interactive-element lists, operation logs, and error stacks consume substantial context.
The model must identify target elements on dynamic pages, plan an action sequence, wait for asynchronous states, and recover after failures.
Major risks include:
- Clicking high-risk buttons such as Submit, Pay, Delete, or Authorize;
- Relying on fragile selectors that fail after minor page changes;
- Repeating an operation because of uncontrolled retries;
- Consuming excessive context while observing long pages;
- Failing to isolate login sessions, credentials, and sensitive information;
- Lacking rollback or human takeover after an exception.
An “external constraints first” design is recommended:
- Maintain an allowlist of target websites;
- Classify actions by risk level;
- Require secondary confirmation for high-risk actions;
- Use idempotency keys;
- Save state snapshots before and after operations;
- Set timeouts and maximum step limits;
- Maintain observable logs;
- Provide a human takeover mechanism.
The model can handle planning and decision-making, but final execution must pass through a tool gateway and policy engine.
4. Common Failure Modes and Engineering Responses
1. A Skill Package Is Not Recognized Correctly
This does not necessarily mean that the model lacks the required capability. The problem may be a mismatch between the skill description and the registry mechanism, an ineffective skill retrieval method, or an overly long context that dilutes important information.
Recommended responses include adding regression tests for skill names and trigger words, performing structured matching before invocation, and preserving diagnostic logs that explain why a skill was not called.
2. High-Risk Actions Lack Confirmation
Confirmation mechanisms should not depend entirely on the model.
The tool gateway should classify actions into levels such as read-only, low-risk write, high-risk write, and irreversible action. High-risk actions must be evaluated by an external policy engine and, when necessary, require human confirmation.
A model response claiming “confirmed” cannot replace actual authorization.
3. Abnormal Context Consumption
Track input consumption, generation consumption, retry consumption, and the proportion of tool output returned to the model.
Common sources of waste include repeatedly reading the same file, injecting the full page history into every step, retrying without limits, and placing large raw logs directly into the context.
Costs can be reduced through summary compression, incremental reads, result caching, subtask isolation, and budget limits.
4. Insufficient Long-Task Stability
Long-running tasks should use checkpoints that record:
- Stage objectives;
- Completed work;
- Open questions;
- The location of the current artifact.
After each stage, the model should write structured state rather than relying on an extremely long conversation history. If the session is interrupted, execution can then resume from the latest checkpoint.
5. Uncontrolled Quotas and Budgets
Evaluation and production environments should manage budgets separately.
Set soft and hard thresholds for every task. When a soft threshold is approached, issue an alert. When a hard threshold is exceeded, pause execution and request a human decision.
For long-running tasks, also limit concurrency and the maximum number of steps per task. Preserve failed samples for regression testing.
5. Cost and Quota Governance: Turning Model Selection into Routing
Production agent systems generally should not depend on a single model.
A more robust approach is model routing:
- Use a low-cost path for simple classification, format conversion, and short summaries;
- Use a more capable path for complex planning, code repair, and cross-document reasoning;
- Do not reduce the confirmation level of high-risk tasks merely to optimize cost.
A simplified policy can be expressed as follows:
policy:default task budget: threshold_Aif estimated risk level is high:require human confirmationif retry count exceeds threshold:switch to a fallback model or pauseif the task is read-only and structurally simple:use the low-cost pathif the task involves a long multi-tool workflow:enable staged checkpoints and cost alertsif repeated failures bring the budget close to its limit:stop automatic execution and hand off to a human
The recommended metric is cost per successful task:
total task cost =input context consumption+ generated content consumption+ replayed tool results+ retry overheadcost per successful task =total task cost / task success rate
Cost variance should also be recorded. Some models appear inexpensive per request but have high variance and occasionally produce unusually long execution chains, which can significantly increase total spending.
For agents that must remain online for extended periods, stability, failure recovery, and observability are often more important than the nominal price of an individual call.
6. Scorecards and Acceptance Gates
A weighted score can be used for overall evaluation, but the weights should reflect business priorities:
overall score =0.30 × task quality+ 0.20 × stability+ 0.20 × safety and compliance+ 0.15 × cost per successful task+ 0.15 × maintainability and observability
These weights are only examples, not industry standards. Before production launch, define hard gates. For example:
- A serious permission violation or an unconfirmed high-risk action is an automatic rejection;
- A task success rate below the business threshold excludes the model from consideration;
- A sustained cost per successful task above budget prevents full-scale deployment.
Run at least three rounds of repeated evaluation and reserve a set of holdout tasks.
During human review, blind-evaluate the artifacts without revealing the model source in advance. This helps reduce subjective bias.
For every failure sample, label the root cause. Possible causes include:
- Model misunderstanding;
- Unclear skill description;
- Poor tool interface design;
- Overly restrictive permission policy;
- Information lost during context compression;
- Ambiguity in the task itself.
7. Recommended Deployment Architecture
A reusable agent evaluation and runtime architecture can be divided into seven layers:
- Model adaptation layer: Unified interfaces, fixed versions, timeout policies, and retry strategies;
- Skill registry layer: Structured manifests, parameter validation, version management, and rollback;
- Memory and state layer: Task state, checkpoints, historical summaries, and isolated storage;
- Tool gateway layer: Parameter validation, permission control, action classification, and confirmation mechanisms;
- Policy and security layer: Least privilege, sensitive-information isolation, and interception of risky actions;
- Evaluation and observability layer: Task suites, scorecards, trace tracking, and cost dashboards;
- Artifact storage layer: Code, documents, presentations, reports, and log snapshots.
At the infrastructure level, general capabilities such as object storage, logging, monitoring, and alerting from Baidu AI Cloud can provide unified support for evaluation artifacts, runtime logs, and cost metrics.
The key objective is to make every model call traceable, every failure attributable, and every version change testable through regression evaluation.
8. Conclusion
Model selection for AI agents is fundamentally a systems engineering problem.
Controlling variables makes evaluation fair, but real-world operation also requires safety policies, tool governance, cost controls, observability, and human takeover mechanisms.
Instead of repeatedly asking which model is the strongest, teams should ask a more useful question:
For which tasks, permissions, budgets, and stability requirements is a particular model the best fit for the current agent architecture?
Reproducible benchmarks, explicit acceptance gates, externalized safety constraints, and continuous regression testing are the foundation for moving from experimentation to production use.
Models will evolve and prices will change. A reliable evaluation methodology and strong engineering guardrails help teams maintain sound judgment as the surrounding technology changes.
