How to Select and Evaluate Agent Skills: An Engineering Approach

    Agent capabilities depend heavily on the skills they can call. This article presents a structured, engineering-driven method for selecting and evaluating agent skills, covering the five key dimensions of a skill, a reproducible evaluation workflow with scoring and canary release practices, common selection pitfalls, and a practical pre-launch checklist for engineering teams.

    Engineering Approaches to Selecting and Evaluating Agent Skills

    Why Skill Selection Determines Agent Success

    An agent’s capability boundary is largely defined by the skills it can invoke — tools, plugins, or functions. The model is responsible for understanding the task and planning steps, while skills handle execution: querying data, reading and writing files, calling internal APIs, or generating reports.

    In practice, a common problem is that the same task can yield dramatically different success rates when you swap in a different set of skills. The cause is usually not the model itself, but whether skill descriptions are clear, whether parameter contracts are reasonable, and whether failures are recoverable. Skill selection should therefore be treated as an independent engineering effort, not a matter of picking tools that “seem good” based on gut feeling.

    The goal is straightforward: choose a skill combination that is stable, secure, maintainable, and cost-controlled, while meeting the task requirements.

    The Anatomy of a Skill and Evaluation Dimensions

    Before evaluating a skill, you need to understand what it consists of. A skill generally has the following five aspects.

    1. Functional Description and Invocation Boundary

    The functional description is what the model relies on to decide whether to call a skill. It should state what the skill can and cannot do, and which input scenarios it applies to. A description that is too broad causes the model to invoke the skill in unrelated tasks; one that is too narrow causes the model to skip calls it should make.

    When evaluating, check three things: whether the description includes explicit applicability conditions, whether it lists typical non-applicable cases, and whether the example inputs cover the main business scenarios.

    2. Input and Output Contracts

    Input parameters should have clear types, value ranges, and required-field rules. The output structure should be as stable as possible — ideally with fixed fields — and should distinguish between three return types: successful results, business failures, and system errors. Structured output allows the upstream agent to parse results more reliably and makes later log analysis easier.

    3. Permission and Security Boundary

    A skill’s permission scope directly determines its risk level. Read-only skills carry the lowest risk, write skills raise the possibility of unintended data modification, and admin-level skills can affect the entire system configuration. Selection should follow the principle of least privilege: grant only the minimum access needed to complete the task, and keep a confirmation step or audit record for write operations.

    Also check whether the skill writes sensitive information to logs, whether it sends requests to external addresses, and whether it is vulnerable to instruction injection through crafted input.

    4. Stability and Performance

    Stability metrics include success rate, timeout rate, success rate after retries, and behavior under abnormal inputs. The main performance metrics are average latency and tail latency. For interactive scenarios, tail latency often affects user experience more than the average, so it deserves particular attention.

    5. Maintainability and Ecosystem

    Maintainability shows up in clear version management, complete change notes, and the ability to quickly diagnose problems when they occur. For skills used within a team, also consider documentation quality and whether there is a clearly assigned maintainer.

    Building a Reproducible Evaluation Process

    Subjective trial use only surfaces obvious problems; it cannot compare subtle differences between skills. Establish an evaluation process in four steps.

    Step 1: Define Task Scenarios and Success Criteria

    Start by listing the typical tasks the agent must complete, such as “look up shipping status by order number” or “convert meeting notes into a to-do list.” Write clear success criteria for each task, ideally expressed as verifiable rules — for example, whether all required fields are returned, whether values match the source of truth, and whether the task completes within a specified time.

    Step 2: Build an Evaluation Set

    An evaluation set is a collection of task samples with expected answers or judgment rules. The sample size doesn’t need to be large, but it should cover normal inputs, edge cases, missing parameters, malformed input, and malicious instructions. Keep the evaluation set as close as possible to the production data distribution, and regularly add newly discovered failure cases.

    Step 3: Score Quantitatively

    Combine metrics such as success rate, latency, permission risk, and auditability into a comparable score. Weights should reflect business priorities. For security-critical scenarios, increase the weight of the permission penalty; for real-time scenarios, increase the weight of latency.

    The following is a simplified scoring example to illustrate the approach. Fields and weights should be adjusted to your actual business.

    1. from dataclasses import dataclass
    2. @dataclass
    3. class SkillCandidate:
    4. name: str
    5. success_rate: float # Task success rate on the evaluation set, from 0 to 1
    6. avg_latency_s: float # Average latency in seconds
    7. permission_scope: str # "read" / "write" / "admin"
    8. has_audit_log: bool # Whether invocation audit logs are recorded
    9. PERMISSION_PENALTY = {"read": 0.0, "write": 0.1, "admin": 0.3}
    10. def score(c: SkillCandidate) -> float:
    11. # Latency score drops to zero when average latency exceeds 30 seconds
    12. latency_factor = max(0.0, 1 - c.avg_latency_s / 30)
    13. base = 0.7 * c.success_rate + 0.3 * latency_factor
    14. # Unknown permission levels are treated as highest risk
    15. base -= PERMISSION_PENALTY.get(c.permission_scope, 0.3)
    16. if not c.has_audit_log:
    17. base -= 0.05
    18. return round(base, 4)
    19. candidates = [
    20. SkillCandidate("order_query_v2", 0.92, 2.1, "read", True),
    21. SkillCandidate("order_update_v1", 0.95, 1.4, "write", False),
    22. ]
    23. for c in sorted(candidates, key=score, reverse=True):
    24. print(c.name, score(c))

    In this example, an unknown permission level is treated as the highest risk. This is a common practice in security design: when encountering an unrecognizable situation, default to the more conservative path.

    Step 4: Canary Release and Rollback

    Scoring results only justify inclusion in the candidate pool; a canary rollout is still required before full deployment. Route a small amount of traffic through the new skill while recording success rates, error types, and user feedback, and compare against the existing solution. If abnormal metrics appear, you should be able to switch back to the old version quickly. The rollback plan should be written and rehearsed before launch.

    Common Pitfalls

    Relying on demos alone. Demos typically use ideal inputs and do not reflect long-tail errors. Base decisions on statistical results from the evaluation set.

    Ignoring failure paths. Many skills only describe the success case, leaving timeouts, rate limits, and partial failures undefined. When the upstream agent encounters these situations, it can easily take incorrect follow-up actions.

    Granting excessive permissions up front. Granting admin-level access for convenience amplifies the blast radius of a single mistaken invocation. Permissions should be split by task and tightened or loosened as requirements change.

    Lacking version awareness. After a skill upgrade, its description or parameters may change, while dependent prompts and workflows are not updated in sync — leading to hard-to-diagnose failures. Record the version number and impact scope for every change.

    Overlooking cost. Invocation counts, wait times, and retry overhead all add to total cost. Include the end-to-end cost per task in your evaluation.

    Implementation Checklist

    Before introducing a new skill, verify the following items one by one:

    • The functional description includes applicability conditions and non-applicable cases.
    • Input parameters have defined types and value ranges, and the output structure is fixed.
    • Errors fall into three categories — business failure, parameter error, and system error — with explicit return codes.
    • The permission scope follows the principle of least privilege, and write operations have audit records.
    • Logs contain no secrets, personal data, or other sensitive information.
    • The evaluation set covers normal, edge-case, abnormal, and malicious inputs.
    • The composite score meets the preset threshold, and tail latency satisfies business requirements.
    • The canary release plan and rollback steps are documented and rehearsed.

    There is no once-and-for-all answer to skill selection. As business needs evolve and model capabilities improve, previous conclusions need periodic review. By institutionalizing the evaluation process and leaving data and records for every selection decision, a team can maintain consistent judgment in an environment where tools change frequently — and detect and resolve problems faster.