Can Plugins and Skill Packages Help AI Agents Continuously Improve?

    Plugins and skill packages can make AI agents more extensible and reusable, but they do not automatically make them self-improving. This article distinguishes models, agents, plugins, skills, and runtime frameworks, then uses a test-debugging workflow to show how controlled feedback, validation, versioning, and rollback can turn operational experience into safe, measurable improvements.

    1. Models, Agents, and Runtime Frameworks Have Different Responsibilities

    Give a language model a test log, and it can suggest possible causes and code changes. Give an AI agent access to tools, and it can also read project files, run approved tests, and adjust its next step based on the results. The difference is not simply the format of the response. An agent can obtain new information from its environment and change task state within an authorized scope.

    The model interprets the goal, forms judgments, and proposes actions. An agent is the software system that organizes those judgments and actions around a task: what to read first, how to respond to a failure, and when to stop are all part of task execution. The runtime framework provides the execution environment and constraints. It loads capabilities, checks permissions, invokes tools, returns results, and records the process. Even when a model proposes an operation, the runtime framework—not the model—should decide whether that operation is permitted and executable.

    This is also why an agent runtime framework should not be confused with a model evaluation framework. An evaluation framework organizes test cases, collects results, and scores them against defined criteria. A runtime framework makes the task happen. They can work together: the runtime executes the task, and the evaluation process determines whether the result was correct, whether the agent exceeded its authority, and whether the cost was acceptable. Being able to run a task and running it well are two different things.

    2. Plugins Provide Operations; Skills Provide Methods

    Plugins are well suited to clearly bounded operations that require actual execution, such as reading workspace files, running pre-approved tests, or generating a diff report. A plugin design should not merely tell the model what the tool does. It should also define which resources can be accessed, whether the operation can write data, how failures are reported, and whether an operation might continue after a timeout. For example, a file-reading plugin should enforce workspace boundaries on the execution side rather than rely on the model to always provide a compliant path.

    Skill packages address a different question: given a type of task, in what order should these capabilities be used, and what evidence is sufficient to declare the task complete? A skill for investigating test failures might define its applicable conditions, required inputs, diagnostic steps, acceptance evidence, and stopping conditions. It could require the agent to first confirm whether the failure is reproducible, then distinguish an environment problem from a behavioral change in the code. If a modification is needed, the agent should explain the basis for the proposed change. After validation, it should report which checks passed and which areas remain uncovered.

    A skill package can include instructions, templates, and examples. It can also reference existing plugins, but a reference is not an authorization.

    One way to decide where a capability belongs is to ask whether it directly changes external state:

    • “Run an approved test task” is a plugin operation.
    • “Run the smallest failing case first, then expand validation according to the impact scope” is a method defined by a skill.
    • “Do not access files outside the workspace” is a policy that the runtime framework must enforce.

    Mixing these layers commonly turns permission rules into mere prompt text, or forces teams to duplicate tool logic every time they add a new working method. Keeping them separate allows the same plugin to support multiple skills, while the same skill can be reused in different execution environments that satisfy equivalent constraints.

    The idea that “everything should be a plugin” is therefore better understood as integrating extension capabilities through explicit contracts, not as turning every internal function into a plugin. Fundamental controls such as permission checks, task termination, and audit logging should remain under the control of a trusted runtime layer. Excessively fine-grained plugins also increase the cost of capability selection, version compatibility, and troubleshooting. A better starting point is a small set of stable operations that can be tested independently.

    3. Validate the Architecture with a Test-Debugging Task

    Consider the task: “Identify why the order-amount rounding test is failing and propose the smallest fix.”

    The first integration should not allow the agent to modify code directly. A safer goal is to determine whether it can explain the issue within a read-only scope and provide evidence that another person can verify.

    Step 1: Prepare an isolated workspace

    Use a project copy without real business data or a dedicated sample project. Define the directories the agent may access, the test tasks it may run, the execution time limit, and the resource limits. If the model is accessed through a remote service, first determine which file contents may be transmitted, then configure credentials and the call budget.

    Credentials should not appear in the task instructions, project copy, or logs, and they should not be passed automatically to test subprocesses. An isolated directory only defines the working scope; it is not complete security isolation. Test scripts are executable code, so process-level file and network access must also be restricted separately.

    Step 2: Establish a comparable baseline

    Ask the agent to run the specified test and record the failing case, expected value, actual value, and any necessary execution conditions. If dependencies are missing or the test never started, the agent should report an environment problem rather than classify it as an implementation defect.

    If the failure cannot be reproduced consistently, preserve the observation and the uncertainty. Do not modify the implementation based on a single log. Any instructions found in files or logs should be treated only as data for analysis; they must not override the task’s authorization.

    Step 3: Separate diagnosis from writes

    A read-only deliverable can include the relevant code locations, an explanation of the rule difference, candidate changes, and a validation plan. If a fix must be tested, grant separate and limited write access within the workspace, and preserve the diff before and after the change.

    During acceptance, rerun the original failing case first, then check adjacent behavior affected by the change. Passing one test is not sufficient. A result is still unacceptable if the fix merely weakens an assertion or encodes an unconfirmed rounding rule into the implementation.

    These three steps also provide a clear troubleshooting order:

    • If the task never started, check the model connection and capability loading.
    • If the test did not execute, check the workspace, dependencies, and authorization.
    • If the test returned a result but the conclusion is unreliable, check whether the skill specified sufficient acceptance evidence.

    Not every failure should be attributed to insufficient model capability, and a permission error should not automatically lead to broader permissions.

    4. Extensibility Is Not Yet Continuous Improvement

    Adding plugins or installing skill packages expands the system through human-controlled changes. Adjusting a plan during execution based on test results is the agent completing its current task. A system gains limited continuous-improvement capability only when experience from one task is filtered, validated, and applied as a controlled change to later tasks.

    This process does not necessarily require updating model parameters. The change might affect skill steps, acceptance rules, or capability-selection strategies instead.

    For example, suppose repeated runs show that an agent often classifies intermittent failures as business regressions. The appropriate response is not to let the agent rewrite its own rules directly. Instead, it could propose a candidate improvement: when failure results are inconsistent, repeat the check within a defined budget and mark the reproducibility status in the report.

    That rule should then be tested against historical cases and independent validation cases. It should reduce false diagnoses without causing ordinary, consistently failing investigations to run unnecessarily many times. It must also avoid incorrectly reporting a genuine intermittent fault as “resolved.” A new version should be released only when both its benefits and its costs can be observed.

    A practical improvement loop contains four stages:

    1. Collect evidence. Record the skill version, key operations, failure category, and acceptance result associated with each task. Keep enough information to reconstruct the reasoning process, but do not place sensitive inputs or complete environment variables in the logs.

    2. Propose a candidate change. The agent may suggest changing a procedure, adding a check, or introducing new cases. It must not independently expand permissions, modify acceptance criteria, or turn a temporary observation into the team’s default rule.

    3. Run comparative validation. Evaluate the old and candidate versions on the same representative task set. Compare result correctness, the scope of unintended changes, execution cost, and unauthorized operations. Tool-call success rates show only whether an operation completed; they cannot replace evaluation of the final deliverable.

    4. Release and roll back by version. First deploy approved changes to a limited set of tasks, and record the version combination of skills and plugins. If quality declines, revert to a validated combination. Version rollback does not undo file writes or external actions that have already occurred. Those side effects require separate change review, backups, or compensating procedures.

    5. Stabilize One Task Before Expanding Autonomy

    When introducing this architecture, it is more effective to select one frequent, low-risk, and easily verifiable task than to build a large plugin catalog immediately. Start with read-only analysis. Confirm that the system can produce reliable evidence, then allow limited writes, and only afterward decide which recurring problems are worth formalizing as skills.

    Production changes, external notifications, and bulk data modifications should continue to require independent authorization and confirmation. A validated skill package should not automatically authorize such operations.

    Plugins make capabilities replaceable and composable. Skill packages make working methods reusable and evaluable. Together, they can provide a foundation for continuous improvement, but the ability of an agent to “evolve” does not depend on how many plugins it has or whether it can rewrite its own instructions.

    The decisive factor is whether experience can be converted into a verifiable change—and whether the system can stop and recover promptly when that change fails.