Plugin-based Agent Frameworks: From Model-Centric to Capability Orchestration

In traditional AI applications, the model often occupies the absolute center: developers select a model and then build around it with tool calling, context management, and task execution logic. As application complexity increases, this design exposes critical issues—model upgrades cannot automatically resolve engineering problems like permission control, task orchestration, environment isolation, failure retry, and state persistence.

A more universal approach is to split the agent into two layers:

Agent = Model Capability + Harness Runtime Framework

Here, the model is responsible for understanding, reasoning, and generating decisions. The Harness transforms these decisions into executable, traceable, and recoverable task workflows. Crucially, the Harness should not be a monolith; tools, planners, schedulers, sessions, storage, sandboxes, and execution loops should all be abstracted into replaceable plugins.

The core philosophy of this architecture is: all capabilities should have clear boundaries and evolve independently.

1. Why You Shouldn’t Build Agents Around Models Alone

In simple Q&A scenarios, the model inputs a prompt, outputs text, and the system closes the loop. However, in real-world scenarios like development, DevOps, data analysis, or business automation, agents must perform continuous actions:

1. Understand user goals;
2. Break down goals into subtasks;
3. Read files or query data;
4. Call external tools;
5. Revise plans based on tool results;
6. Execute operations in isolated environments;
7. Save intermediate states;
8. Handle exceptions and retry;
9. Return explainable results to the user.

The model only handles part of this. Writing all logic into a massive loop might work for quick prototyping, but leads to obvious issues later.

First, difficulty in replacement. Changing models might affect prompts, tool formats, context length, and error handling, requiring systemic modifications.

Second, testing difficulty. When planning, tool calling, retries, and state recovery are tangled together, it is hard to verify individual modules.

Third, blurred security boundaries. If model-generated instructions directly enter the OS or production environment, they risk unauthorized access, data leaks, and operational errors.

Finally, high evolution cost. As tools increase, the system forms numerous conditional branches, eventually becoming an unmaintainable “Giant Agent Loop.”

Therefore, agent engineering should focus on establishing a stable runtime architecture, not just improving model reasoning.

2. What Are the Responsibilities of a Harness?

The Harness serves as the runtime infrastructure for the agent. It is neither a specific model nor limited to coding scenarios; it organizes the collaboration between models, tools, and execution environments.

A comprehensive Harness typically includes the following modules.

2.1 Model Adaptation Layer

This layer masks differences between model APIs, providing a unified calling method. It typically handles:

• Text generation and structured output;
• Streaming responses;
• Tool calling requests;
• Context concatenation;
• Timeouts and retries;
• Token or quota control;
• Model response validation.

The value here is that business logic is not bound to a specific model service. Developers can choose different models based on cost, latency, context capacity, and task type.

2.2 Tool Plugin Layer

Tools are the primary entry points for agents to connect with real systems, such as file I/O, database queries, code execution, knowledge retrieval, API calls, and task submission.

A qualified tool plugin must at least define:

• Tool name;
• Function description;
• Input parameter schema;
• Parameter validation rules;
• Permission requirements;
• Execution result format;
• Error types;
• Timeout and cancellation mechanisms.

Tool descriptions must be accurate. The model needs to know what the tool can and cannot do, and when to use it. Tool returns shouldn’t be unparseable strings; they should include status, summary, detailed results, and error information.

For example, an abstract file reading plugin might use the following structure:

{
  "name": "read_file",
  "description": "Read text files within the specified workspace",
  "input_schema": {
    "type": "object",
    "properties": {
      "path": {
        "type": "string",
        "description": "Relative path within the workspace"
      }
    },
    "required": ["path"]
  }
}

The key is not the specific format, but making tool capabilities, input constraints, and security boundaries explicit.

2.3 Task Planning Plugin

The planner converts user goals into executable steps. Simple tasks can use one-time planning, while complex tasks require dynamic planning—adjusting subsequent actions based on intermediate results.

The planner usually resolves three issues:

• Can the task be decomposed?
• Are there dependencies between subtasks?
• Which steps can run in parallel?

For example, “analyze a project and fix test failures” might be broken down into:

1. Scan project structure;
2. Identify test entry points;
3. Execute tests;
4. Summarize failure causes;
5. Modify relevant files;
6. Re-execute tests;
7. Generate change summary.

The planner shouldn’t just output natural language plans; it should output structured task objects for scheduling and auditing.

2.4 Scheduling Plugin

The scheduler decides “what to execute and when.” It handles task dependencies, priorities, concurrency, resource limits, and failure retries.

In real systems, the scheduler must also consider idempotency. A repeated data write task might create duplicate records; a retried publish task might trigger multiple changes. Therefore, each task should have a unique identifier and explicitly state whether retries are supported.

2.5 Sandbox Plugin

Any agent with execution capabilities should run in an isolated environment. A sandbox can restrict:

• File access scope;
• Network access scope;
• Process permissions;
• CPU and memory usage;
• Execution duration;
• Callable system capabilities.

A sandbox is not merely “changing a directory to execute a command.” A more robust approach combines containers, virtualization, or restricted runtimes with whitelists, resource quotas, and audit logs.

2.6 Session and Storage Plugins

The session plugin saves the current task’s context, including user input, model decisions, tool calls, execution results, and exceptions.

The storage plugin persists states to local files, relational databases, key-value stores, or object storage. Separating them allows different implementations based on task scale:

• Local debugging uses lightweight storage;
• Single-machine applications use embedded databases;
• Multi-instance deployments use centralized databases;
• Large-scale tasks use object storage for logs and intermediate artifacts.

3. What Does “Everything is a Plugin” Mean?

Pluggability isn’t just splitting code into multiple files; it requires every module to have an independent contract.

A plugin should meet at least these conditions:

1. Clear interfaces: Inputs, outputs, and exception behaviors are predictable;
2. Controlled dependencies: Does not arbitrarily access other modules’ internal states;
3. Explicit lifecycle: Knows when to initialize, run, pause, and destroy;
4. Independently testable: Can be verified without the full system;
5. Observable: Can log call counts, durations, failure reasons, and resource consumption;
6. Replaceable: Changing implementation does not require modifying upper-level business logic.

For example, a planning plugin could have rule-based, model-based, and human-confirmation implementations. A tool execution plugin could be local, containerized, or remote. As long as they follow unified interfaces, upper-level scheduling logic remains unchanged.

This design brings three main benefits.

Reduced Coupling

Models, tools, and execution environments are independent; upgrading one part doesn’t require rewriting the whole system.

Improved Reusability

The same set of tool plugins can serve multiple agents for coding, data processing, and DevOps automation.

Easier Troubleshooting

When a task fails, it’s easy to determine if the issue comes from model output, parameter validation, permission checks, tool execution, or external dependencies.

4. Complete Execution Flow of a Task

A typical agent task runs as follows:

Step 1: Establish Session

The system creates a unique session for the task, recording the user goal, identity info, permission scope, and resource limits.

Step 2: Generate Initial Plan

The model generates a plan based on the goal and available tools. The system must structurally validate the plan and not blindly trust model output.

Step 3: Schedule Tasks

The scheduler analyzes task dependencies and selects executable steps. High-risk operations should enter a human-confirmation state first.

Step 4: Execute Tools

Tool plugins run under permission and sandbox constraints, returning structured results.

Step 5: Update Context

The system writes tool results, execution status, and error info back to the session. The model then determines the next action.

Step 6: Handle Exceptions

If a tool fails, the system should differentiate between parameter errors, permission errors, insufficient resources, external service anomalies, and unrecoverable errors, rather than retrying uniformly.

Step 7: Complete Task

Upon completion, the system outputs a result summary, execution process, generated files or changes, and retains full audit logs.

This loop can be represented as:

User Goal
   ↓
Session Initialization
   ↓
Task Planning
   ↓
Permission Validation
   ↓
Task Scheduling
   ↓
Tool Execution
   ↓
Result Write-back
   ↓
Continue Planning or End

5. Key Practices in Plugin Development

5.1 Define Contracts Before Implementation

Do not write massive business code first and patch interfaces later. Define inputs, outputs, errors, and permission models upfront.

5.2 Ensure Explainable Tool Results

Tool returns should include machine-readable fields and user-facing summaries. For example:

{
  "status": "failed",
  "error_type": "permission_denied",
  "message": "The current session lacks permission to access the target resource",
  "retryable": false
}

This allows the model and scheduler to decide whether to retry, switch tools, or request human intervention.

5.3 Distinguish Retryable and Non-Retryable Errors

Network timeouts and temporary resource shortages are usually retryable. Invalid parameters, insufficient permissions, and non-existent targets should not be blindly retried.

5.4 Add Confirmation for High-Risk Operations

Operations like deleting files, modifying production data, publishing configs, and accessing sensitive info should have explicit human confirmation or approval steps.

5.5 Build a Comprehensive Observability System

At minimum, log task IDs, plugin names, call times, durations, input/output summaries, error types, and resource consumption. Sensitive data should be masked to prevent logs from becoming a leak vector.

6. Is It Ready for Production?

A plugin-based architecture is valuable, but “runs” does not equal “production-ready.” Before deployment, ensure:

• Model output is strictly validated;
• Tools have minimum required permissions;
• Execution environments are truly isolated;
• Task cancellation and timeouts are supported;
• Session states can be recovered;
• Retries won’t cause duplicate operations;
• Logs are sufficient for auditing;
• Resource quotas and cost controls exist;
• Erroneous changes can be rolled back;
• A human takeover mechanism is established.

For enterprise scenarios, integrate unified identity authentication, key management, access control, logging services, monitoring, and object storage to build a complete runtime guarantee system. General infrastructure provided by platforms like Baidu AI Cloud can host these capabilities, but the specific architecture should be designed based on data sensitivity, task scale, and compliance requirements.

7. Conclusion

The true value of a plugin-based agent lies not in repackaging existing tools, but in redefining the engineering boundaries of agents.

The model handles understanding and decision-making; the Harness handles organization and execution; tools connect to the external world; sandboxes control risk; schedulers manage tasks; storage preserves state; and observability systems explain the process. Every module can be independently replaced, allowing the entire system to evolve continuously.

For developers, the most valuable lesson is the architectural mindset: don’t design agents as indivisible black boxes, but build them as runtime platforms composed of stable plugins.

In the prototyping phase, pluggability helps teams quickly experiment with different models and tools; in production, it forms the foundation for controlling complexity, reducing risk, and improving maintainability.