Everything Is a Plugin: AI Agent Runtime Framework Design and Practice

1. From Chatting to Getting Work Done: Why a Runtime Layer Is Needed

Large language models are essentially probabilistic sequence-prediction engines. They can understand natural language and generate code or text, but they cannot read files, execute commands, access networks, or operate external systems on their own. What enables an agent to carry out real tasks is a runtime layer between the model and the real environment. The industry commonly refers to this intermediary layer as a harness, or a load-bearing layer.

A useful analogy is that the model is the brain, responsible for reasoning and decision-making, while the runtime is the hands, feet, and senses. It translates decisions into executable actions and feeds execution results back to the model to form a closed loop. Neither part is sufficient on its own: without a runtime, a model can only talk without doing; without a model, a runtime lacks the intelligent decision-making that drives it.

2. Core Architecture Idea: Model + Harness = Agent

A widely accepted design formula is:

Model + Harness = Agent

• Model is responsible for thinking: reasoning, planning, language understanding, and code generation.
• Harness is responsible for doing: tool orchestration, context management, state persistence, permission and security constraints, and collecting and feeding back execution results.

The value of this formula is that it decouples agent capabilities from any specific model. A stronger model can raise the capability ceiling, but a well-designed runtime determines whether the entire system can stably, safely, and scalably complete end-to-end tasks. In other words, the runtime is the core variable that determines the engineering usability of an agent.

3. Design Principle One: Everything Is a Plugin

“Everything is a plugin” is one of the most distinctive architectural claims of this type of framework. It abstracts capability units in the system—tools, model access, context strategies, interfaces, and executors—into pluggable plugins. The core kernel is only responsible for plugin discovery, loading, dependency resolution, and lifecycle management.

The benefits of this model are straightforward:

• Low coupling: Adding a new external tool, such as a database query, file operation, or HTTP call, does not require changes to the kernel. You only need to implement the agreed interface and register the plugin.
• High reuse: The community can independently develop and distribute plugins around the same interface contract, forming a healthy extension ecosystem.
• Easy composition: Users can freely assemble plugin combinations according to task needs and customize differentiated agent forms.

At the center of this approach is a stable plugin contract. A representative interface definition may look like this:

interface AgentPlugin {
  name: string;                 // Unique plugin identifier
  version: string;              // Semantic version
  dependencies?: string[];      // Other plugins it depends on
  setup(ctx: PluginContext): void;  // Register tools, events, and configuration
  teardown?(): Promise<void>;       // Release and clean up resources
}

As long as the contract is implemented, any capability can be integrated in a unified way. This is the engineering foundation of “everything is a plugin.”

4. Design Principle Two: Model-Agnostic Architecture

Another type of runtime adopts an open “model-agnostic” strategy. It does not deeply bind to any single model provider. Instead, it abstracts a unified model invocation protocol and encapsulates differences between providers in adapters. In this way, the model access layer can support dozens of mainstream model services, and users can switch models through configuration without changing business code.

A unified model abstraction usually contains several key elements:

1. Message format normalization: Unify roles such as system, user, assistant, and tool, as well as multimodal content, into a standard structure.
2. Unified tool-calling protocol: Wrap differences in function calling or tool use across providers into a consistent call and return format.
3. Streaming output adaptation: Unify token streams, incremental events, and error-handling semantics.
4. Capability detection: Identify whether a specific model supports multimodal input, structured output, long context, and other features, and handle degradation when needed.

The significance of this design is to avoid vendor lock-in and allow the lifecycle of an agent framework to remain independent of the rise and fall of any single model.

5. Preset Execution Modes: Matching Capabilities to Task Complexity

Real-world tasks vary greatly in complexity. Mature runtimes therefore often provide multiple preset modes, allowing users to flexibly balance capability, stability, and resource cost. Common categories in this type of framework include:

• Standard mode: Conventional multi-step reasoning and tool calling, balancing capability and controllability. It is suitable for most everyday tasks.
• Programmatic Tool Calling mode: Orchestrates multiple tool calls into an executable program or workflow, reducing repeated decision-making at each step by the model. This lowers token cost and improves determinism and execution efficiency.
• Minimal mode: Minimizes context injection and the tool surface exposed to the model. It is suitable for lightweight, low-latency, privacy-sensitive scenarios.
• Creative mode: Relaxes constraints and encourages divergent generation and exploration. It is suitable for open-ended tasks such as creative writing and solution design.

Mode selection is essentially an engineering trade-off. The more deterministic and reproducible a task is, the more suitable it is for programmatic or minimal modes. The more open-ended a task is, the more it requires creativity and a broader context.

6. Connecting to Real Environments: The Tool-Calling Execution Loop

The core value of a runtime lies in the closed loop of “decision → execution → feedback.” A typical tool-calling loop works as follows:

1. Assemble the context: system prompt, conversation history, and available tool descriptions
2. Call the model and receive a response
3. If the response contains a tool-call request, parse the arguments
4. Execute the corresponding plugin tool and capture the result or exception
5. Feed the result back to the model as new context
6. Repeat steps 2–5 until the model returns a final answer or a termination condition is triggered

The corresponding illustrative logic is as follows:

while (!done) {
  const resp = await model.chat({ messages, tools });
  if (resp.toolCalls?.length) {
    for (const call of resp.toolCalls) {
      const result = await executeTool(call.name, call.args); // Execute the plugin
      messages.push(toolMessage(call.id, result));           // Feed the result back
    }
    continue; // Continue reasoning in the next iteration
  }
  done = true; // No more tool calls; return the final answer
}

From an engineering perspective, additional attention should be paid to limits on the number of iterations or maximum steps, timeouts and retries, idempotency, result truncation, and validation of model-hallucinated parameters.

7. Safety Boundaries and Engineering Constraints

Once an agent is allowed to operate in real environments, safety guardrails must be established at the same time. Otherwise, risks can be significantly amplified:

• Principle of least privilege: Each plugin should request only the minimum permissions required to complete its task, avoiding a “master key” that can access everything.
• Secondary confirmation for dangerous operations: Irreversible operations such as deletion, overwriting, and external writes should require manual confirmation or a whitelist.
• Sandboxing and isolation: Command execution and file reads or writes should be performed in a controlled sandbox, limiting the range of accessible resources.
• Auditability and observability: Log every tool call, including parameters and results, and connect to monitoring, logging, and alerting systems to support tracing and troubleshooting.
• Context and secret management: Sensitive credentials should not enter the model context. They should be injected on demand through a secrets-management component.

8. Engineering Implementation Recommendations

If you are preparing to build or select a plugin-based agent runtime, you can start from the following steps:

1. Define the contract first: Prioritize designing a stable plugin interface and model abstraction layer, then gradually fill in specific capabilities.
2. Abstract before implementing: Interface tools, models, and execution strategies to avoid coupling with specific implementations.
3. Build in security: Treat permissions, sandboxing, and auditing as first-class architectural concerns rather than later patches.
4. Layer execution modes: Provide differentiated execution modes for tasks of different complexity, balancing cost and quality.
5. Validate closed loops: Run the full “decision → execution → feedback” loop end to end with real tasks, and supplement it with exception and edge-case tests.

Overall, “Model + Harness = Agent” reveals a key truth in agent engineering: the model determines the upper limit of intelligence, while the runtime determines the lower limit of practical deployment. Using “everything is a plugin” and “model-agnostic” as design foundations, building an extensible, governable, and replaceable agent runtime is a key path to turning large-model capabilities into real production productivity.