The Shared Architecture of AI Agents: Models vs. Runtimes

1. Separate Thinking from Acting

Before discussing agents, it is necessary to separate two things. Otherwise, most of the following discussion will lose focus.

The model is only responsible for “thinking.” Given a piece of text, it performs understanding, reasoning, and generation, then outputs another piece of text. It knows nothing about the file system, has no connection to a database, does not know the current time, and cannot take action after a user clicks something. Essentially, it is a nearly pure mapping function: input context, output the next fragment. The model capability determines the upper bound of reasoning quality.

The agent runtime—also called the agent framework or harness layer—is responsible for “doing.” It prepares materials outside the model, registers tools, initiates real calls, feeds results back to the model, and decides when to continue and when to stop. Tool-call selection, context trimming, failure retries, and whether a write operation requires human confirmation are all determined by this layer.

This boundary is often overlooked because of three product-level illusions:

• Naming is often aligned with the model, making it look like a single integrated product.
• Interaction appears as a continuous conversation, so the internal layering is invisible.
• Evaluation often emphasizes the score of the underlying model, while the contribution of the runtime is implicitly counted.

Once this boundary is clear, a counterintuitive conclusion emerges: the coding assistant, general task agent, data analysis agent, and browser-operation agent you use every day may look very different at the product level, but their underlying architecture is isomorphic. They all use the same “reason-act-observe” loop, combined with different tool sets, context strategies, and governance boundaries. The difference is not in structure, but in parameters.

2. Five Layers: How to Decompose Any Agent

1. Inference layer

The model itself. The key fact affecting portability is that this layer is replaceable. With the same runtime, if the backend model is replaced with another model of the same specification, the system still works. Only the quality curve changes. Any architecture that hard-codes a specific model’s proprietary prompt syntax or output format into business logic will eventually face migration costs.

2. Orchestration layer

The heart of an agent is a loop, not a single question-and-answer exchange. It must clearly define:

• the entry state of the loop;
• what should happen at each step;
• the termination condition;
• how to handle single-step failure;
• the upper limit of total steps and total cost.

Without a termination condition, an agent will almost inevitably evolve into “infinite self-confirmation” in practice, which burns money and is difficult to control.

def run(task, tools, model, budget):
    history = [system_prompt(), user(task)]
    for step in range(budget.max_steps):
        reply = model.generate(history, tools=tools.schemas())
        history.append(reply)
        if reply.tool_calls is None:          # model considers the task complete
            return reply.text
        for call in reply.tool_calls:
            tool = tools.get(call.name)
            if not validate(call, tool):      # parameter validation; reject out-of-scope fields
                history.append(error_msg(call, "invalid arguments"))
                continue
            obs = tool.run(**call.arguments)  # execute in a sandbox with permission constraints
            history.append(observation(call.id, truncate(obs)))
            budget.spend(obs.tokens)
        if budget.exceeded():                 # step / token / time limits
            return partial_report(history)
    return give_up(history)

None of these lines belong to “the model.” This is the runtime itself.

3. Context layer

The model has no memory, only a window. The “memory” an agent appears to have is entirely assembled by the runtime: system instructions, task descriptions, tool lists, historical trajectories, retrieved materials, and compressed summaries of historical trajectories.

The core tension of context engineering is: more information is safer, but the context window and latency are limited. Common practices include:

• truncating tool results;
• summarizing old trajectories;
• changing full document injection to on-demand retrieval;
• externalizing intermediate state to files or storage and passing only references.

4. Tools and execution layer

Tools are the agent’s hands and feet, and they are also where the risk exposure is greatest. The tool description itself is the interface contract: naming should be clear, parameters should be few and orthogonal, return values should be structured, and error messages should be understandable by the model so that it can correct itself.

On the execution side, the system also needs:

• sandbox isolation;
• timeout limits;
• output size limits;
• explicit modeling of side effects, distinguishing read-only operations, reversible writes, and irreversible writes.

5. Governance and observability layer

This layer determines whether the system can go into production. It includes:

• least privilege;
• human approval for sensitive operations;
• full trajectory logging;
• metrics and alerting;
• cost and quota management.

In enterprise scenarios, trajectory logs and audits often determine project success more than “a smarter model.”

3. Where the Differences Come From: Six Variables in the Same Loop

• Prompt and role settings: determine the model’s default behavioral tendencies and are the cheapest adjustment lever.
• Tool granularity: providing search(query) versus providing search_repo / search_doc / search_history directly affects the difficulty of the model’s decisions.
• Context compression strategy: the combination of summarization, truncation, and external references determines the stability of long-running tasks.
• Error recovery: whether failure information is fed back to the model, whether alternate paths are allowed, and how many retries are permitted all affect task completion rates.
• Planning method: single-step reactive execution versus planning first and then executing step by step. Plan-based execution is more controllable but brittle; reactive execution is flexible but prone to divergence.
• Human-in-the-loop points: which actions require confirmation determines whether the agent can be used with production data.

4. Practical Recommendations

Prioritize tool design over model tuning. Many cases of “the model is not smart enough” are actually caused by vague tool descriptions, verbose return values, or unusable error messages. Write the contract clearly first.

Separate read and write permissions. Read-only tools can be allowed to execute automatically. Write operations and irreversible operations should introduce approval or allow-listed paths. This is a security baseline and has nothing to do with model capability.

Make context budgets explicit. Set separate quotas for system instructions, history, tool results, and retrieved fragments, rather than simply using everything until the limit is reached.

Build an evaluation set instead of relying on intuition. Collect real tasks, run fixed scripts offline, and record task success rate, average steps, token consumption, number of human interventions, and trajectory replay. When a prompt is changed or a model is replaced, this provides comparable numbers. This is the watershed for engineering agents.

Design for replaceability. Consolidate model calls into a thin adapter layer that uniformly handles input and output formats, streaming responses, tool calls, and error semantics. Business code should depend only on this abstraction.

For infrastructure, you can directly reuse general cloud capabilities: object storage for trajectories and artifacts, logging services for runtime records, monitoring and alerting for success rates and costs, and container platforms for sandbox execution environments. For example, when building on Baidu AI Cloud, these capabilities belong to standard categories and do not require building custom solutions.

Move permissions, auditing, and redaction to the front. Tool calls are real operations. For any agent that can access production data, trajectory logging and sensitive-field redaction should be built in from the first version.

5. Common Pitfalls

1. Attributing all quality issues to the model while ignoring the contributions of prompts, tools, and context.
2. Piling up tools without restraint, making the model choose incorrectly among dozens of similar tools.
3. Missing termination conditions and budget limits.
4. Having only demos without an evaluation set, judging every change by whether it “looks better.”
5. Treating conversation history as the only memory, which inevitably causes forgetting in long-running tasks.
6. Deeply binding to a single model’s private syntax, postponing migration costs until the most expensive moment.

6. Conclusion

Four seemingly different agents can be decomposed into the same thing: a replaceable inference backend, plus a self-authored orchestration loop, a set of well-designed tools, a context strategy, and a layer of governance constraints.

What truly requires long-term investment is the latter four. And they happen to live in your own code.