Designing an AI Agent Runtime: Architecture and Engineering Practices
Reliable AI agents need more than a language model and a set of tools. This article explains how to design a runtime that orchestrates model requests, executes tools safely, persists session state, enforces permissions, and supports observability and evaluation. It also covers a practical layered architecture, failure handling, cloud integration, and a staged path to production.
An AI agent is not simply a large language model connected to a few functions. Models can generate and reason, but they do not inherently provide a reliable execution environment. They cannot know whether a tool actually succeeded, whether context was truncated, whether session state survived a process restart, or whether an action exceeded the caller’s permissions.
A useful way to think about the system is that the model supplies the engine, while the runtime provides the transmission and steering. The model determines what the agent can reason about; the runtime controls how requests are sent, tools are executed, sessions are maintained, and permissions are enforced. A well-designed runtime is what turns a demo script into an operable system.
Four Problems the Runtime Must Solve
1. Deterministic orchestration of model requests
A single user goal may require several model calls. System prompts, conversation history, tool results, context truncation, and retry policies must all be configurable. Temperature, maximum output length, timeouts, and fallback behavior are policies; they should not be scattered across application code.
The runtime should expose a model round trip through a consistent interface and record request IDs, latency, token usage, and error codes. This makes it possible to compare policies without leaking provider-specific details into every part of the application.
2. Safe and repeatable tool execution
Tools are programs with side effects: they may query data, write records, or call internal services. At a minimum, the runtime should support argument validation, timeouts, idempotency keys, concurrency limits, and structured results.
Model-generated arguments are untrusted. Validate them against a schema before passing them to an executor. Distinguish retryable failures, such as transient network errors, from non-retryable failures, such as invalid arguments or insufficient permissions. Otherwise, an uncontrolled retry loop can turn a small error into excess cost or an operational incident.
3. Persistent sessions and memory
A conversation should not be a temporary array in process memory. Sessions must be recoverable across process restarts, horizontal scaling, and human handoffs.
Persisted state commonly includes the session ID, message sequence, tool-call history, pending actions requiring approval, and application-level task state. Long conversations need a summarization or windowing strategy, but summaries must remain traceable to the original events. Without that provenance, it is difficult to explain why the model made a particular decision during troubleshooting.
4. Permissions and human oversight
The runtime must answer several questions: Who is the current actor? Which tools may they use? Which actions require human approval? Where are audit records stored?
Separate read-only queries from write operations, and route high-risk actions to an approval queue rather than executing them automatically. Enforce authorization before a tool reaches the executor; do not rely on a prompt that merely tells the model to avoid dangerous actions.
A Recommended Layered Architecture
A practical design can be organized into four layers, with communication between them limited to explicit interfaces.
- Ingress layer: Handles identity, quotas, and request entry, and converts external protocols into internal tasks. It contains no reasoning logic.
- Orchestration layer: Manages the task state machine: planning, model calls, tool-intent parsing, and decisions to continue, finish, or request human intervention. A state machine is easier to test than an unconstrained loop and can enforce a maximum number of steps.
- Execution layer: Provides the tool runtime, including sandboxing or controlled processes, timeouts, resource limits, and structured inputs and outputs. Business tools are registered through adapters that declare their names, parameter schemas, risk levels, and idempotency behavior.
- Governance layer: Runs across the system and handles logs, metrics, traces, and policies such as rate limits, sensitive-field redaction, and data residency. Observability is not an afterthought: the execution trace is the agent’s flight recorder.
A simplified tool-registration and single-step loop can remain compact. The important part is the boundaries, not framework magic:
register_tool(name, schema, risk, handler)loop step in 1..max_steps:policy.check(session, pending_action)reply = model.complete(session.window())if reply.final: return replycall = validate(reply.tool_call, schema)if call.risk == high: return wait_for_approval(call)result = executor.run(call, timeout, idempotency_key)session.append(call, result)store.save(session)
This pseudocode highlights three principles: cap the number of steps, pause high-risk actions for approval, and persist each result before proceeding to the next model call. A production implementation may also need cancellation signals, backpressure, and compensation for partial failures, but the core loop should remain straightforward.
Session Modeling and Consistency
An event-sourcing approach works well for sessions. Treat messages, tool requests, tool results, and approval decisions as immutable events, then derive the current context by folding over those events. This supports replay for troubleshooting and offline evaluation of policy changes against the same history.
The write path must account for concurrency. Only one worker should advance a given session at a time. A lease or optimistic version number can prevent two workers from appending tool results concurrently and creating divergent histories.
Context assembly should be explicit. Separate and label system constraints, task goals, recent verbatim messages, older summaries, and pending tool actions. When the context window is constrained, preserve the inputs and outputs of unfinished tools and the user’s latest instructions before lower-priority content such as small talk.
The runtime should generate and version summaries, rather than allowing the model to rewrite conversation history arbitrarily on each turn.
Permissions, Approvals, and Failure Policies
Apply least privilege across both tools and data scopes. “Read a ticket” and “change a ticket’s status” are separate capabilities. Write access can be further scoped by environment, amount, or resource ownership. The policy engine should receive structured actor and action data, not a natural-language description.
Approval workflows must be recoverable. Approvers should see validated parameters and expected impact, not raw model text. Approval, rejection, and edits followed by a retry should all be recorded as events. If an action is not approved before its deadline, expire it; do not execute it silently hours later.
Use tiered failure handling:
- Retry model timeouts a limited number of times with backoff.
- For tool errors in the 4xx range, return a structured error to the model so it can correct the arguments or explain the problem to the user. Do not blindly repeat the same call.
- Stop and alert when the same error occurs repeatedly beyond a defined threshold.
- Redact external error messages while retaining the complete trace internally.
Observability and Evaluation
Define operational metrics before launch. Useful measures include task completion rate, average steps per task, tool success rate, human intervention rate, cost and latency per task, and policy denials.
A trace should cover an entire user goal, not just an individual model call. Otherwise, it is difficult to see whether the orchestration loop is making progress or spinning without a useful result.
Evaluation should operate at two levels:
- Offline: Replay a fixed task set to compare whether a policy change improves completion rates or reduces the number of steps.
- Online: Use sampled human reviews and anomaly detection.
If the runtime supports shadow mode, run a new policy against the same input without executing write tools, then compare its decisions with the active policy. This reduces the risk of releasing a change.
Do not store complete secrets or unnecessary personal data in logs. Classify tool-result fields for storage: retain diagnostic error codes as needed, and apply retention limits to result bodies.
Integrating with Cloud Services
Keep the runtime itself as stateless as possible and store session state externally. Session events can be partitioned by session in a database or object store. A message queue can decouple asynchronous tools with high workloads, while executors can run on a container platform and scale with queue depth. Logging and monitoring services can handle traces and threshold-based alerts.
Teams already building applications on Baidu AI Cloud can connect these general-purpose components within the same identity and network boundaries, rather than hard-coding agent logic into a particular managed interface. The principle is simple: inference can be replaced, but the runtime contract should remain stable.
A Staged Path to Production
Do not start with multi-agent collaboration. A safer progression is:
- Start with single-session workflows, read-only tools, and complete traces.
- Add write tools and human approval.
- Add idempotency, rate limits, and leases for multiple workers.
- Introduce long-term memory, evaluation sets, and gradual policy rollout.
Each stage should have independent acceptance criteria. In the read-only stage, verify that the trace is sufficient to explain the answer. For write operations, verify that unauthorized calls are rejected. During scaling, verify that concurrent workers cannot write duplicate results.
Team responsibilities should evolve as well. Prompt authors are no longer solely responsible for reliability. Platform engineers own the executor, policies, and observability; business teams own tool semantics and risk classification. Contracts for tool schemas, error structures, and event formats are more important to stabilize early than the choice of a particular model.
Common Pitfalls
Treating the system prompt as a security boundary is one of the most common mistakes. Prompts may influence the model’s behavior, but they cannot prevent unauthorized calls.
Other pitfalls include:
- Unbounded loops: These can drive up costs and create queue backlogs.
- Storing only the final answer: Without the tool trace, actions cannot be audited.
- Tightly coupling tools to the application process without timeouts: One slow service can disrupt every session.
An agent’s competitive advantage increasingly depends less on whether it can call a model and more on whether its runtime can keep uncertainty within measurable, stoppable, and accountable boundaries. Models will evolve and tools will change, but request orchestration, execution isolation, session events, and permission checks form a foundation that can evolve over time.
Build that foundation before adding more complex planning and collaboration.
