Native Multimodal Agent Architecture and Engineering Practice
This article explains why "external vision" pipelines fall short in real-world business scenarios, defines the goals of a native multimodal agent, and details a six-layer engineering architecture covering input messaging, perception encoding, cross-modal alignment, reasoning planning, execution feedback, and memory governance. It also covers training strategies, tool-chain loop design, evaluation metrics, and practical deployment recommendations.
Why “External Vision” Is Not Enough
In real business environments, a large amount of critical information does not exist purely as text: web interfaces, mobile screenshots, design mockups, scanned documents, monitoring video, and demo recordings all carry state, layout, timing, and operational intent. If we adopt a “text-centric model + external vision module + interface stitching” approach, the system typically only obtains an image description at one step, then passes that descriptive text to a reasoning module. This chained structure may look loosely coupled, but it has three fundamental problems.
First, perception and reasoning are disconnected. The vision module outputs compressed natural language descriptions, which easily lose fine-grained details such as layout, hierarchy, control relationships, and temporal changes, leaving the reasoning module unable to make judgments based on the original evidence. Second, errors are amplified layer by layer. Once the upstream recognition makes a mistake, downstream planning continues on a false premise, producing a plan that looks plausible but is in fact not executable. Third, closed-loop capability is weak. An agent needs to “see the result — judge whether it succeeded — decide to retry or roll back,” but external-structure systems usually lack a unified mechanism to feed execution results back into the reasoning process as visual observations.
Therefore, the core of a “native multimodal agent” is not simply adding a visual encoder next to a language model. It is about making images, videos, and text share the same protocol and closed loop from input representation, context organization, reasoning planning, and tool execution. Vision is not a post-hoc patch; it is a first-class input alongside text.
Goals and Definition of a Native Multimodal Agent
A native multimodal agent can be understood as a system with four unified capabilities: unified input, unified representation, unified reasoning, and unified execution.
- Unified input: Text, images, video frames, and structured observations enter the context in a consistent message format.
- Unified representation: Cross-modal information is aligned in the same semantic space, preserving spatial positions, temporal order, and source information.
- Unified reasoning: The model can directly reference visual evidence during planning, instead of relying on second-hand descriptions.
- Unified execution: Action results return to the system as new observations, forming a verifiable perception–reasoning–execution loop.
The goal also upgrades from “look and describe” to “look and operate.” The former focuses on whether descriptions are accurate; the latter focuses on whether tasks are completed. The system needs to understand interface states, locate actionable regions, generate reasonable steps, invoke tools, check results, and recover from failures. The evaluation criteria change accordingly: task success rate, step efficiency, error recovery rate, hallucination rate, latency and cost, and safety and compliance all need to be considered.
Architecture Layers: From Message Protocol to Execution Feedback
An engineering-ready native multimodal agent can typically be divided into six layers.
1. Input and Message Layer
The system should define a unified observation message containing modality type, content reference, timestamp, source, resolution or layout information, permission tags, and so on. The purpose is to let the model and the tool chain share the same “source of truth,” avoiding each module maintaining its own separate context.
2. Perception Encoding Layer
Visual input needs to address multi-resolution handling, region cropping, document layout, and temporal sampling of video. The key is not to stuff all pixels into the context at once, but to retain locatable evidence as required by the task — for example, control coordinates, text regions, layer relationships, and time segments.
3. Cross-Modal Alignment Layer
Text, images, and video segments should be associated within a unified representation so that “a certain button,” “a certain description,” and “the result of the previous action” can be referenced together. Alignment happens not only during pre-training, but also during context organization at inference time and when tool results are returned.
4. Reasoning and Planning Layer
The model needs to perform task decomposition, state tracking, tool selection, and risk judgment. During planning, it should explicitly reference observational evidence and include verification steps for uncertain information, rather than directly filling in gaps.
5. Execution and Feedback Layer
The action space should be constrained to forms that are auditable, retryable, and rollback-capable — such as clicking a region, entering text, scrolling, invoking a general tool, or finishing with an answer. Execution results must be fed back as new observations so the system can determine success, failure, or the need for human intervention.
6. Memory and Governance Layer
Short-term memory preserves the current task state; long-term memory stores reusable experience and audit records. At the same time, permission control, sandbox isolation, logging services, and monitoring alerts are required to keep agent behavior traceable.
Training Strategy: Let Visual Evidence Participate in Decision-Making
Native multimodal capabilities require coordinated multi-stage training.
- Pre-training: Cover multi-source data including images with text, documents, interfaces, and videos to learn cross-modal alignment and spatial localization.
- Instruction fine-tuning: Add task-oriented samples so the model learns to generate steps based on visual states.
- Tool-use training: Introduce action trajectories, success and failure cases, and retry/rollback samples, so the model understands the consequences of actions.
- Preference optimization: Reinforce behaviors such as “verify before acting” and “ask or stop when evidence is insufficient.”
For data construction, negative samples are especially important in addition to positive ones: wrong localization, misjudged states, skipped verification, repeated operations, and unauthorized calls should all be annotated. Training objectives can jointly include evidence localization, state prediction, action prediction, and result verification. To reduce hallucinations, the system should encourage the model to reference verifiable observations, output confidence or uncertainty statements, and set checkpoints before critical actions.
Tool-Chain Loop: Protocol, Sandbox, and Observability
The tool chain is not an accessory; it is part of the native closed loop. A schematic pseudo-code example:
type Observation = {kind: "text" | "image" | "video_frame";content: string | bytes;timestamp?: number;meta?: { source: string; resolution?: [number, number]; };};interface AgentLoop {observe(input: Observation[]): State;reason(state: State): Plan;act(plan: Plan): Action;verify(result: Observation): boolean;}
The action protocol should also stay controlled:
Action =| { type: "click", target: Region }| { type: "type", text: string }| { type: "scroll", delta: number }| { type: "call_tool", name: string, args: Record<string, unknown> }| { type: "finish", answer: string };
Engineering requirements include:
- Observation compression and on-demand cropping to avoid context bloat.
- Action idempotency and retry strategies to prevent duplicate submissions.
- Sandbox isolation and least privilege to limit high-risk operations.
- Trajectory replay and log auditing to support issue diagnosis.
- Object storage for trajectories, logging services for audit records, and monitoring alerts for anomaly detection.
Execution results must be fed back as observations to form a “plan → execute → verify → correct” loop.
Evaluation and Implementation Recommendations
Evaluation should not only look at single-turn answer quality, but at closed-loop task performance. Recommended focus metrics include:
- Task success rate
- Average number of steps
- Failure recovery rate
- Visual evidence citation accuracy
- Hallucination rate
- End-to-end latency
- Cost per task
Test sets should cover general task types such as interface-to-code, document understanding, mobile device operation, and temporal video judgment, and should be evaluated in a replayable environment.
For implementation, it is advisable to start with tasks that have clear boundaries and are verifiable, then gradually expand the action space. Start with read-only perception and suggestions before moving to controlled execution. Add human confirmation for high-risk actions. Make permission, audit, and rollback first-class capabilities from the very beginning, rather than adding them as an afterthought. The ultimate value of a native multimodal agent lies in its ability to stably convert visual evidence into an executable, verifiable, and governable action loop.
