Native Multimodal Agents: Architecture, Training, and Toolchains
This article explores the limitations of "bolt-on" multimodal approaches in complex tasks and introduces the concept of native multimodal agents. It details the architecture design focusing on unified representation and closed-loop interfaces, the multi-stage training path from alignment to execution feedback, and the essential toolchains and engineering practices. Finally, it provides evaluation dimensions and implementation recommendations, emphasizing that systems engineering is key to achieving a reliable perception-reasoning-execution loop.
I. Problem Background: Why “Bolt-on” Multimodal Approaches Fall Short
Large language models (LLMs) have demonstrated strong capabilities in language understanding, code generation, and logical reasoning. However, real-world information exists far beyond text. User interfaces, mobile screenshots, design drafts, scanned documents, surveillance videos, charts, and forms are all critical information carriers. If a model can only understand visual content “indirectly” through text interfaces, the system typically needs to invoke an independent vision module for recognition and then concatenate the results into text for the language model to reason over. While this pattern allows for rapid prototyping, it exposes several structural issues in complex tasks.
First, there is an information disconnect between perception and reasoning. Vision modules often output descriptions or bounding boxes at a fixed granularity. The language model cannot “look back” at a specific region based on reasoning needs, nor can it jointly model local details with the global layout. Second, errors accumulate along the pipeline. Minor mistakes in OCR, element detection, or layout analysis are amplified through text concatenation, ultimately manifesting as hallucinations or incorrect operations. Third, execution feedback struggles to form a closed loop. After an agent clicks, types, or scrolls within an interface, it needs to replan based on the new visual state. Bolt-on architectures usually lack a unified trajectory representation and feedback mechanism, resulting in a system that can “see” but “cannot operate accurately.”
Therefore, the core of a “native multimodal agent” is not simply adding a vision encoder next to a language model. Instead, it unifies images, videos, and text into a perception-reasoning-execution closed loop across three dimensions: architecture, training, and toolchains. Vision is no longer an afterthought but a first-class input in the model’s decision-making process.
II. Goals and Boundaries of Native Multimodal Agents
Native multimodal agents typically need to possess four categories of capabilities simultaneously:
- Cross-modal understanding: The ability to jointly process text instructions, UI screenshots, document layouts, and video temporal information.
- Spatial and element localization: The ability to map natural language goals to specific regions, controls, or temporal segments.
- Action planning and tool invocation: The ability to output executable actions such as clicking, typing, scrolling, searching, and code generation.
- Post-execution state verification: The ability to revise plans based on new observations, reducing the risk of irreversible operations.
It should be emphasized that “native” does not mean all capabilities are handled end-to-end by a single model. A more engineering-sound approach is: the model is responsible for unified representation, reasoning, and action decision-making, while the toolchain handles high-precision perception, sandbox execution, and observability. The key is that interface design should not break cross-modal information, and training objectives should cover real execution trajectories rather than isolating visual capabilities into a one-time preprocessing step.
III. Architecture Design: Unified Representation and Closed-Loop Interfaces
3.1 Vision Access Layer: From “Description Input” to “Queryable State”
The vision access layer must handle inputs of varying resolutions, aspect ratios, and modalities. Common designs include adaptive tiling and resolution normalization, local region cropping and zooming, multi-image and video frame sampling, as well as positional encoding and layout structure encoding. For UI screenshots, in addition to the overall image, control hierarchies, text regions, and coordinate mappings should be preserved; for documents, reading order and layout relationships must be retained; for videos, timestamps and event boundaries are essential.
More critically, the output of the vision access layer should not just be “a description,” but a state representation that can be repeatedly queried during the reasoning phase. For example, when planning a click action, the model can request a zoomed-in view of a specific region or an OCR result; after an execution failure, it can compare the differences between before and after screenshots. This queryable state is the foundation of a native closed loop.
3.2 Cross-Modal Fusion: Sharing Context Between Language Reasoning and Spatial Localization
The cross-modal fusion layer is responsible for unifying text instructions, visual features, and tool return results into the same context. Engineering approaches can employ multimodal positional encoding, region tokens, and action history encoding, enabling the model to perceive both “target semantics” and “spatial positions” when generating actions. Training should encourage the model to learn coreference relationships, such as “the Save button in the top right corner,” “the outlier in the second table,” or “the person’s action around the 30-second mark in the video.”
The fusion layer also needs to handle long context issues. Multi-images, video frames, and tool returns quickly consume the context budget. Therefore, token compression, keyframe selection, region summarization, and hierarchical memory mechanisms are required. Compression should not only aim for brevity but also preserve traceable coordinates, timestamps, and evidence chains.
3.3 Action and Tool Interfaces: Defining Stable Execution Protocols
The agent’s action space should be as explicit and verifiable as possible. Common actions include region click, coordinate click, text input, scrolling, dragging, keyboard shortcuts, file reading, code generation and execution, and search queries. Each action should carry a target region, confidence score, expected result, and rollback strategy. Tool interfaces should return structured results, including success status, observations, error codes, and new visual states.
A simplified pseudocode for the closed loop is as follows:
class NativeMultimodalAgent:def run(self, goal, context):# Perceive the initial state from the contextstate = self.perceive(context)# Generate an initial plan based on the goal and stateplan = self.reason(goal, state)while not plan.done:# Select the next action based on the plan and stateaction = self.select_action(plan, state)# Execute the action in an isolated sandboxresult = self.execute_in_sandbox(action)if not result.success:# Replan based on error information if execution failsplan = self.replan_with_error(goal, state, result)# Update the state with the new observationstate = self.update_state(state, result.observation)# Replan the remaining stepsplan = self.replan(goal, state, plan)# Verify the final state against the goalreturn self.verify(goal, state)
This pseudocode reflects three principles: execution must occur in a sandbox or controlled environment; the observed state must be updated after each execution; failures must trigger replanning based on error information rather than blind retries.
IV. Training Path: From Alignment to Execution Feedback
4.1 Data System: Covering Real Task Trajectories
The training data for native multimodal agents should not consist solely of “image-text pairs” but should cover “goal-observation-action-result” trajectories. Typical data includes UI screenshots with operation sequences, document Q&A with layout evidence, design drafts with code mappings, video events with temporal localization, and tool invocation logs with failure recovery records. Data annotation must preserve coordinates, timestamps, control types, action results, and verification evidence.
For data governance, processes for deduplication, desensitization, permission auditing, and quality sampling should be established. For UI and document data, ensure no sensitive information is included; for execution trajectories, record environment versions and reproducible conditions.
4.2 Multi-Stage Training: Align, then Reason, then Execute
Training can typically be divided into multiple stages.
- Stage 1: Cross-modal alignment, establishing correspondences between text, image regions, layout structures, and temporal segments.
- Stage 2: Instruction tuning, covering tasks like UI understanding, document Q&A, chart reasoning, and video summarization.
- Stage 3: Agent trajectory training, enabling the model to learn the loop between observation, planning, action, and verification.
- Stage 4: Execution feedback, optimizing action selection through success/failure signals, rule verification, and preference data.
Training objectives should focus on both result correctness and process reliability. For instance, it requires not only the correct final answer but also the correct citation of key evidence regions, replayable action sequences, and recovery capabilities after failure. To maintain pure text capabilities, high-quality text corpora should be mixed into the training, alongside regression evaluations to prevent language capability degradation caused by visual training.
4.3 Feedback and Reinforcement: Constraining Hallucinations with Verifiable Signals
A major advantage of agent tasks is the presence of verifiable signals: did the UI change after a click? Was the form submitted successfully? Did the code pass the tests? Does the search result contain the target evidence? These signals can be used to construct rewards or filtering rules, training the model to reduce invalid actions and false descriptions. For irreversible operations, the model should be prioritized to output a confirmation action or request human confirmation.
The feedback mechanism should also include trajectory replay and error attribution: distinguishing between perception errors, localization errors, planning errors, and execution environment errors. Only with clear attribution can training data and toolchains be specifically improved.
V. Toolchains and Engineering Implementation
5.1 Perception Tools: Emphasizing High Precision and Traceability
Perception tools in the toolchain can include screenshot cropping, region zooming, OCR, control detection, layout analysis, color and state recognition, and video keyframe extraction. Tool outputs should preserve coordinate mappings and confidence scores to facilitate model traceback of evidence. For UI tasks, it is recommended to maintain the mapping of “screenshot coordinates - control tree - text content”; for document tasks, maintain the mapping of “paragraph - layout region - reading order.”
5.2 Execution Sandbox: Isolation, Rollback, and Auditing
The execution phase must take place in an isolated environment, restricting network, file system, and system permissions. Each action should record its input, output, duration, and result status, supporting trajectory replay. For high-risk actions, such as deletion, payment, or form submission, secondary confirmation, permission verification, and human handover mechanisms should be set up.
5.3 Observability: Making the Closed Loop Debuggable
During engineering implementation, logging services, monitoring alerts, and distributed tracing are needed to observe agent behavior. Key metrics include task success rate, average action steps, retry rate, hallucination rate, tool invocation failure rate, end-to-end latency, and cost. On the deployment side, general services like object storage, container platforms, logging services, and monitoring alerts from providers such as Baidu AI Cloud can be used to build data management, training and inference, execution auditing, and observability pipelines. However, the specific architecture should be designed in accordance with business permissions and compliance requirements.
VI. Evaluation and Capability Retention
Evaluation should not only look at the final answer but also the process quality. It is recommended to design evaluations across five dimensions: cross-modal understanding and evidence citation, spatial localization and action accuracy, long-horizon task success rate, failure recovery capability, and pure text capability regression. Front-end code generation tasks can evaluate the mapping from design drafts to structured code; mobile agent interaction tasks can assess multi-step operations and state verification; document and video tasks can test evidence localization and temporal understanding.
Simultaneously, latency, cost, and stability must be tracked. Native multimodal agents often require multiple rounds of observation and tool invocations. Without caching, parallel tool calls, and context compression, end-to-end costs will escalate rapidly. The evaluation set should cover different resolutions, languages, complex layouts, and abnormal states to prevent the model from only adapting to ideal samples.
VII. Engineering Implementation Recommendations
- Define verifiable task boundaries first: Start with low-risk, rollback-capable scenarios, such as document information extraction, UI-assisted operations, and design draft inspection.
- Establish a unified trajectory format: Ensure that observations, actions, results, and evidence are replayable.
- Decouple perception tools from model decisions but keep them queryable: Avoid information loss caused by one-time text descriptions.
- Mix text regression data in training: Continuously monitor language capabilities to prevent degradation.
- Set up least privilege, human confirmation, and auditing mechanisms: Especially when operations involve external systems.
Overall, the technical challenge of native multimodal agents lies not in “plugging in a vision encoder,” but in the systems engineering of unified representation, trajectory training, and a toolchain closed loop. Only when visual observation, language reasoning, action execution, and result verification form a stable cycle can the model reduce hallucinations and improve reliability in real-world UI, document, and video tasks.
