Native Multimodal Agents: Architecture, Training, and Production Practices
This article examines why bolt-on visual modules create bottlenecks in real multimodal tasks and introduces a native multimodal agent architecture built on unified multimodal representation and a closed-loop state machine. It covers input representation, vision-language fusion, the perception-reasoning-execution loop, multi-stage training with execution feedback, toolchain and security design, observability and cost control, layered evaluation, and practical deployment guidance. Readers will learn how to connect perception, reasoning, tool use, and feedback into a stable, auditable, and recoverable agent system for interface, document, video, and automation tasks.
1. Problem Background: The Breakpoints of Bolt-On Multimodality
Information in real business scenarios does not exist only as text. Web interfaces, mobile screenshots, design files, scanned documents, surveillance video, and operation recordings all contain substantial visual semantics. The traditional approach usually attaches a visual module outside the language model: a vision model first generates descriptions, tags, or structured text; the language model then reasons over that output; finally, external tools are called to execute actions. This bolt-on approach can work quickly in demos, but it exposes several structural problems in complex tasks.
First, there is an information bottleneck between perception and reasoning. Natural language descriptions produced by the visual module often lose spatial position, hierarchy, fine-grained control state, and temporal changes. The language model can only make judgments based on compressed text, which easily leads to hallucinations that “look right but fail during execution.” Second, errors are amplified across stages. A visual grounding bias can lead to incorrect parameters in subsequent tool calls, while execution failures are difficult to use to correct the original perception conclusion. Third, state is inconsistent. During multi-turn operations, screenshots, documents, tool return values, and historical actions are scattered across different modules. The model struggles to form a unified global state, and long-task success rates decline. Fourth, the engineering chain is complex. Multiple models, multiple services, multiple prompt sets, and coordinate-mapping logic are stacked together, making latency, cost, and observability difficult to control.
Therefore, a native multimodal agent is not about simply adding a visual input port. It makes vision part of model reasoning and action: images, video, and text are understood in the same representation and state machine, and tool execution results enter the next round of reasoning as new observations, forming a closed perception-reasoning-execution loop.
2. Definition and Goals of Native Multimodal Agents
A native multimodal agent can be understood as an intelligent system that is based on a unified multimodal representation, has cross-image, video, and text perception capabilities, can reason and plan according to a goal, call tools, and continuously correct its behavior using execution feedback. Its key differences from a “text model plus visual plugin” are threefold.
First, unified representation. Visual content is not merely transcribed into text. Fine-grained visual features participate in attention computation and jointly determine the next action together with the language context. Second, joint training. During pretraining, instruction fine-tuning, and agent trajectory training, the model is exposed to both multimodal data and tool-use data, learning the joint distribution of “see-think-act” rather than stitching them together at inference time. Third, closed-loop execution. The model output can be an answer or a tool call. Screenshots, document parsing results, page states, execution logs, and other tool returns enter a unified state and drive the next round of perception and decision-making.
This type of system is suitable for tasks such as interface-to-code generation, screenshot question answering, document field extraction, video event localization, mobile or desktop operation automation, and multi-step information retrieval and verification. Its core goal is not a strong single-point metric, but interpretable, recoverable, and auditable task completion in real environments.
3. Architecture Design: Unified Representation and a Closed-Loop State Machine
3.1 Unified Input Representation
A native multimodal architecture first needs to map different modalities into a sequence or feature space that can be jointly modeled. Images usually require dynamic resolution processing: global encoding for the whole image and crop-and-zoom for key regions, balancing layout structure and small text. Video requires sparse sampling of keyframes and injection of temporal position or timestamp information to avoid losing event order through simple frame extraction. Text, visual features, and structured results returned by tools are distinguished by type markers, but they enter the same context and are attended to by the model in a unified way.
The key tradeoff here is between information retention and computational cost. Higher resolution makes small controls and text clearer, but the number of visual tokens increases, raising latency and memory pressure. Engineering practice often uses adaptive tiling, region focus, multi-scale fusion, and caching strategies: the first round looks at the global view, and subsequent rounds zoom into local regions according to task focus, preserving key evidence under a limited budget.
3.2 Vision-Language Backbone Fusion
The visual encoder extracts local and global features, while the language backbone handles cross-modal reasoning. Fusion can use general approaches such as early fusion, cross-attention, or gated injection. The point is not a fixed structure, but enabling visual features to be repeatedly queried across multiple layers of reasoning rather than being converted into a description only once at the input stage. To maintain text-only capability, training needs to mix high-quality text data and use strategies such as replay, regularization, or parameter isolation to prevent multimodal training from degrading language ability.
3.3 Perception-Reasoning-Execution Loop
Agent operation can be abstracted as a state machine: the state contains the goal, current observation, action history, tool results, and memory; the policy network selects either “answer directly” or “call a tool” based on the state; the executor calls the tool and returns an observation; the observation updates the state through the perception module and enters the next round. The pseudocode is as follows:
state = {"goal": user_goal,"observation": perceive(initial_input),"history": [],"memory": []}for step in range(max_steps):decision = policy(state)if decision.type == "answer":return decision.contentresult = tool_registry.call(name=decision.tool,args=decision.args,timeout=decision.timeout,permission=decision.permission)state["history"].append((decision, result))state["observation"] = perceive(result.observation)state["memory"] = update_memory(state["memory"], decision, result)return fallback_or_confirm(state)
This state machine looks simple, but engineering details determine the upper limit. The action space needs strict constraints, and tool parameters need structured schemas and validation. Observations should be standardized, for example with a unified Cartesian coordinate system, element identifiers, confidence scores, and timestamps. Failure results should not return only error text; they should include recoverable information such as timeout, insufficient permission, target not found, or interface not fully loaded, so the model can choose to retry, take an alternate path, or request human confirmation.
4. Training Path: From Cross-Modal Alignment to Execution Feedback
4.1 Multi-Stage Training
Native multimodal agents usually require multi-stage training. The first stage is cross-modal alignment pretraining, where the model learns correspondences among image regions, document layouts, video frames, and text semantics. The second stage is multimodal instruction fine-tuning, covering tasks such as screenshot question answering, interface element grounding, document extraction, chart understanding, and video event description. The third stage is agent trajectory training, using “goal-observation-action-result-correction” trajectory data so the model learns multi-step planning and tool calling. The fourth stage is preference optimization or reinforcement optimization, using signals such as task success rate, step efficiency, execution correctness, and human feedback to optimize the policy.
Training data should combine real and synthetic data. Real data preserves business complexity, while synthetic data can cover long-tail interfaces, abnormal states, and boundary conditions. For interface-to-code tasks, data should include screenshots, structured elements, layout relationships, style information, and executable validation results. For mobile operation tasks, it should include pre- and post-operation states, action types, target controls, and failure recovery trajectories. For document tasks, it should include layout, reading order, field relationships, and extraction answers. Attention should also be paid to de-leakage and debiasing so the model does not memorize fixed templates instead of learning transferable perception and reasoning capabilities.
4.2 Execution Feedback and Preservation of Text-Only Capability
The key to closed-loop training is letting execution results participate in learning. Tool call success or failure, whether a page navigates, whether code compiles, whether rendering is consistent, and whether fields pass validation can all serve as weak supervision or verification signals. The model needs to learn to correct its plan based on feedback rather than generating an unverifiable long chain in one shot. At the same time, a text-only regression evaluation set must be established to monitor whether language understanding, logical reasoning, and long-context capabilities decline due to multimodal training. Multimodal capability should not come at the expense of base language ability.
5. Toolchain and Engineering System
5.1 Perception Service Layer
The perception service layer handles common capabilities such as image preprocessing, text recognition, layout analysis, interface element detection, and video keyframe extraction. Engineering practice should unify the output format: element boxes, text, type, hierarchy, interactive state, confidence, and a unified coordinate system. For interface tasks, control grounding is more important than whole-image description. For document tasks, reading order and table structure are more critical than OCR alone. For video tasks, timestamps and event boundaries must be traceable. Perception results should retain an evidence chain to facilitate failure attribution.
5.2 Execution and Security Layer
The execution layer should manage capabilities through a tool registry. Each tool defines input schema, output schema, timeout, idempotency, permissions, and audit fields. High-risk actions require least privilege, sensitive data masking, human confirmation, and rollback mechanisms. The execution environment should use sandbox isolation and restrict network and file access. Idempotency keys should be set for repeated calls, and retry limits and backoff strategies should be set for failed calls. Tool return results should be standardized to avoid the model directly parsing uncontrolled long logs.
5.3 Observability and Cost Control
Production systems need to record the trajectory of each task: per-round observation summaries, model decisions, tool parameters, execution results, latency, and token consumption. Failure classification statistics can identify perception errors, reasoning errors, tool errors, and permission errors, enabling continuous optimization of prompts, tool descriptions, and action constraints. Object storage can be used to save screenshots and video clips, logging services and monitoring alerts can track success rates and latency, and caching can reduce repeated perception. For long videos and high-resolution images, layered summaries and on-demand zoom should be used to avoid feeding too many visual tokens at once.
6. Evaluation Methods: Examine the Closed Loop in Layers, Not Just Single Points
Evaluation should be layered. The perception layer examines element grounding, text recognition, layout order, and temporal localization. The reasoning layer examines multi-step planning, tool selection, and parameter correctness. The execution layer examines task success rate, step efficiency, failure recovery rate, and side-effect control. The end-to-end layer examines real business completion quality. Interface-to-code tasks can be evaluated from multiple dimensions such as structural similarity, rendering differences, interaction usability, and human ratings. Mobile operation tasks can be evaluated by success rate, average number of steps, timeout rate, and human takeover rate. Document tasks can be evaluated by field accuracy and evidence consistency. Video tasks can be evaluated by temporal localization error and event coverage.
At the same time, text-only regression sets and hallucination detection sets should be established. The advantage of native multimodal agents is not a particular benchmark score, but verifiable action capability under complex observations. Evaluation sets should cover difficult scenarios such as low resolution, occlusion, dynamic loading, pop-ups, abnormal pages, long documents, and long videos, while retaining samples for manual review.
7. Deployment Recommendations
For deployment, it is advisable to start with a minimal closed loop: first choose a scenario with clear observations, a constrained action space, and automatic verification; connect the perception, decision, execution, feedback, and logging chain; then gradually expand tools and modalities. A unified state structure is the core, avoiding each module maintaining its own context. Tool descriptions should be strict, parameters should be validatable, and failures should be recoverable. Observations should be compressed but retain evidence, and key regions can be zoomed in again. Permissions and auditing must be established upfront, and high-risk operations should default to human confirmation. Evaluation should not chase only single-point scores, but focus on task success rate, recovery capability, and preservation of text-only capability. When deployed on general cloud infrastructure, object storage, logging services, monitoring alerts, and other general-purpose capabilities can be combined to build observability and elastic scaling. In compliant environments such as Baidu AI Cloud, engineering deployment should also follow the principles of least privilege and data isolation.
Overall, the technical value of native multimodal agents lies in turning vision from a post-hoc patch into a native capability and incorporating perception, reasoning, execution, and feedback into the same closed loop. Architecturally, they emphasize unified representation and a state machine. In training, they emphasize multi-stage joint optimization and execution feedback. In engineering, they emphasize toolchain standardization, security constraints, and observability. Only when these three work together can an agent achieve stable, interpretable, and continuously optimizable capabilities in real-world visual tasks.
