Native Multimodal Agents: Architecture, Training, and Engineering Practices

    This article examines the architecture, training, and engineering practices of native multimodal agents. Unlike systems that bolt a vision module onto a language model, native multimodal agents are designed so that vision, language, reasoning, and execution share context from the outset. The article covers the need for native multimodal capability; a layered system architecture spanning multi-scale visual encoding, vision-language alignment, unified reasoning, and tool execution; a staged training pipeline from cross-modal pretraining to multimodal instruction tuning, agent trajectory training, and outcome-oriented reinforcement optimization; key engineering challenges such as resolution and context trade-offs, coordinate grounding, long-horizon memory, and hallucination; typical applications including Design2Code, mobile automation, document and table understanding, and video process analysis; evaluation dimensions; and practical deployment guidelines.

    In real software and business environments, information does not exist only in text. Web page layouts, mobile interfaces, design files, flowcharts, scanned documents, operation logs, and video frames may all contain critical clues needed to complete a task.

    Traditional language models are good at understanding structured text, generating code, and performing logical reasoning. However, when faced with screenshots, complex pages, or continuous video, they often need an additional vision module to convert information. The approach of “recognize first, transcribe second, reason last” can be deployed quickly, but it easily causes information loss: the vision module may miss layout relationships, OCR may destroy spatial structure, and the language model may struggle to determine which visual details actually affect task outcomes.

    Therefore, the core goal of a native multimodal agent is not simply to add an image input interface to a language model. Instead, vision, language, reasoning, and execution should work together from the beginning of system design, forming a complete perception–decision–action loop.

    1. Why Native Multimodal Capability Is Needed

    1.1 Visual Information Contains More Than Text

    Take a web page screenshot as an example. The effective information on the page includes at least four layers:

    • Text content, such as titles, button labels, and prompts;
    • Spatial relationships, such as a button below a form or an error message near an input field;
    • Visual hierarchy, such as color, font size, and contrast indicating the importance of elements;
    • Interaction states, such as whether a control is selected, whether a modal is open, or whether the page is loading.

    If the system only converts an image into a block of text and then passes it to a language model, spatial relationships, visual hierarchy, and interaction states are often compressed or even lost. For tasks such as interface operation, design restoration, and visual quality inspection, this information often determines the final result.

    1.2 Multimodal Tasks Are Continuous

    Real tasks are usually not one-shot question answering but continuous operations. For example, a user may ask an agent to open a page, fill out a form, submit a request, and continue processing based on page feedback. In this case, the system needs to repeatedly complete the following loop:

    1. Observe the current screen;
    2. Determine the page state;
    3. Choose the next action;
    4. Execute a click, input, scroll, or back action;
    5. Check whether the action took effect;
    6. Adjust the plan based on the new screen.

    This requires the model not only to “see” but also to understand the causal relationship between visual changes and actions. Image recognition capability alone is not enough to support long-chain tasks.

    1.3 Perception and Reasoning Must Share Context

    If the visual encoder, text recognition module, and language model process information separately, the system usually needs to pass content through intermediate text or structured results. The more complex the intermediate representation, the more obvious the error accumulation.

    Native multimodal approaches emphasize a unified context. Image regions, text fragments, page structure, and action history can enter the reasoning process together, allowing the model to combine “what it sees,” “what it has done,” and “what goal it needs to achieve next” when making decisions.

    2. Basic Architecture of a Native Multimodal System

    A complete visual agent usually consists of the following layers.

    2.1 Multi-Scale Visual Encoding Layer

    The visual encoder converts images or video into visual representations that the model can process. For high-resolution screenshots, using only low-resolution input can easily lose small fonts, icons, and fine-grained controls. Therefore, multi-scale processing strategies are often needed:

    • Low-resolution views to understand the overall layout;
    • Local high-resolution views to recognize details;
    • Region cropping to focus on content relevant to the current operation;
    • Continuous frame sampling to capture state changes.

    Multi-scale mechanisms do not mean blindly increasing input size. Too large an input increases computational cost, congests context, and adds irrelevant information. A more reasonable approach is to first build a global overview and then dynamically select local regions based on the task goal.

    2.2 Vision-Language Alignment Layer

    Visual features cannot be directly equated with language tokens. The system needs to learn correspondences between visual regions and language concepts, for example:

    • A certain region corresponds to a “submit button”;
    • A certain color change indicates “validation failed”;
    • A certain icon indicates “expand menu”;
    • A certain layout relationship indicates “input field bound to label.”

    The role of the alignment layer is to establish a stable mapping between visual representations and language reasoning. Good alignment is not only about recognizing objects in an image; it should also support the expression of positions, relationships, states, and operation intent.

    2.3 Unified Reasoning Layer

    The unified reasoning layer integrates text, images, video frames, action history, and external tool feedback. It needs to handle three types of problems:

    • Understanding the current environment;
    • Formulating an action plan that meets the goal;
    • Dynamically revising the plan based on execution results.

    For complex tasks, the model should not output only a single action. It should form inspectable intermediate states. For example, it can first determine whether the page has finished loading, then locate the target region, and finally execute the action and verify the result. This reduces the probability that a single error causes the entire workflow to fail.

    2.4 Tool Execution Layer

    Visual agents often need to call browsers, mobile environments, code execution environments, or business systems. The tool execution layer should convert model outputs into controlled actions and return structured feedback.

    Common actions include:

    • Clicking a specified position;
    • Entering or modifying text;
    • Scrolling the page;
    • Switching windows or tabs;
    • Reading page state;
    • Executing code and obtaining results;
    • Saving screenshots, logs, and intermediate artifacts.

    Engineering practice must distinguish between “model intent” and “actual execution result.” The model believing that a click succeeded does not mean the interface actually changed. Therefore, after each action, the environment should be observed again, and the operation should be verified through state differences.

    3. Training System: From Understanding to Doing It Right

    Native multimodal capability usually requires staged training rather than directly using final task data for end-to-end optimization.

    3.1 Cross-Modal Pretraining

    The first stage focuses on building basic associations between vision and language. Training data can include images and descriptions, image-text question answering, page screenshots and structural descriptions, document images and text content, and more.

    This stage mainly addresses three questions:

    • Can the model recognize entities and text in images?
    • Can the model understand relationships between regions?
    • Can the model convert visual information into representable semantic representations for reasoning?

    Data quality matters more than quantity. Simply increasing the number of images does not guarantee that the model understands complex layouts. Training samples should cover different resolutions, fonts, page structures, document types, and visual noise.

    3.2 Multimodal Instruction Training

    After basic alignment is complete, instruction data is needed to teach the model to handle visual tasks according to user goals. Sample forms can include:

    • Explaining page issues based on screenshots;
    • Generating interface structures from design files;
    • Analyzing fault causes based on error prompts;
    • Comparing differences across multiple images;
    • Summarizing operation processes from video frames;
    • Deciding the next action based on visual observation.

    At this stage, special attention should be paid to the verifiability of outputs. For tasks that require execution, training data should include not only final answers but also stages such as observation, judgment, action, and verification.

    3.3 Agent Trajectory Training

    Single-turn question answering cannot cover the complex situations in real interaction. Agent training needs to introduce complete trajectories, including:

    • Initial task goal;
    • Initial environment state;
    • Observation results at each step;
    • Actions taken by the model;
    • Environment changes produced by actions;
    • Reasons for success or failure.

    Through these trajectories, the model can learn how to handle page changes, unexpected modals, input errors, invalid actions, and other situations. Compared with learning only “correct answers,” trajectory data is better at improving the model’s error correction and recovery capabilities.

    3.4 Outcome-Oriented Reinforcement Optimization

    In interface operation, code generation, and process automation, the final result is usually more important than whether the language expression is elegant. Therefore, reward signals can be designed based on task completion rate, page state, test results, and execution safety.

    However, reward design must avoid short-sighted behavior. For example, the model may obtain partial rewards by skipping verification, repeatedly clicking, or generating superficially correct results. A more robust evaluation should consider:

    • Whether the goal was completed;
    • Whether operational constraints were followed;
    • Whether side effects were produced;
    • Whether failure reasons can be explained;
    • Whether the task can be completed within a limited number of steps.

    4. Key Challenges in Visual Agents

    4.1 Balancing High Resolution and Context Cost

    Page screenshots are usually wider and longer than ordinary images and contain many small elements. Direct scaling loses details, while inputting the original image brings higher computational cost.

    Practical systems can use a combination of “global thumbnail + local zoomed view.” The model first uses the global image to identify candidate regions, then performs fine-grained analysis on relevant regions. For long pages, the page can be divided by semantic regions rather than mechanically cropped at fixed heights.

    4.2 Coordinate Understanding and Action Grounding

    The model being able to recognize a target element does not mean it can click it accurately. Interface operations also involve coordinate mapping, scaling ratios, scroll offsets, and device resolution.

    Therefore, the execution layer needs to validate the target position given by the model, for example by determining whether the coordinate is within an interactive region, whether it is obscured by other elements, and whether the coordinate is still valid after the page scrolls. For high-risk operations, a secondary confirmation mechanism should also be added.

    4.3 Memory Management in Long-Horizon Tasks

    Multi-step tasks generate large numbers of screenshots, actions, and feedback. If all historical information is retained directly, the context can become too long; if it is compressed excessively, key states may be lost.

    A practical approach is to maintain layered memory:

    • Short-term memory: save screens and actions from the most recent steps;
    • Task memory: save goals, completed steps, and pending items;
    • Environment memory: save page structure, account state, and known constraints;
    • Result memory: save errors, fixes, and final verification results.

    This controls context size while allowing the model to maintain task continuity.

    4.4 Hallucination and Uncertainty

    Visual models may mistake similar icons for target controls, or may give overly confident judgments when images are blurry. For automated systems, the most dangerous situation is not “failure to recognize” but “continuing to execute after incorrect recognition.”

    Engineering practice should allow the model to express uncertainty and adopt conservative strategies when confidence is insufficient, such as requesting a larger region, taking a new screenshot, checking page text, or pausing for human confirmation.

    5. Typical Application Scenarios

    5.1 Design2Code

    The system can extract layout, colors, font hierarchy, and component relationships from design images and then generate page code. The real difficulty is not only recognizing elements but also:

    • Restoring spacing and alignment relationships;
    • Distinguishing decorative elements from interactive elements;
    • Inferring adaptation rules for different screen sizes;
    • Generating maintainable code rather than one-off stitching.

    Therefore, evaluation results should not rely only on static image similarity. They should also consider page structure, interaction behavior, and multi-device adaptability.

    5.2 Mobile Automation

    Mobile tasks usually have strong state dependencies. A button may be in different states due to permissions, network conditions, or preconditions. The agent needs to determine the next step based on the current screen and action history rather than relying on fixed-coordinate scripts.

    5.3 Document and Table Understanding

    For scanned documents, complex tables, and reports with charts, visual understanding can supplement the shortcomings of traditional text extraction. The system should not only recognize text but also preserve heading hierarchy, table row and column relationships, footnote attribution, and image-text correspondences.

    5.4 Video Process Analysis

    The key to video tasks is understanding changes rather than describing every frame. The system needs to identify action start points, state transitions, and abnormal events, and reduce processing cost through keyframe sampling, temporal interval aggregation, and event-level summaries.

    6. How to Design a Reliable Evaluation System

    Multimodal agents cannot be evaluated only by single-turn question answering accuracy. A more reasonable evaluation should cover the following dimensions:

    1. Perception accuracy: whether the target object, text, and spatial relationships are recognized;
    2. Reasoning correctness: whether task constraints are understood and a reasonable plan is formulated;
    3. Execution success rate: whether actions actually change the environment state;
    4. Long-horizon stability: whether performance degrades rapidly as task steps increase;
    5. Exception recovery capability: whether the agent can recover from modals, network delays, or action failures;
    6. Safety: whether operations beyond the authorized scope are executed;
    7. Resource efficiency: the time, computation, and number of tool calls required to complete the task.

    Evaluation data should also distinguish between static images, dynamic pages, long-workflow tasks, and open environments, so that the model does not adapt only to fixed templates.

    7. Engineering Recommendations for Deployment

    When building visual agents, the following principles can be followed:

    • Define task boundaries first, then decide model scale;
    • Split observation, planning, execution, and verification into monitorable modules;
    • Retain screenshots, logs, and state changes for all critical actions;
    • Set permissions, approvals, and rollback mechanisms for high-risk operations;
    • Use caching to reduce repeated visual analysis;
    • Control the cost of high-resolution input through local cropping;
    • Prepare retry, fallback, and human takeover paths for failure scenarios;
    • Constrain model outputs to a structured action space and avoid driving the system directly with free-form text.

    From a system perspective, native multimodality does not mean “a larger vision model” or “more input images.” It means enabling the model to continuously understand the environment, formulate plans, execute actions, and verify results. Only when perception results truly participate in reasoning, reasoning results can reliably drive tools, and tool feedback can in turn correct model judgments will multimodal capability move from demo effects to usable productivity.

    Future visual agents will no longer be merely image question-answering tools. They will gradually become general interaction systems capable of understanding interfaces, operating software, analyzing documents, processing video, and completing complex workflows. The focus of technical competition will also shift from single-item recognition accuracy to overall closed-loop capability: seeing accurately, reasoning clearly, acting correctly, and stopping and recovering in time when errors occur.