Native Multimodal Visual Agents: Technical Architecture for Perception, Reasoning, and Execution

    This article examines the technical architecture of native multimodal visual agents. It explains why visual perception must be integrated into the full training process, how image, video, and text representations can be unified, and why structured tasks help improve spatial reasoning. It also covers layered training, the combination of supervised fine-tuning and reinforcement learning, methods for reducing visual hallucinations, safe tool calling, and evaluation systems that assess perception, reasoning, execution, and recovery.

    Introduction

    In real business environments, much of the critical data does not exist purely as text. Web layouts, application interfaces, design mockups, scanned documents, flowcharts, video frames, and dashboards all require a model to understand them before analysis, decision-making, or action can take place.

    A traditional approach is to attach a visual encoder outside the language model. The vision module converts images into vectors, and the language model then generates text or actions based on those vectors. This approach is relatively low-cost to implement, but it also has clear limitations.

    The vision module and the language reasoning module are often loosely coupled. The model may be able to describe an image but still fail to understand the spatial relationships between elements. It may recognize a button but be unable to determine which button will trigger the target action. It may read text but struggle to connect the text, layout, and interaction state into a single coherent understanding.

    A more reasonable direction is to treat vision as a native component of the model and build a complete loop of perception, reasoning, execution, and feedback. The key is not simply adding image input. It is enabling the model to process visual information through its architecture, training data, optimization objectives, and tool-use mechanisms.

    1. Visual Perception Is Not a One-Time Preprocessing Step

    In many multimodal systems, visual processing is placed at the beginning of the inference pipeline. Images are first passed through a visual encoder and converted into features, and then the language model completes the downstream task. This design implicitly assumes that once feature extraction is complete at the front end, the backend can independently solve the reasoning problem.

    But in practice, visual understanding continues to affect higher-level model capabilities.

    An error in locating an element can cause subsequent actions to fail completely. A misinterpretation of table-column relationships can invalidate an entire analysis. A misunderstanding of the temporal order of video keyframes can also lead to incorrect causal inference.

    Therefore, visual capability should span multiple stages of model training:

    1. Strengthen structural understanding during pre-training.
      In addition to conventional image-captioning and image-text matching tasks, the model can be trained to convert images into structured representations. For example, it can learn to convert an interface or diagram into markup language, a layout tree, a component hierarchy, or a vector description. Such tasks require the model to recognize objects, positions, sizes, hierarchies, and relationships, helping it build more stable spatial representations.

    2. Correct perception errors during supervised fine-tuning.
      Fine-grained annotations can be built for web pages, mobile applications, and documents, including element categories, bounding boxes, text content, interactive attributes, and relationships between elements. When the model identifies the wrong object, misses text, or confuses adjacent controls, targeted samples can be used to correct these errors.

    3. Optimize real-world decision-making during reinforcement learning.
      In interactive environments, perception quality can be indirectly evaluated through task outcomes. For example, clicking the correct control, filling in the right field, or completing page navigation indicates that the model’s visual judgment is broadly consistent with its action choices. If the outcome is wrong, the team must analyze whether the error came from recognition, localization, planning, or execution.

    This shows that the vision module is not merely an independent “input adapter.” It is an important part of the infrastructure that determines the upper limit of reasoning. Perception quality must be continuously evaluated and optimized throughout the entire training cycle.

    2. Unifying Image, Video, and Text Representations

    A native multimodal system needs to process information in different forms, at different time scales, and with different levels of semantic granularity.

    Images usually emphasize spatial relationships. Video additionally introduces temporal relationships. Text is better suited to expressing abstract concepts and operational intent. If there is no unified representation across these three modalities, the model may be able to read an image but fail to reason about it, or it may understand text without being able to map that understanding to the visual interface.

    A common approach is to build a multi-level representation system:

    • Pixel or visual feature layer: preserves basic information such as color, texture, edges, and local shape.
    • Object layer: represents entities such as buttons, input fields, tables, people, icons, and document regions.
    • Layout layer: represents hierarchical relationships, relative positions, alignment, and regional membership.
    • Semantic layer: expresses page intent, task goals, state changes, and business meaning.
    • Action layer: connects visual objects with operations such as clicking, typing, dragging, scrolling, and selecting.

    For video, inter-frame state changes also need to be included. The model should not only know what appears in a given frame. It should also determine which elements changed, when the change occurred, and whether the change was caused by a specific action. In engineering practice, methods such as keyframe sampling, local high-resolution cropping, and state summarization can be used to balance computational cost with temporal accuracy.

    3. Using Structured Tasks to Improve Spatial Reasoning

    A simple image-captioning task usually only requires the model to answer “What is in the image?” However, a visual agent needs to solve questions such as “Where are they?”, “What is their relationship?”, and “What should be done next?”

    Therefore, training data should be expanded from simple descriptions to structured tasks. Examples include:

    • Converting an interface into a component tree.
    • Generating a layout description from a screenshot.
    • Determining the relative position of two elements.
    • Predicting which region a piece of text belongs to.
    • Inferring state changes from screenshots taken before and after an operation.
    • Generating an initial page structure from a design mockup.
    • Mapping visual objects to executable actions.

    For example, when converting an interface into a structural description, the model needs to understand the page hierarchy, component types, size ratios, and content relationships at the same time. Even if the final goal is not to generate structured code, this type of training can improve the model’s spatial organization capability and make it more reliable in tasks such as chart analysis, document reading, and scientific reasoning.

    It is worth noting that structured tasks do not mean that every output must be code. What matters more is whether the model can produce verifiable intermediate representations. Compared with free-form text, intermediate representations are easier to inspect, correct, and trace, and they are also better suited for connection to executors.

    4. Layered Training Is Better Than Direct End-to-End Optimization

    Letting a model directly generate a complete action trajectory from a screenshot may seem like end-to-end learning in theory. In practice, however, it often faces problems such as high data costs, difficulty in locating errors, and sparse rewards.

    A more practical approach is layered training, which decomposes a complex agent into several capability layers.

    1. Element Perception Layer

    This layer is responsible for recognizing objects in a page, including text, icons, buttons, input fields, lists, dialogs, and scrollable regions. The output may include category, text, position, attributes, and confidence.

    2. Target Localization Layer

    This layer locates the target element based on user intent. For example, if a user asks to “open settings,” the model needs to identify the correct object among multiple icons and text items. It is not enough to merely detect that a settings-related word exists on the page.

    3. Single-Step Action Layer

    This layer converts a target into a specific action, such as clicking, typing, swiping, going back, or selecting. Action prediction should consider the current position, page state, control availability, and operational risk.

    4. Trajectory Planning Layer

    This layer generates multi-step plans, determines whether a task is complete, and re-plans after the page state changes. It needs to handle situations such as page loading, popup interference, failed operations, and mismatches between the target state and actual state.

    5. Result Verification Layer

    After an action is executed, the system should determine whether the operation succeeded based on screenshots, text changes, page structure, or business outcomes. If the result does not match expectations, the model should roll back, retry, or change strategy rather than blindly continuing.

    The biggest value of a layered design is that it decomposes “overall failure” into diagnosable problems. For example, if target localization is correct but the click does not work, the issue may be coordinate mapping or page state. If the action is correct but the task is still incomplete, the planning layer may have missed subsequent steps. This makes it possible to construct separate datasets and optimization objectives, improving engineering efficiency.

    5. Combining Supervised Fine-Tuning and Reinforcement Learning

    Supervised fine-tuning is suitable for teaching the model “how it should act.” Reinforcement learning is suitable for optimizing “how to do better in real environments.”

    Supervised data can cover:

    • The mapping between images and page structures.
    • The mapping between user instructions and target elements.
    • Single-step operation demonstrations.
    • Multi-step task trajectories.
    • Common errors and their corrections.
    • Explanations of operation results and causes.

    However, demonstration data alone is not enough. Real environments often contain unseen layouts, dynamic popups, network latency, abnormal pages, and unpredictable state changes. Reinforcement learning can provide feedback based on whether the task is completed, whether operations are safe, and whether steps are redundant.

    The reward function should not focus only on final success. It can also include process constraints, such as:

    • Whether the correct target was selected.
    • Whether dangerous operations were avoided.
    • Whether too many invalid steps were used.
    • Whether the model re-observed the page after it changed.
    • Whether the model could recover from failure.
    • Whether the explanation was consistent with the actual state.

    In this way, the model does not merely pursue “task completion.” It gradually learns execution strategies that are more robust, controllable, and efficient.

    6. Engineering Methods for Reducing Visual Hallucinations

    Visual hallucinations often appear when a model describes elements that do not exist, misreads text, confuses adjacent controls, or gives overly confident conclusions when uncertainty remains. Solving this problem cannot rely only on larger models. It also requires constraints at the system level.

    First, the model should output confidence or uncertainty markers. When the target region has insufficient resolution, the text is blurry, or multiple elements are highly similar, the model should proactively request zooming, re-capturing, or additional context.

    Second, an “observe–act–observe again” loop should be used instead of generating a complete operation sequence all at once. After each action, the system should obtain the current page state again and confirm whether the interface has changed as expected.

    Third, free-text reasoning should be combined with structured validation. For example, when the model plans to click a button, the executor can check whether the coordinates fall within an interactive region. When the model prepares to submit a form, the system can verify whether required fields are complete.

    Finally, specialized counterexamples are needed. These may include similar icons, misaligned text, occluded elements, popup overlays, and targets that disappear after scrolling. The closer these counterexamples are to real environments, the better the model can learn to recognize uncertainty instead of guessing based on experience.

    7. Tool Calling and Execution Safety

    The value of a multimodal agent is ultimately reflected in its execution capability. A visual model usually needs to work with browser controllers, mobile device operators, document parsers, code execution environments, or business APIs.

    Tool design should follow three principles:

    1. Clear interface semantics.
      Tool parameters should clearly distinguish coordinates, text, target elements, and action types. This prevents the model from having to guess parameter meanings based only on natural language.

    2. Observable operation results.
      Each call should return a screenshot, page state, error reason, or structured result so that the model can determine whether the action took effect.

    3. High-risk actions can be intercepted.
      Irreversible operations such as deletion, submission, transfers, and publishing should include confirmation mechanisms, permission checks, and audit records. They should not rely entirely on model judgment.

    In production environments, complete trajectories should also be recorded, including input screens, model decisions, tool parameters, execution results, and exception information. These data are useful not only for troubleshooting but also for future training.

    8. Evaluation Should Not Focus Only on Final Success Rate

    Evaluation of visual agents should cover four aspects: perception, reasoning, execution, and recovery.

    Perception metrics may include element recognition accuracy, text reading accuracy, localization error, and layout relationship judgment accuracy. Reasoning metrics may include target selection accuracy, task planning rationality, and state judgment accuracy. Execution metrics may include action success rate, average number of steps, and the proportion of invalid operations. Recovery metrics focus on whether the model can re-plan when the page changes, an action fails, or information is missing.

    In addition, teams should pay attention to whether there is a significant imbalance between the model’s text capabilities and visual capabilities. An ideal system should not only be able to operate interfaces or describe images. It should be able to switch flexibly among text, vision, and tools, forming a complete problem-solving chain.

    Conclusion

    The core of a native multimodal visual agent is not simply connecting image input to a language model. It is about reconstructing how the model understands the world and takes action. Visual perception needs to span the entire training process. Images, videos, and text need to be represented in a unified way. Complex tasks need to be optimized through layered capabilities. Tool calling needs feedback, validation, and safety boundaries.

    From an engineering perspective, the most effective path is usually not to seek a one-time end-to-end solution for all problems. Instead, teams should first build stable perception and structured representations, then gradually connect planning, execution, and feedback. Only when the model can observe accurately, reason properly, act carefully, and recover after failure can multimodal capability truly move from “understanding content” to “completing tasks.”