Native Multimodal Agents: Architecture, Training, and Evaluation Practices

    This article explains how native multimodal agents differ from pipelines that bolt vision onto a language model. It covers a five-layer architecture that links perception, reasoning, and action, the training stages needed to teach a model to see, reason, and act, common failure modes, engineering practices for developers, and metrics for evaluating end-to-end task completion.

    1. From “Visual Add-ons” to Native Multimodality

    Traditional multimodal systems usually use a pipeline architecture. A separate vision module extracts features from images or video, converts those features into text descriptions, and passes the text to a language model for reasoning. This approach is inexpensive to build and makes it easy to reuse existing models. However, it has a basic problem: visual perception, language reasoning, and external execution often live in different modules, and information is lost during repeated conversions.

    Consider a web page screenshot. The system must do more than read the text. It also needs to determine where the buttons are, how the page is structured, how components relate to each other, and what action to take next. If the vision module only outputs “there is a login button on the page” and discards the button’s coordinates, size, color, context, and interaction state, the agent will struggle to click, fill in fields, and verify results accurately.

    Native multimodality does not simply mean adding a visual encoder. A model with native multimodality is designed from the start to have these capabilities:

    1. Receive multiple types of information in a unified way: Process text, images, video frames, interface states, and tool outputs together.
    2. Fuse modalities within the same reasoning process: Visual content is no longer just an attached note. It directly informs planning, judgment, and decisions.
    3. Turn reasoning into actions: Based on the current environment, generate clicks, text input, scrolling, API calls, or code execution.
    4. Use feedback to correct course: Execution results return to the context, so the model can judge whether an action succeeded and adjust its strategy.

    The core of a native multimodal agent, therefore, is not “being able to see.” It is being able to understand, reason clearly, act, and verify.

    2. Overall Architecture: A Closed Loop of Perception, Reasoning, and Execution

    A complete multimodal agent can be divided into five layers.

    1. Multimodal Input Layer

    The input layer receives different types of data, including:

    • Static images, such as screenshots, flowcharts, design mockups, and scanned documents;
    • Videos or consecutive frames, which capture state changes and operation sequences;
    • Text instructions, structured data, and conversation history;
    • Page states, operation results, and error messages returned by the external environment.

    The key requirement is not only to support more input formats. The system must also establish a unified representation of time and space. Objects in a video change over time. Controls in an interface have explicit coordinates. Titles and tables in a document have hierarchical relationships. If this information is over-compressed before it reaches the model, later reasoning loses the evidence it depends on.

    2. Visual Representation Layer

    The visual representation layer usually handles image encoding, region understanding, text recognition, and spatial relationship modeling. Unlike ordinary image classification, an agent needs to know where objects are, how they relate to one another, and which regions can be operated on.

    For an interface screenshot, the model should retain the following information where possible:

    1. Object type: button, input box, menu, list, icon
    2. Spatial information: coordinates, size, relative position
    3. Semantic information: text content, state, availability
    4. Relationship information: region it belongs to, neighboring elements
    5. Interaction information: clickable, editable, scrollable

    Visual encoding therefore cannot focus only on global semantics. It also needs fine-grained regional detail. Practical systems often use multi-scale representations. Low-resolution features capture the overall layout, while high-resolution features recognize small text, icons, and control boundaries.

    3. Cross-Modal Reasoning Layer

    The cross-modal reasoning layer is the core of the system. It maps visual features and text context into a shared reasoning space and performs task decomposition, conditional judgment, and step planning.

    Take the task “find specified information on a web page and fill in a form.” The model must complete these steps in order:

    1. Determine whether the current page has finished loading;
    2. Locate the search entry point or the target form;
    3. Identify the input fields and the submit button;
    4. Generate the content to enter, based on the user’s requirements;
    5. After acting, check whether the page changed as expected;
    6. If the action fails, analyze the cause and choose to retry, go back, or take a different path.

    This is not a single-turn question-answering task. It is a decision process that keeps updating state. The model must distinguish the current observation, the action history, and the next plan. Otherwise, it may repeat actions or keep using outdated information after the state has changed.

    4. Tool and Environment Interaction Layer

    The practical value of a multimodal agent usually appears in the execution layer. The model can use tools to operate pages, run code, process files, retrieve information, or call business systems.

    A tool call should not be treated as simple function generation. It should include complete action constraints:

    1. {
    2. "action": "click",
    3. "target": {
    4. "x": 420,
    5. "y": 268
    6. },
    7. "reason": "The target button is at the bottom of the form",
    8. "expected_result": "The page moves to the result list"
    9. }

    After execution, the environment returns a new screenshot, a text state, or a structured result. The model then uses the new state to judge whether the action succeeded. The process can be represented as follows:

    1. Observe the environment
    2. -> Identify key objects
    3. -> Generate candidate actions
    4. -> Check action risk
    5. -> Execute the action
    6. -> Get the new state
    7. -> Evaluate the result
    8. -> Continue or revise the plan

    Without this feedback loop, the system can only guess whether an action worked. In complex pages, dynamic content, or abnormal states, it can easily lose control.

    3. Training Methods: Teaching the Model to See, Think, and Act

    Native multimodal capabilities usually cannot come from a single dataset. They require several training stages.

    1. Vision-Language Alignment

    The first stage aims to teach the model how visual content corresponds to text. Training samples can include image captions, visual question answering, region recognition, text reading, and layout understanding.

    Simple image captioning is not enough to support agent capabilities. More useful data should also include:

    • Precise references to key regions;
    • Descriptions of spatial relationships;
    • Judgments about interface states;
    • Structured understanding of tables, flowcharts, and complex documents;
    • Explanations of state changes across consecutive frames.

    2. Multimodal Instruction Training

    After basic alignment, the model needs to learn to solve practical problems by following instructions. Examples include answering questions about a screenshot, extracting fields from a document, comparing interface states, and explaining the cause of an error.

    This stage focuses on generalizing across tasks. The data must not cover only fixed templates. Otherwise, the model may memorize page styles without being able to handle new layouts or new ways of phrasing requests.

    3. Trajectory and Process Training

    Agent tasks often involve multiple steps. Training data should therefore record the complete trajectory, not just the final answer:

    1. Initial observation
    2. -> Goal identification
    3. -> Step planning
    4. -> First action
    5. -> Environment feedback
    6. -> State re-evaluation
    7. -> Subsequent actions
    8. -> Final result verification

    Process data helps the model learn to confirm intermediate steps. It also helps the model understand what separates failed actions from successful ones.

    4. Result Verification and Preference Optimization

    For multimodal tasks, correctness cannot be judged by the text answer alone. A code generation task needs to verify whether the page meets the design requirements. An interface operation task needs to confirm that the target control was actually triggered. A document processing task needs to check that all fields are complete.

    Training should therefore include executable validators, rule-based checkers, or environment feedback signals. When several approaches can complete a task, the system can prefer the path with fewer steps, lower risk, or clearer explanations.

    4. Why Visual Agents Make Mistakes

    1. Loss of Detail

    When an image is compressed into a small number of visual tokens, small fonts, borders, and icon details may disappear. Remedies include dynamic resolution, region cropping, multi-scale encoding, and zooming on demand.

    2. Inaccurate Spatial Localization

    A model may understand that it should click the submit button but fail to determine the button’s center precisely. In engineering terms, semantic recognition and coordinate prediction should be validated separately. The system should also check that the target lies in an interactive area before execution.

    3. Delayed State Understanding

    If a page has already navigated but the model still acts on an old screenshot, it may click twice or enter the wrong input. Key actions should be followed by a fresh environment state check. Where needed, the system can add page change detection.

    4. Error Accumulation in Long Plans

    If the model plans a dozen or more steps at once, an error in any step affects every step after it. A safer approach is short-horizon planning. Generate a few steps, execute them, verify the results, and only then plan the next steps.

    5. Treating Uncertain Content as Fact

    When a screenshot is blurry, content is missing, or the page is still loading, the model may fill in gaps on its own. The system should output an explicit confidence score. Low-confidence cases should trigger re-observation, region zooming, or human confirmation.

    5. Engineering Recommendations for Developers

    1. Structure the Context

    Do not stack every screenshot and historical message into the context without distinction. Organize the information with a structure such as this:

    1. Task goal
    2. Current environment state
    3. Identified objects
    4. Executed actions
    5. Action results
    6. Unresolved issues
    7. Candidate next steps

    This structure reduces redundancy and makes debugging and auditing easier.

    2. Set Action Safety Boundaries

    High-risk operations, such as deleting data, submitting forms, transferring funds, or publishing content, should require secondary confirmation, permission checks, and rollback mechanisms. The model can propose actions, but a rules engine should constrain final execution.

    3. Decouple Observation from Execution

    The observation module identifies pages and states. The planning module generates strategies. The execution module calls tools. Keeping these three modules clearly separated allows each to be replaced and tested independently, which lowers maintenance costs.

    4. Keep Complete Run Logs

    Logs should record at least an input summary, the model’s judgments, tool parameters, environment feedback, error types, and retry counts. For visual tasks, the system should also save key screenshots or cropped regions. These records help determine whether a failure came from recognition, planning, or execution.

    6. How to Evaluate Native Multimodal Agents

    Evaluation should cover more than question-answering accuracy. It should also include the following dimensions:

    • Perception accuracy: Whether the system correctly recognizes text, objects, positions, and states;
    • Reasoning correctness: Whether the system draws reasonable conclusions from visual evidence;
    • Execution success rate: Whether the system actually completes the target task;
    • Step efficiency: The number of actions and the time required to finish a task;
    • Recovery ability: Whether the system can self-correct after loading failures, wrong actions, or page changes;
    • Safety: Whether the system ever performs actions outside its authorized scope;
    • Generalization: Whether performance stays stable across different layouts, themes, and device sizes.

    Answer correctness and task completion must be evaluated separately. A system may describe a page accurately and still fail to complete a form, navigate an interface, or fix code. Meaningful evaluation focuses on end-to-end outcomes and also records process metrics.

    7. Summary

    Building native multimodal agents is a systems engineering effort. The model needs visual understanding. The training data needs to include real operation processes. The toolchain and environment feedback must support continuous decision-making.

    The key principles from this technical path are:

    1. Unify the representation of images, video, text, and environment state;
    2. Preserve the spatial, temporal, and interactive properties of visual information;
    3. Train the model on complete perception, reasoning, and execution trajectories;
    4. Use feedback and verification mechanisms to reduce hallucinations and wrong actions;
    5. Measure actual capability with end-to-end task results;
    6. Set clear safety boundaries for high-risk actions.

    Multimodal systems become capable of real workflow automation only when vision does more than add input to a language model. Vision must directly take part in planning, execution, and verification.