Native Multimodal Agents: Training, Evaluation, and Safe Deployment

    A native multimodal agent goes beyond attaching a vision module to a text model—it brings perception, reasoning, and execution into one training, evaluation, and safety system. This article explains why real interface, document, and video tasks need a closed perception–reasoning–execution loop, how to distinguish a native multimodal agent from a bolt-on vision approach, and why perception quality must span the entire training cycle. It presents a layered method that decomposes a GUI agent into element perception, GUI localization, single-step action prediction, and trajectory-level planning, together with acceptance metrics, annotation criteria, and pseudocode for layered training and closed-loop execution. It then covers the safety boundaries the toolchain must include—sandboxed execution, least privilege, human confirmation, logging and auditing, sensitive-data masking, and retry limits—and closes with a Q&A checklist for engineering deployment.

    1. Real Tasks Require a “Perception–Reasoning–Execution” Loop

    Pure text models are good at understanding language, generating code, and reasoning logically. But in real business systems, much of the critical information is not text: web layouts, app interfaces, phone screenshots, design mockups, PDF tables, monitoring videos, charts, and widget states are all visual information. When a user asks to “replicate this page as code,” “find anomalies from a report chart,” or “complete an operation flow on a phone,” relying only on text capabilities—or compressing the visual result into a short description before handing it to a language model—often causes information loss.

    For example, the position of a “Save” button on screen, its hierarchy, whether it is obscured, whether it is currently clickable, and whether the icon is grayed out or highlighted—if these details are reduced to text like “there is a save button on the page,” later reasoning may lose critical evidence. The model appears to understand the interface, but is actually guessing based on a vague description, which makes misreads, misclicks, and hallucinations harder to avoid.

    Therefore, agents built for interfaces, documents, and videos should not be just “a text model plus a vision module,” but should gradually form a native multimodal agent system that integrates “perception–reasoning–execution.”

    2. Choosing Between a Native Multimodal Agent and a Bolt-On Vision Module

    To judge whether a system approaches a native multimodal agent, you cannot just look at whether a vision encoder is integrated; you must look at whether the visual signal truly participates in the training, reasoning, and execution loop. The key differences are summarized below.

    Dimension Bolt-on vision approach Native multimodal agent direction
    Perception–reasoning relationship The vision module first outputs a text description or vector, then hands it to the language model Raw screenshots, documents, and video frames enter a unified representation with text instructions, and the reasoning process can look back at details
    Training method Vision module and language model trained separately, high alignment cost Perception, localization, action, and trajectory enter the same training and evaluation system
    Execution capability The model can usually only generate suggestions or code Can invoke atomic operations such as click, input, scroll, and read, and retry based on feedback
    Diagnosability Hard to tell after a failure whether it misperceived, misspoke, or mis-executed Can evaluate by layer: whether the element was recognized, whether the target was localized, whether the action was correct, whether the trajectory succeeded
    Risk boundaries Easy to overlook permissions, auditing, and human confirmation Sandboxing, access control, and sensitive-action confirmation can be built into the design

    For selection: if the task is only “generate a description for an image,” a bolt-on vision approach may be sufficient; if the task requires operating interfaces, understanding complex documents, tracking video changes, and continuously correcting errors across multiple steps, the native multimodal agent route is more suitable.

    3. Why Perception Quality Must Span the Entire Training Cycle

    A common misconception is to treat perception as a one-time image-encoding problem in the early stage, assuming that once the vision encoder is trained, the rest can be handed to the reasoning model. But in engineering practice, perception quality continuously affects the ceiling of higher-level capabilities. Whether interface elements are fully recognized, whether the widget hierarchy is correct, whether button states can be determined, and whether document tables are structured correctly all affect subsequent localization and action decisions.

    It must be emphasized that the role of perception tasks cannot be simply understood as “training a module,” but should be seen as a cross-stage, cross-capability factor. In some controlled experiments, if the pretraining stage adds a large amount of “image-to-structured-representation” data—such as image-to-SVG, screenshot-to-widget-tree, and page-to-hierarchical-layout—the model may show gains in tasks like layout understanding, element-relationship judgment, and front-end code generation. However, whether it simultaneously improves STEM reasoning or other capabilities depends on data quality, task definition, evaluation-set design, and baseline methods, and cannot be treated as a universal conclusion.

    Similarly, adding “discriminative” data in the instruction fine-tuning or reinforcement learning stage—such as samples specifically labeled “button obscured,” “icon not clickable,” “text and icon semantically inconsistent,” or “widget state misread”—may reduce misreading rates in some interface-understanding evaluations. But the evaluation should clearly report: what dataset was used, how large the data scale was, whether it covers long-tail interfaces, which baseline model it was compared against, and whether the metric is accuracy, recall, or task success rate. Only by comparing under the same criteria can you determine whether an improvement actually holds.

    Common sources of perception error include:

    1. Screenshot compression or reduced resolution, causing small icons, thin borders, and low-contrast text to be lost;
    2. Widget occlusion, truncated scroll regions, and overlaying pop-ups making target elements invisible;
    3. The same icon having multiple states, such as enabled, disabled, loading, selected, and unselected;
    4. The accessibility tree being inconsistent with the visual interface, such as elements hidden in code but visible on screen;
    5. Merged cells, cross-page tables, and scanned PDFs causing structure-parsing errors.

    Therefore, perception quality cannot be “solved early and then ignored,” but should be treated as a continuous evaluation item, repeatedly validated across pretraining, supervised fine-tuning, reinforcement learning, and canary rollout.

    4. How Layered Optimization Replaces End-to-End, and How Each Layer Is Accepted

    An end-to-end agent sounds straightforward: output actions directly from pixels. But in real engineering, end-to-end training faces problems such as high environment cost, difficult trajectory annotation, unverifiable intermediate steps, sparse rewards, and difficulty attributing failures. A more robust approach is layered training and layered evaluation, decomposing a GUI agent into four layers: element perception, GUI localization, single-step action prediction, and trajectory-level planning.

    Layer Input Output Example minimum acceptance metric Annotation criteria Typical errors
    Element perception Screenshots, accessibility tree, document structure Widgets, text, icons, regions, states Widget recall, text-recognition accuracy, state-classification accuracy Bounding boxes, labels, visibility, enabled state Missing icons, misjudging button states
    GUI localization Screenshots, semantic instruction, perception results Target widget coordinates or region Click center-point hit rate, localization error, failure rate Target widget ID, bbox, center point Clicking a similar button, localizing to an invisible region
    Single-step action prediction Current state, history, available action set Next action and parameters Action accuracy, illegal-action rate, parameter correctness Action type, target, input content, timestamp Wrong clicks, empty input, wrong scroll direction
    Trajectory-level prediction Task goal, current state, trajectory history Multi-step plan, rollback, retry Task success rate, average steps, backtracking rate, termination correctness Step sequence, success conditions, rollback points Long-range planning drift, no correction after failure

    The value of layered evaluation lies in attributability. If a task fails, you can quickly determine whether the widget was not recognized at all, whether it was recognized but the coordinates were wrong, whether the single-step action was correct but the overall planning failed, or whether the execution feedback did not enter the next round of decisions. Without clear layered metrics, a team can only reach the vague conclusion “the task was not completed,” which makes continuous optimization difficult.

    For data-mixing ratios, low-level perception tasks can usually use larger-scale weakly labeled or structured-parsing data, such as screenshots with widget trees and documents with layout annotations; high-level trajectory data is scarcer and more expensive, and is better suited for fine-tuning critical scenarios. A common practice is to first build a foundation with low-cost perception data, then train single-step and multi-step decisions with a small sample of high-quality trajectory data, and finally align through reinforcement learning with environment feedback.

    5. Layered Training and Closed-Loop Execution: Pseudocode

    The following is pseudocode, used to illustrate how training and execution are organized; it cannot be run directly. Actual implementation needs to combine PyTorch, transformers, OpenCV, Playwright, Android ADB, PDF parsers, video frame-extraction tools, action executors, and evaluation environments.

    1. # Pseudocode: only illustrates layered training and closed-loop execution; not directly runnable.
    2. # Example dependencies: PyTorch, image/video encoder, VLM/LLM, action executor, environment simulator
    3. LAYERS = [
    4. "element_perception", # element perception
    5. "gui_grounding", # GUI grounding
    6. "single_step_action", # single-step action prediction
    7. "trajectory_planning", # trajectory-level prediction
    8. ]
    9. # Data format example (JSONL)
    10. # {
    11. # "image": "screenshots/task_001.png",
    12. # "instruction": "Click the save button in the top-right corner",
    13. # "perception": [{"label": "button", "text": "Save", "bbox": [x1, y1, x2, y2]}],
    14. # "action": {"type": "click", "target_bbox": [x1, y1, x2, y2], "value": ""},
    15. # "state": {"page": "editor", "dirty": true},
    16. # "label": "success"
    17. # }
    18. def build_sft_dataset(layer):
    19. # Return the supervised data for this layer
    20. # Perception layer: screenshots, widgets, states
    21. # Localization layer: instructions, target widgets
    22. # Action layer: current state, actions
    23. # Trajectory layer: full steps and rollback points
    24. return []
    25. def design_layered_reward(layer):
    26. # Reward examples; in practice they must be defined per the business
    27. # Perception layer: label match, state match
    28. # Localization layer: center-point distance, target hit
    29. # Action layer: correct action type, valid parameters, execution success
    30. # Trajectory layer: task completion, step efficiency, rollback reasonableness, safety penalty
    31. return lambda sample, prediction: 0.0
    32. for layer in LAYERS:
    33. sft_data = build_sft_dataset(layer)
    34. model.train_sft(sft_data, layer=layer)
    35. env = build_interactive_env(layer)
    36. reward = design_layered_reward(layer)
    37. model.train_rl(env, reward, init_from=model, layer=layer)

    The SFT-to-RL ratio has no fixed standard. One reference example: the element perception layer relies mainly on high-quality annotated SFT, with a small amount of RL for state discrimination; the GUI localization layer uses SFT as a base, then local alignment with click-success rewards; the single-step action layer can adopt a higher proportion of trajectory SFT, then introduce environment feedback; the trajectory planning layer, due to scarce trajectory data, usually first learns successful trajectories with SFT, then optimizes rollback and correction with a small amount of RL. Whether there is improvement should be validated on an independent test set, not assumed that RL is always better than SFT.

    In the closed-loop execution stage, the agent needs to convert model decisions into structured actions and feed the execution results back into the perception layer. The minimum interface conventions can include:

    1. class Observation:
    2. screenshot: bytes
    3. instruction: str
    4. a11y_tree: dict
    5. page_url: str
    6. timestamp: float
    7. class Action:
    8. type: str # click / input / scroll / back / read / submit
    9. target_bbox: list # [x1, y1, x2, y2]
    10. value: str
    11. reason: str
    12. risk_level: str # low / medium / high
    13. def agent_step(observation, model, tools, history, retry_limit=3):
    14. # 1. Perception: encode the screenshot, instruction, accessibility tree
    15. perception = model.encode(observation)
    16. # 2. Reasoning: decide whether to call a tool or perform an interface action
    17. decision = model.reason(perception, history)
    18. if decision.type == "tool_call":
    19. return execute_tool(decision, tools)
    20. # 3. Localization: map the semantic description to the target widget
    21. target = model.localize(decision.description, perception)
    22. # 4. Safety check: high-risk actions require confirmation
    23. if decision.risk_level == "high" and not has_human_confirmation(decision):
    24. return {"status": "blocked", "reason": "high_risk_action"}
    25. # 5. Execution: apply the action
    26. result = perform_action(target, decision)
    27. # 6. Feedback: the execution result enters history and the next round
    28. history.append((decision, result))
    29. return result

    The value of the closed loop is that the agent no longer outputs a one-time answer, but can adjust behavior based on feedback. For example, if the page does not navigate after a click, if a form is not submitted successfully, or if a blocking pop-up appears, the system can roll back, retry, or request human intervention.

    6. The Toolchain Closed Loop Must Include Safety Boundaries

    A native multimodal agent is different from an ordinary chat model because it can actually click, input, submit, scroll, or modify data. Therefore, the toolchain closed loop must include safety design. Emphasizing only “can execute” without emphasizing “execute safely” is a common risk in engineering deployment.

    At a minimum, the following measures are recommended:

    1. Sandboxed execution: Prefer executing in isolated environments, test accounts, mock interfaces, or snapshot environments, avoiding direct action on production systems. For systems that cannot be fully sandboxed, restrict the scope of operations and keep rollback mechanisms.
    2. Least privilege: The agent should only have the minimum permissions needed to complete the current task. Read-only tasks should not be granted write permissions; query tasks should not have delete, payment, send, or approval capabilities by default.
    3. Human confirmation for critical actions: For high-risk actions such as deleting data, submitting forms, payments, transfers, publishing content, changing permissions, and sending messages, human confirmation or secondary confirmation should be required.
    4. Logging and auditing: Record pre- and post-operation screenshots, model decisions, target widgets, action types, execution results, elapsed time, failure reasons, and human-confirmation records, for review and compliance auditing.
    5. Sensitive-information masking: When dealing with sensitive interfaces involving accounts, phone numbers, ID numbers, bank cards, addresses, passwords, and verification codes, masking or redaction should be applied in perception encoding, log storage, and model input.
    6. Retry limits: Avoid the agent clicking indefinitely on a wrong page. Set maximum retry counts, timeouts, consecutive-failure circuit breakers, and human-takeover mechanisms.

    7. Engineering Deployment: Q&A Checklist

    1. How do I choose between a native multimodal agent and a bolt-on vision module?

    If the task only needs to “understand an image and answer a question,” you can start with a mature vision-language model; if the task requires multi-step interface operation, correcting errors based on feedback, handling documents and videos, and directly executing clicks or inputs, the native multimodal agent architecture is more suitable. The key is not whether a vision model is used, but whether visual information can continuously participate in the training and decision loop.

    2. What are the common mistakes?

    Common mistakes include: giving the model only a screenshot without an accessibility tree; compressing the vision module’s output into vague text; training only the final action without training localization ability; having no rollback mechanism after a trajectory fails; no human confirmation for high-risk actions; and evaluating only on good cases, without covering failure scenarios such as occlusion, weak network, pop-ups, and insufficient permissions.

    3. How do I evaluate perception quality?

    Do not just look at “whether the model can answer correctly”; evaluate the perception layer separately: widget recall, text-recognition accuracy, icon-state classification accuracy, region-segmentation accuracy, and document table structure-parsing accuracy. The evaluation set should cover different resolutions, themes, widget densities, occlusion, and scroll states.

    4. How do I handle localization failures?

    If the model recognizes the “save button” but fails to localize it, first determine whether it is a perception error or a localization error. You can introduce a candidate-widget list and let the model choose the target; add the accessibility tree as an aid; add a center-point distance penalty; request user confirmation when localization confidence is low; or use multimodal retrieval or multi-candidate voting instead of forcing a click.

    5. How do I evaluate end-to-end task success rate?

    Define task-level metrics: success rate, average completion steps, misclick rate, rollback rate, human-takeover rate, timeout rate, and sensitive-action violation rate. It is recommended to also report layered metrics; otherwise you cannot determine whether the system bottleneck is in perception, localization, action, or planning.

    6. How do I reduce the risk of misoperation?

    It is recommended to adopt a canary, gradual rollout: first read-only, then low-privilege write operations, and only then open high-risk actions; record audit logs at every step; require human confirmation for high-risk actions; trigger a circuit breaker on abnormal behavior; and conduct red-team testing before launch, including scenarios such as misclicks, wrong pop-ups, insufficient permissions, data changes, and exposure of privacy-sensitive interfaces.

    8. Summary

    The core of a native multimodal agent is not simply equipping a text model with a visual input, but bringing perception, reasoning, and execution into the same training, evaluation, and safety system. In engineering deployment, perception quality should be treated as a key variable spanning the entire cycle; layered training and layered metrics should replace vague end-to-end evaluation; structured toolchains and feedback loops should improve execution capability; and sandboxes, least privilege, human confirmation, and audit logs should control risk. For teams that want AI to truly operate interfaces, handle documents, and understand videos, building an agent system that is evaluable, debuggable, and safely deployable via canary rollout is more practically valuable than simply scaling up model size.