Native Multimodal Agents: Architecture and Training Paths

    This article explains why native multimodal agents should incorporate vision into a unified perception-reasoning-execution loop instead of attaching a vision module to a text model. It covers the limitations of modular stitching, a four-layer architecture, joint training strategies, toolchain design, and practical evaluation and deployment recommendations.

    1. Starting Point: Vision Should Not Be an After-the-Fact Patch

    In real business scenarios, a large amount of information does not exist in plain text form. Web interfaces, mobile screenshots, design mockups, scanned tables, PDF documents, surveillance footage, and screen recordings are all first-class inputs that models need to understand. If a system uses a “text model + external vision module” stitching approach, the vision module often generates a description first, which is then passed to the language model for reasoning. This may look fast, but it has three structural flaws:

    1. Information loss: The vision module outputs a compressed text description, so fine-grained information such as layout, control relationships, hierarchy, color, and position is easily lost.
    2. Perception-reasoning disconnect: There is a lack of joint optimization between the “description” the model sees and the task it actually needs to execute, which can lead to hallucinations in complex interfaces.
    3. Fragmented execution chain: From understanding an interface to generating an operation action, multiple modules often need to collaborate in series, causing errors to accumulate step by step.

    Therefore, the core of a “native multimodal agent” is not adding an encoder next to a language model. Instead, images, video, and text participate in perception, reasoning, and execution within a unified representation space, forming a closed loop. Here, “native” includes at least three layers of meaning: unified input representation at the architecture level, joint optimization of multimodal tasks during training, and a toolchain that enables the model to call external capabilities and verify execution results.

    2. Overall Architecture: From Unified Representation to Closed-Loop Execution

    A complete native multimodal agent can be abstracted into four layers:

    • Input layer: Receives images, video frames, text instructions, interface structure information, historical operation records, and more.
    • Unified perception layer: Maps different modalities into an alignable representation space while preserving spatial position, temporal relationships, and semantic information.
    • Reasoning and planning layer: Performs state understanding, step decomposition, tool selection, and risk assessment based on task goals.
    • Execution and feedback layer: Generates structured actions, calls tools or environment interfaces for execution, and returns results to the model for continued reasoning.

    Unlike serial approaches that “describe first, then reason,” the closed-loop design emphasizes re-observing the environment after every action. For example, in an interface operation task, the model needs to first identify buttons, input fields, and list relationships, then decide whether to click or type. After execution, it captures a new screenshot to confirm whether the state has changed, and adjusts its strategy if the action failed. This “observe-think-act-observe again” cycle is closer to how real agents work.

    A simplified pseudocode example is as follows:

    1. state = observe_environment() # screenshots, text, structural information
    2. history = []
    3. while not task_finished(state):
    4. plan = model.plan(
    5. instruction=user_goal,
    6. observation=state,
    7. history=history
    8. )
    9. action = model.select_action(plan, state)
    10. result = execute(action)
    11. history.append((state, action, result))
    12. state = observe_environment()
    13. return model.summarize(history)

    The key point of this pseudocode is not the specific interfaces, but the control flow: the model does not output an answer all at once. Instead, it continuously corrects itself through multiple rounds of perception and execution.

    3. Training Strategy: Joint Tasks Matter More Than Isolated Capabilities

    Training a native multimodal agent usually needs to cover multiple types of tasks and make them work together under the same objective:

    1. Visual understanding and grounding: Not only identify “what is there,” but also locate “where it is,” such as control boundaries, text regions, chart elements, and their relationships.
    2. Structured understanding of documents and interfaces: Convert screenshots or design mockups into hierarchical structures and understand semantics such as containers, lists, forms, and navigation.
    3. Cross-modal reasoning: Combine text instructions with visual states for conditional judgments, such as “close the pop-up only when it appears.”
    4. Action generation and trajectory learning: Learn action sequences such as clicking, typing, scrolling, and dragging, and support rollback or retry after failure.
    5. Preservation of text-only capabilities: Multimodal training should not damage language understanding, code generation, or logical reasoning. Balance should be maintained through data mixing and joint training.

    For training data construction, common practices include aligning interface screenshots with structured annotations, aligning operation trajectories with natural language goals, pairing design mockups with code samples, and aligning video frames with event descriptions. The key is not the volume of a single data source, but whether the correspondences between data are fine-grained enough. For example, a “screenshot-to-full-page-code” pair alone is not enough. The model also needs to know which structure a certain button corresponds to and which style rule a certain layout constraint corresponds to.

    4. Toolchain: Let the Model Both “See” and “Act”

    For native multimodal agents to be deployed, toolchain design is critical. The toolchain should provide at least:

    • Environment observation tools: Screenshots, DOM/structure trees, accessibility information, video frame extraction, and more.
    • Action execution tools: Clicking, typing, scrolling, opening pages, calling business APIs, and more.
    • Result verification tools: State comparison, page difference detection, and judgment of task success conditions.
    • Security and permission control: Action allowlists, confirmation for sensitive operations, sandbox execution, and audit logs.

    From an engineering perspective, it is recommended to decouple “model decision-making” from “actual execution”: the model outputs structured action intents, and the executor performs permission checks and parameter validation. Execution results are returned in a unified format, avoiding direct model access to high-risk interfaces. This makes debugging easier and allows reuse across multiple environments.

    5. Evaluation and Deployment Recommendations

    When evaluating native multimodal agents, single-turn question-answering accuracy is not enough. More reasonable metrics include task completion rate, average number of steps, failure recovery rate, execution safety, human takeover rate, and overall performance on tasks such as interface-to-code, mobile operation, and document understanding. Benchmarks for interface-to-code and mobile operation tasks essentially test whether “perception-reasoning-execution” is truly closed-loop, rather than isolated visual recognition capability.

    For deployment, a staged approach is recommended: first handle read-only understanding tasks such as interface review and document information extraction; then introduce low-risk write operations such as form draft generation; finally expand automated operations in sandbox environments. Each stage should retain observable logs and human fallback mechanisms. For enterprise users, the value of native multimodal agents is not “adding a vision interface,” but incorporating visual information into a unified reasoning chain, reducing information loss and hallucination risks caused by module stitching, and providing a verifiable execution loop for complex real-world tasks.