Native Multimodal Agents: From Bolted-On Perception to Closed-Loop Reasoning

    This article explains what native multimodal agents are and how they replace traditional bolted-on perception pipelines with a single unified model. It covers the three structural flaws of the vision-to-text-to-reasoning approach, what it means to unify images, video, and text within one representation space, and the staged joint-training strategy used to align modalities. It then details how to implement the perception–reasoning–execution loop, typical application scenarios, how to evaluate such agents across visual understanding, task completion, and text-capability retention, and practical engineering advice for developers.

    1. In the Real World, Vision Is the Main Stream

    Open almost any device and you’ll find that a large share of the information that matters does not exist as plain text: the layout and controls of a web page, the UI state of a mobile app, the layers and grids of a design mockup, the formatting of a PDF, or the moving objects in a surveillance feed. An agent built around a text-only LLM that wants to “see” and “act” in these scenarios usually relies on a bolted-on pipeline: first call a vision model to turn the image into a text description, then feed that text to a language model to make decisions, and finally call scripts to execute actions.

    This pipeline looks clean, but in practice it hard-cuts perception and reasoning into two separate stages.

    2. Three Structural Flaws of the Bolted-On Approach

    1. Lossy information and semantic drift
    Converting an image to text inevitably loses information. The position of a button, its color contrast, or an occlusion relationship tends to be heavily generalized once described in natural language. The language model is left with “second-hand” information and struggles to make pixel-precise localization decisions.

    2. Disconnected perception and reasoning lead to hallucination
    When the vision module and the language module are trained separately and do not share a representation space, the language model tends to “fill in” things it never actually saw. For example, it may infer from common sense that there is a “login button” in the UI when none exists on screen, so the resulting action misses its target. This is the classic perception–reasoning gap hallucination.

    3. No task closure
    The real value of an agent lies in the “see–think–do” loop: observe the current state, reason about the next step, execute an action, and observe the result as feedback. In the bolted-on approach, perception is a one-time upfront step, and reasoning cannot in turn guide where perception should focus. The loop is broken into a one-way pipe.

    3. What “Native” Multimodal Means: Unifying into One Representation Space

    The core claim of native multimodal is to unify images, video, and text within the same model, the same set of parameters, and the same representation space, rather than bolting an extra vision encoder onto a language model. A few points are key here.

    Unified tokenization and representation
    A native approach lets image/video frames and text tokens share the same sequence modeling scheme. After an image is encoded, it is mapped into a token sequence that is isomorphic to word embeddings, and the model treats them as the same kind of “symbol” during autoregressive or bidirectional attention. In this way, visual information is no longer an add-on input but a first-class citizen the model can understand natively.

    Perception and reasoning interact within the same attention layer
    Because the representations are unified, language tokens and visual tokens can attend to each other within the same forward pass. When the model reasons about “where is this button,” its attention can focus directly on the corresponding pixel region in the image. Perception provides evidence for reasoning, and reasoning in turn guides what to look at in the next round.

    4. Joint Training: Truly Aligning the Two Modalities

    Native multimodal is not finished by simply welding on an encoder; the training strategy determines the ceiling of its capabilities. The industry commonly adopts a staged approach:

    1. Stage 1 Modality alignment (align the vision and language representation spaces)
    2. Goal: teach the model that "this visual segment corresponds to this concept"
    3. Stage 2 Instruction tuning (multi-task instruction following)
    4. Goal: screen understanding, document parsing, UI operation, visual QA, etc.
    5. Stage 3 Agent behavior alignment (action trajectories and tool calls)
    6. Goal: the observe → decide → execute closed-loop capability

    Among these, the agent behavior alignment stage is especially important. Developers need to collect “observation–action” trajectory data (for example, the successful sequence of performing an operation on a given screen), so the model learns to choose the appropriate action for a given visual state and to keep deciding based on the new screen after execution. Constructing data at the trajectory level is the key engineering step in building a closed-loop agent.

    5. Implementing the Perception–Reasoning–Execution Loop

    To make an agent truly run, engineering needs to connect several layers:

    • State observation layer: abstract inputs such as web DOM screenshots, mobile screens, and video keyframes into structured visual input, with necessary coordinates or element metadata as the action space.
    • Decision and reasoning layer: the model outputs structured actions (such as a click coordinate, input text, or a scroll command) based on the current visual state, rather than free-form text.
    • Execution and feedback layer: actions are executed in the real system environment, and the returned new state re-enters the observation layer, forming a loop until the task is completed or times out.

    A schematic scheduling loop looks like this:

    1. def run_agent_task(model, env, goal, max_steps=20):
    2. state = env.observe() # capture the current visual/structured state
    3. for step in range(max_steps):
    4. action = model.plan(state, goal) # the native multimodal model outputs a structured action
    5. if action.type == "finish":
    6. return action.result
    7. state = env.execute(action) # execute in the real environment, return the new state
    8. return {"status": "timeout", "last_state": state}

    The key is that model.plan takes visual state directly and outputs a structured action; the intermediate “describe the image” step is absorbed by native capability, so perception and reasoning are no longer two separate stages.

    6. Typical Application Scenarios

    • Design-to-code: take a UI screenshot as input and output a usable front-end structure, testing the model’s visual understanding of layout, hierarchy, and styling.
    • Mobile UI automation: recognize the screen state and drive clicks, input, and swipes to complete cross-app task flows.
    • Document and layout understanding: parse the mixed text-and-image information in PDFs, scanned documents, and table screenshots.
    • Video understanding and retrieval: identify and localize key events in surveillance or operational recordings.

    7. How to Evaluate a Native Multimodal Agent

    Evaluation should cover multiple dimensions rather than a single score:

    • Pure visual understanding: benchmarks such as screenshot QA, document parsing, and chart reading.
    • Agent task completion rate: the success rate on end-to-end tasks such as design-to-code and mobile operation, which measures closed-loop capability.
    • Retention of pure-text capability: this is often overlooked—after adding visual capability, the model’s original code, reasoning, and language understanding should not degrade noticeably, otherwise it is a case of fixing one problem while creating another.

    8. Practical Advice for Developers

    1. Prioritize unified representation: do not simply chain a vision model onto a language model in series, as that is just another bolted-on approach.
    2. Value trajectory data quality: the ceiling of an agent’s capability depends largely on how well high-quality “observation–action” trajectories are constructed and cleaned.
    3. Structured action space: have the model output parseable actions rather than natural language, so the execution layer can be driven reliably.
    4. Establish regression evaluation: while enhancing visual capability, be sure to monitor whether pure-text capability degrades.
    5. Cloud-side engineering: you can use general container platforms to host inference services, object storage to manage multimodal datasets, and logging services plus monitoring alerts to track task success rates, building an observable and scalable deployment pipeline.

    9. Summary

    The essence of a “native multimodal agent” is turning vision from an external patch into an internal first-class capability: unified representation lets perception and reasoning interact within the same space, staged joint training truly aligns the two modalities, and the perception–reasoning–execution loop gives the model a continuous “see–think–do” decision-making ability. For developers, mastering this complete line of thinking—from architecture to training to tooling—is more valuable than chasing a particular model’s score, and it transfers more readily into self-built agent systems.