Native Multimodal Agents: Designing a Closed Perception-Reasoning-Execution Loop
This article explains why native multimodal agents require a unified representation and a closed-loop system rather than a bolt-on vision module. It covers a six-layer architecture, layered visual perception, evidence-grounded reasoning, trajectory-focused training, toolchain contracts, evaluation methods, common challenges, and a practical deployment checklist. Readers will learn how to organize perception, reasoning, execution, validation, and observability around a verifiable loop for UI understanding, screen-to-code, document processing, video analysis, and multi-step operations.
1. Why Native Multimodal Agents Are Needed
A large amount of high-value real-world information exists in visual form: web interfaces, mobile screenshots, design files, layout documents, surveillance footage, presentation materials, charts, and video streams. The traditional approach often adds a vision module alongside a text-only model, and then upper-layer business code converts recognition results into text for the reasoning model. This can provide a quick start, but it remains a bolt-on integration at its core.
Bolt-on solutions commonly suffer from four problems:
- Fragmented representation spaces. The vision module outputs bounding boxes, labels, or description text, while the reasoning model sees second-hand information. If a visual description omits key controls, coordinates, or hierarchy relationships, subsequent planning is built on incorrect premises.
- Error amplification across stages. Recognition errors propagate into planning, action generation, and execution verification. Without end-to-end feedback, errors are difficult to locate and even harder to correct automatically.
- State and action become disconnected. Tasks such as UI operations, code generation, and document parsing require continuous observation and multi-step execution. If visual perception is only a single-turn input, the model struggles to maintain consistency among the current UI state, executed actions, and the next goal.
- Fragmented toolchains. Screenshot capture, text recognition, layout parsing, coordinate mapping, executors, and validators operate separately, lacking a unified state model and observable链路. Engineering complexity rises quickly as scenarios increase.
Therefore, a “native multimodal agent” should not be understood as “a language model with a vision encoder added beside it.” It should be understood as a set of systemic constraints: images, videos, text, and action history enter a unified representation space; the model continuously references visual evidence during reasoning; execution results flow back as new observations; training data covers complete trajectories rather than isolated question-answer pairs; and the toolchain provides a replayable, verifiable, and auditable closed loop.
2. Overall Architecture: From Single-Turn Input to a Closed-Loop System
A practical native multimodal agent usually contains six layers: input, perception, representation, reasoning, execution, and feedback. Its core is not any single module, but the contracts between layers.
Input layer: text instructions, images, videos, action history, environment statePerception layer: layout parsing, control recognition, text recognition, visual grounding, video temporal samplingRepresentation layer: cross-modal semantic alignment, resolution and frame-rate adaptation, evidence indexingReasoning layer: task planning, tool selection, state tracking, hypothesis generation, result verificationExecution layer: code generation, UI operations, document extraction, video event output, tool callsFeedback layer: execution result verification, failure retry, log tracing, memory updates
There are three key design principles:
- Observations must be structured and traceable. Each piece of visual evidence should carry its source, region, timestamp, and confidence, preventing the model from outputting only an unverifiable natural-language description.
- Actions must be executable and rollback-capable. The action space should be explicitly defined, for example answering a question, calling a tool, clicking a control, typing text, generating code, or requesting human confirmation. Execution results must be able to update state.
- Validators must enter the main loop. A validator can be a rule, a tool return value, a rendering difference, a unit test, a field consistency check, or a secondary visual verification. Its role is not decorative; it determines whether to continue, retry, or terminate.
3. Perception Layer: Vision Is More Than “Converting to Text”
The perception layer of a native multimodal system needs to handle both global layout and local detail. Relying only on text recognition degrades UI understanding into a collection of text fragments, losing spatial relationships, control hierarchy, and visual salience.
3.1 Layered Visual Parsing
A three-level representation of “global—region—element” can be used:
- Global level: page type, main functional areas, reading order, overall layout, and task-relevant regions.
- Region level: division into forms, lists, toolbars, modals, canvases, video shots, and other regions.
- Element level: buttons, input fields, icons, text blocks, table cells, timestamps, and other elements, along with their coordinates, hierarchy, and state.
This hierarchy allows the model to establish task context first and then locate actionable targets, reducing the problem of “seeing text but failing to find the control” in complex interfaces.
3.2 Temporal Modeling for Video
Video is not a simple stack of images. The system needs to consider frame sampling, keyframe selection, event boundaries, action continuity, and temporal grounding. Common engineering practices include:
- First build a global summary at a low frame rate to locate candidate time ranges;
- Then increase sampling density for candidate clips and perform fine-grained recognition;
- Bind timestamps, shot changes, and action descriptions to the evidence index;
- Use hierarchical memory for long videos to avoid inserting all frames at once, which would cause context and cost to spiral out of control.
3.3 Resolution and Token Budget
Higher visual resolution generally helps recognize small text, icons, and details, but it significantly increases computation and latency. A reasonable strategy is dynamic allocation: first use low resolution for page-level localization, then crop and enlarge key regions; for video, first perform shot-level filtering, then conduct clip-level fine analysis. The perception layer should also output uncertainty, such as “the button text is blurry” or “this region is occluded,” so the reasoning layer can choose to ask follow-up questions, retry, or switch to another tool.
4. Representation and Reasoning: Let Visual Evidence Participate in Decisions
The key to native multimodality is that visual evidence must be continuously referenceable by the reasoning layer, just like text context. The system can map different modalities into a shared semantic space and maintain structure through modality markers, spatial coordinates, timestamps, and evidence IDs.
The reasoning layer usually has four responsibilities:
- Task decomposition: break “complete a UI task” into steps such as observe, locate, input, submit, and verify.
- Tool selection: decide when text recognition, layout parsing, code execution, image difference comparison, or retrieval is needed.
- State maintenance: record executed actions, current UI state, failure reasons, and unmet constraints.
- Self-verification: perform consistency checks before and after key actions, such as confirming field completeness before submission or checking renderability after code generation.
A common mistake is to merge planning and execution into one step. A more robust approach is a state machine of “plan—act—observe—correct”: the planner proposes the next step, the executor returns a verifiable result, and the validator decides whether to enter the next round. For high-risk actions such as deleting data, submitting forms, or sending messages, permission checks or human confirmation should be introduced.
5. Training Methods: Put Trajectory Quality First
Native multimodal capabilities cannot be trained only on static image-text pairs. Agents need to learn how to observe, act, and recover in continuous tasks.
5.1 Multimodal Foundational Alignment
The first stage should establish foundational semantic alignment among text, images, and video, including object and region reference, layout reading order, chart understanding, video event description, and the association between text and coordinates. This stage should retain sufficient text-only data replay to avoid language capability degradation.
5.2 Task Supervised Fine-Tuning
The second stage constructs instruction data around specific tasks, such as UI understanding, screen-to-code, document field extraction, video event localization, tool calls, and multi-step operations. The data should include not only final answers, but also intermediate evidence, action rationales, and failure recovery examples.
5.3 Trajectory Learning and Feedback Optimization
The core training signal for agents comes from trajectories: observations, actions, execution results, and verification conclusions. High-quality trajectories should satisfy the following:
- Actions correspond strictly to observations;
- Failed steps are retained and annotated with correction methods;
- Verification results are reviewable;
- State changes are replayable;
- No sensitive information or unauthorized operations are included.
In the feedback optimization stage, signals such as task success rate, code executability, rendering consistency, field accuracy, and human intervention rate can be used to construct rewards. However, reward hacking must be guarded against: the model may learn to bypass the validator rather than truly complete the task. Validators therefore need to be diverse and executed in a sandbox.
5.4 Regression Protection for Text-Only Capabilities
Multimodal training can easily dilute language capabilities. In practice, techniques such as data ratio control, text replay, staged unfreezing, and capability regression tests should be used. Before release, at least one text benchmark should be maintained, covering question answering, summarization, structured output, code comprehension, and long-context reasoning, to ensure that multimodal gains do not come at the cost of a significant decline in text capability.
6. Toolchain: Whether the Loop Works Depends on Engineering Contracts
The toolchain is the most easily underestimated part of native multimodal agents. No matter how strong the model is, if observations are inconsistent, actions cannot be rolled back, and logs cannot be traced, the system will be difficult to run stably.
6.1 Unified Observation and Action Protocol
It is recommended to explicitly define observation, action, result, and error types. An illustrative pseudo-code example is as follows:
type Observation = {text?: stringimageRef?: stringvideoRef?: stringviewport?: { width, height, scale }timestamp?: numberhistory: ActionRecord[]}type Action =| { kind: "answer", content: string }| { kind: "tool", name: string, input: object }| { kind: "ui", target: Region, operation: "click" | "type" | "scroll", value?: string }| { kind: "code", language: string, content: string }type ActionResult = {ok: booleanoutput?: objecterror?: { code: string, message: string }newObservation?: Observation}function agentLoop(obs, planner, executor, verifier, maxSteps) {memory = []for step in range(maxSteps) {plan = planner.plan(obs, memory)result = executor.run(plan.action)memory.append({ plan, result })if (verifier.done(obs, memory, result)) {return result}obs = updateObservation(obs, result)}return fallbackWithHumanReview(memory)}
The point of this protocol is not the code form, but the contract: every action has structured input and output, every result can update observations, the validator has termination authority, and after the step limit is exceeded, the system can fall back to human handling.
6.2 Sandbox, Rollback, and Permissions
The execution environment should isolate network, file system, input devices, and sensitive data access. UI operations should support state snapshots and rollback; code execution should run in a sandbox with resource and timeout limits; tool calls should follow the principle of least privilege. Irreversible actions must have secondary confirmation or an approval process.
6.3 Observability
The full链路 should record input observations, model outputs, tool call parameters, execution results, verification conclusions, retry reasons, latency, and cost. Logs should be desensitized and support replay by task. Without replay capability, it is difficult to determine whether “perception was wrong, planning was wrong, or the executor was wrong.”
7. Evaluation: Measure Both Component Capabilities and End-to-End Results
Evaluation should cover component-level and system-level metrics.
- Perception metrics: region localization accuracy, control recognition accuracy, text recognition accuracy, video event temporal localization error, and robustness in occluded and blurry scenarios.
- Task metrics: UI operation success rate, screen-to-code renderability and visual consistency, document field extraction accuracy, video event recall, and average number of completion steps.
- Closed-loop metrics: first-attempt success rate, success rate after retry, human intervention rate, rollback rate, error recovery rate, and timeout rate.
- Language regression metrics: performance on text-only question answering, summarization, structured output, code comprehension, and long-context tasks.
- Security and compliance metrics: unauthorized operation interception rate, sensitive information leakage risk, robustness to malicious instruction injection, and audit completeness.
- Engineering metrics: first-token latency, end-to-end latency, cost per task, and concurrency stability.
Beyond offline evaluation, small-traffic canary rollouts, online A/B testing, failure sample feedback, and red teaming should also be conducted. Particular attention should be paid to visual prompt injection: text in screenshots or documents may contain诱导性 instructions, and the system must distinguish between “observed content” and “executable instructions.”
8. Major Challenges and Mitigation Strategies
- Visual hallucination. The model may describe controls that do not exist or misread coordinates. Mitigations include evidence indexing, region crop re-verification, confidence thresholds, and validators.
- Resolution versus latency conflict. Use layered perception, dynamic cropping, caching, and on-demand fine-grained analysis.
- Long-video cost. Summarize first and localize later, use keyframes and event segmentation, and avoid feeding all frames.
- Difficult action grounding. Coordinates, controls, and semantic targets must be explicitly bound, and screenshots or state read-back verification should be performed immediately after execution.
- Catastrophic forgetting. Protect language capabilities through text replay, data ratios, and regression testing.
- Tool error propagation. Treat tool results as evidence with confidence rather than absolute facts; use cross-validation with multiple tools on critical paths.
- Security and privacy. Desensitize screenshots, documents, and videos, restrict tool permissions, record audit logs, and support human takeover.
- Insufficient reproducibility. Standardize observation normalization, action protocols, random seeds, and replay mechanisms to ensure issues can be located.
9. Deployment Checklist
- Has a unified protocol for observations, actions, results, and errors been defined?
- Have complete trajectories including failure recovery been collected, rather than only static question-answer pairs?
- Are text-only replay and regression tests retained?
- Does a validator enter the main loop and have the ability to terminate, retry, and hand over to humans?
- Are code and UI operations executed in a sandbox and able to roll back?
- Are desensitized, replayable logs and evidence chains recorded?
- Have red team tests been conducted for screenshot injection, unauthorized operations, and sensitive information leakage?
- Are component-level perception capabilities, end-to-end task success rates, and engineering costs all evaluated?
10. Conclusion
The value of native multimodal agents lies not in “connecting more modalities,” but in letting visual evidence truly participate in reasoning, letting execution results truly flow back as observations, and organizing training data and toolchains around the closed loop. Architecturally, representations and state must be unified; in training, trajectories, feedback, and text regression must be emphasized; in engineering, an executable, verifiable, rollback-capable, and observable toolchain must be established. Only when these three work together can the system maintain a low hallucination rate and high operability in real tasks such as screen-to-code, screen understanding, document processing, video analysis, and multi-step operations, while providing an engineering foundation for continuous iteration.
