Native Multimodal Agents: Architecture and Training Paths for Visual Capabilities
This article explores the architecture and training approaches for native multimodal AI agents, emphasizing the necessity of integrating visual perception as a foundational capability. It covers the engineering implications of native multimodality, the perception-reasoning-execution loop, typical application scenarios, and key engineering practices for real-world deployment.
1. Why Visual Capability Must Be Native
An increasing amount of work-related information exists in image form. Web interfaces, mobile app screenshots, design drafts, scanned contracts, chart-heavy reports, and surveillance footage cannot be fully described by plain text. The relative position of buttons, color hierarchies, spacing, and alignment in a design draft are difficult to accurately convey in a few sentences.
The current common approach is a “bolt-on” solution: an independent vision module first converts the image into text descriptions or structured results, which are then handed over to a language model for processing. This method is simple to deploy but presents three problems:
- Information loss during conversion: The descriptive text only retains what the vision module considers important, often omitting details like layout, gradients, and the state of small icons.
- Disconnect between perception and reasoning: The language model cannot look back at the original image and can only answer based on intermediate descriptions. Errors from the perception module directly become premises for reasoning, and they are difficult to detect.
- Increased hallucination: When descriptions are missing, models tend to fill in the gaps with common sense, generating content that seems plausible but contradicts the original image.
The native multimodal approach treats vision as a foundational perceptual capability of the model. Encoded images and text exist in the same representation space, and a single set of parameters understands both modalities. Visual capability participates in learning from the pre-training stage; modules added after model training is complete do not fall into this category.
2. The Engineering Implications of “Native”
Simply adding a vision encoder does not make a system native. An engineering-viable native multimodal system is typically designed across the following four layers:
- Representation Layer: Images are split into patches or regions and mapped into a vector sequence compatible with text tokens, which are then input into the backbone network alongside text.
- Training Layer: Pre-training data contains large volumes of interleaved image-text documents, UI screenshots with corresponding descriptions, and video clips with temporal annotations. Image-text alignment can be integrated throughout various training stages, rather than just being supplemented with small amounts of data at the end.
- Task Layer: The model simultaneously learns tasks such as visual question answering, UI understanding, code generation, and spatial reasoning, ensuring visual understanding directly serves specific actions.
- Tool Layer: The model outputs structured action commands and receives new screens after execution, forming a continuous feedback loop.
These four layers require collaborative design. If the representation layer supports images but training data remains predominantly text-based, the model will often “see but not see accurately.”
The main differences between the two approaches are as follows:
| Dimension | Bolt-on Vision Approach | Native Multimodal Approach |
|---|---|---|
| Visual Information Retention | Relies on intermediate descriptions; details easily lost | Directly utilizes image representations; retains more details |
| Perception and Reasoning | Sequential execution; errors propagate level by level | Joint modeling; both can mutually correct each other |
| Training Cost | Independent modules; faster iteration | Requires large-scale multimodal data and computing power |
| Text-only Capability | Basically unaffected | Requires specialized design to prevent capability degradation |
| Engineering Complexity | Multiple interfaces; difficult to optimize uniformly | Single model handles perception; training pipeline is more complex |
3. The Perception-Reasoning-Execution Loop
Multimodal models designed for agents typically operate around a loop: perception, reasoning, execution, and then back to perception.
3.1 Perception
The goal of the perception stage is to convert the current screen into a state representation usable by the model, primarily including:
- Identifying UI elements (e.g., buttons, input fields, list items) and providing their positions and types;
- Reading text content, including small fonts and non-standard typefaces;
- Understanding hierarchical relationships, such as determining which control belongs to which container;
- Determining states, such as whether a switch is toggled on or if a loading process is complete.
Perception quality determines the upper limit of subsequent steps. If a “submit button in a disabled state” is misidentified as “clickable,” all subsequent actions will fail.
3.2 Reasoning
The reasoning stage determines the next step based on the task goal and current state. Typical work includes task decomposition, progress evaluation, anomaly identification, and fallback. For example, if a user requests to “export rows with values greater than 100 in the table to a CSV file,” the model needs to break this down into sub-steps: locating the table, identifying the numerical column, filtering the data, exporting the file, and confirming the result.
3.3 Execution
The execution stage translates decisions into actions acceptable by the environment. Actions can be UI operations (e.g., clicks, inputs, scrolls) or calls to external capabilities (e.g., querying data, reading/writing files, sending requests). The action space should be clearly defined in advance, and each parameter requires validation.
3.4 Feedback
After execution, the screen is captured again or results are returned, serving as the input for the next round of perception. The value of the loop lies in allowing the model to detect when an “action did not take effect” and adjust accordingly, rather than assuming each step was successful.
Below is illustrative pseudocode explaining the loop structure. Specific interfaces should be designed based on the actual runtime environment.
function run_agent(task, env, model, max_steps = 30):history = []for step in range(max_steps):screen = env.capture() # Perception: capture current screenstate = model.perceive(screen, task, history)if state.task_done:return {status: "success", steps: step}action = model.plan(state, task, history) # Reasoning: determine next stepif not env.is_allowed(action): # Validate action space before executionreturn {status: "rejected", action: action}result = env.execute(action) # Executionhistory.append((action, result.summary)) # Record for next iteration referenceif result.error:model.handle_error(result)return {status: "timeout", steps: max_steps}
This code embodies three engineering principles: re-perceive at every step without relying on outdated states; perform whitelist validation before execution; and set a maximum step count to prevent infinite loops.
4. Typical Application Scenarios
4.1 UI to Code Conversion
Converting design drafts or screenshots into frontend page code. The key is to first understand the layout structure—including grids, flexbox arrangements, and hierarchical nesting—before mapping them to style rules. Evaluation should consider both visual fidelity and code maintainability. Pursuing only pixel-level similarity often results in heavily absolute-positioned code that is difficult to modify later.
4.2 Mobile and Desktop Automation
Performing operations in applications without open APIs, such as batch organizing photo albums, verifying order information, or filling out repetitive forms. These tasks require high positioning accuracy and robust error recovery, as a single mis-click can derail all subsequent steps.
4.3 Document and Chart Understanding
Extracting tables, formulas, and chart data from scanned documents or PDFs, and answering questions in conjunction with the main text. Difficulties include cross-page references, multi-column layouts, and the correspondence between charts and descriptive text.
4.4 Video Event Understanding
Generating timeline summaries for surveillance or instructional videos to locate when key events occurred. The sampling strategy requires a trade-off: too high a frame rate increases computational costs, while too low a frame rate may miss very short-lived events.
5. Key Engineering Implementation Points
5.1 Data Construction
Training and evaluation data should cover the distribution of real-world scenarios, including low-resolution screenshots, dark mode, multilingual interfaces, and unexpected pop-ups. Synthetic data can expand sample sizes but should be periodically compared against real samples to ensure the model learns UI content rather than just rendering styles.
5.2 Maintaining Text-Only Capabilities
Multimodal training can easily lead to a decline in language capabilities. Common practices include mixing in pure text corpora, adding text task replays during the later stages of training, and conducting regression tests on text-based tasks (such as coding, math, and writing) before deployment.
5.3 Toolchains and Permissions
Action interfaces should follow the principle of least privilege. Irreversible operations like deletion, payment, and sending require human confirmation steps. Logs should record the screen summary and executed action for each step to facilitate post-hoc auditing and troubleshooting.
5.4 Evaluation Metrics
A single metric can be misleading. It is recommended to monitor the following simultaneously:
- Task success rate: Whether the goal is achieved end-to-end;
- Average steps: Reflects decision-making efficiency;
- Error recovery rate: Whether the model can self-correct after encountering anomalies;
- Latency and cost: Response time and computational overhead per step;
- Violation rate: Whether prohibited actions are triggered.
6. Practical Recommendations
- Clarify the input format of the task first. If the task relies solely on text information, native visual capabilities may not be necessary.
- Design a clear action space and place validation logic outside the model, executed by deterministic code.
- Retain human confirmation for critical steps, gradually relaxing the level of automation as reliability is verified.
- Establish a regression test suite and run it every time the model or prompts are updated.
