Native Multimodal Agents: Architecture Design and Engineering Practices
Why bolted-on vision pipelines break down on complex tasks, and how to build native multimodal agents that treat vision as a first-class input. This article covers a unified token space for images, video, and text; structured action outputs; layered context management; a four-stage training path from cross-modal alignment to trajectory-level post-training; the toolchain required for stable serving; a three-layer evaluation system with anti-hallucination testing; common pitfalls; and a go-live checklist for production teams.
1. Problem Definition: Why “Bolt-On Vision” Can’t Handle Complex Tasks
Over the past two years, pure text-based large models have made remarkable progress in code generation, logical reasoning, and long-form writing. As a result, many teams have settled into a habitual approach: when the model isn’t capable enough, patch the gap with interfaces. To read a screen, they string a screenshot-parsing service in front of the inference pipeline; to read a document, they attach a layout-reconstruction service; to operate software, they wrap another layer of click scripts. This “capability bolting-on” often looks good in demos, but once it enters real business scenarios, the problems surface all at once.
First, perception and reasoning live in two disconnected semantic spaces. A vision module typically outputs things like “what text is in this region” or “what objects are present,” whereas the decision model needs “which stage of the workflow the current screen is in, where to click next, and how to roll back if it fails.” The interface in between—compressed into a short string of text—is precisely where the most information is lost: pixel-level layout, control hierarchy, and temporal animation information get dropped during transcription, leaving the model to reason over an incomplete description. Hallucinations follow.
Second, errors can’t close the loop. In a bolted-on pipeline, the vision model doesn’t know it’s wrong because it can’t see the task objective; the decision model can’t correct the vision model because it doesn’t have access to the original image. Errors can only accumulate in one direction.
Third, cost and latency spin out of control. Multiple serial connections mean multiple network round trips, redundant image uploads, and separate quota management for each service. The longer the task chain, the higher the marginal cost—and a single failure forces a full rerun of the entire chain.
Therefore, the core proposition of a “native multimodal agent” is not “swap in a stronger vision model,” but rather to treat vision as a first-class input modality on par with text, so that perception, reasoning, and execution share the same representation and the same context, forming a closed loop.
2. Architecture Design: Unified Representation and a Perception–Reasoning–Execution Loop
2.1 A Unified Token Space
The key design is mapping images, video frames, and text into the same sequence space. The typical approach: an image is split into patches by a vision encoder, aligned to the language model’s hidden dimension through a projection layer, then inserted into the context as vision tokens that share the same attention mechanism and positional encoding as text tokens.
Three engineering points are worth attention here:
- Dynamic resolution instead of fixed stretching. Key information in a screen capture is concentrated in small text and icons; crude scaling directly wipes out readability. The industry commonly adopts an “adaptive tiling + block-wise encoding” approach: split the image into several sub-images according to its original aspect ratio, encode each separately, then concatenate them, applying overlap between adjacent sub-images to reduce semantic fragmentation at the boundaries.
- Budget control for vision tokens. A single high-resolution screenshot can consume over a thousand tokens. Viable strategies include compressing by information density (reducing sampling for large solid-color regions), re-encoding only changed regions (leveraging differences between consecutive frames), and setting a hard token budget on the inference side—when exceeded, crop first, then send to the model.
- Modeling the temporal dimension. Video and operation sequences are not just a stack of independent images. Only by explicitly encoding frame timestamps and the correspondence between actions and frames into the sequence can the model understand causal structures like “the screen jumps after a click.”
2.2 Bringing Actions into the Unified Space
The essential difference between an agent and a pure question-answering model is that the output is not just text but also actions. In engineering practice, structured output is typically used to make actions a constrained token sequence:
{"observation": "Currently on the order detail page; a 'Submit' button is present in the bottom-right corner","thought": "All required form fields are filled in, so it is safe to submit","action": {"type": "click","target": {"bbox": [0.82, 0.91, 0.94, 0.96]},"confidence": 0.86},"fallback": "If the page does not change within 2 steps, switch to accessibility-tree targeting"}
Note two design points here: coordinates are expressed as relative proportions rather than absolute pixels, which naturally adapts to different resolutions and scaling; and a fallback and confidence must be carried, so that the execution layer can switch targeting methods when the model is uncertain. Beyond visual coordinate targeting, symbolic information such as accessibility trees, control text, and page structure complements pixel information and is the key redundancy for improving real-world success rates.
2.3 Unified Message and Context Management
Routing all modalities through the same context means you must solve the question of “whether to keep historical screenshots around indefinitely.” A layered memory is recommended: retain original images for the most recent few steps, keep only text-based observation summaries and key frames for earlier steps, and compress and archive completed subtasks into summaries. This preserves the ability to trace back while avoiding linear context growth.
3. Training Path: From Modal Alignment to Trajectory-Level Agent Capability
A native multimodal model can’t be trained in a single stage. A reasonable path is usually divided into four progressively deepening layers.
Layer 1: Cross-modal alignment pre-training. Train on large-scale interleaved image-text corpora (web snapshots, documents with mixed text and images, subtitled videos, etc.) with the goal of aligning visual representations with the language semantic space. Interleaving order matters: keeping the natural reading or browsing order of images and text lets the model learn layout understanding beyond merely “describing an image.”
Layer 2: Visual instruction fine-tuning. Targeted enhancement for specific capabilities: OCR and layout reconstruction, reading numeric values from charts, semantic localization of UI elements (grounding), and structured document extraction. This layer directly determines the model’s usability on real screenshots, and data quality matters far more than quantity—incorrect annotation boxes will systematically degrade localization ability.
Layer 3: Trajectory-level agent post-training. This is the dividing line from an ordinary multimodal model. Training data is no longer “image–answer” pairs but complete task trajectories: initial observation, reasoning, action, execution result, next observation… until success or termination.
# A sample trajectory used for agent post-training (illustrative structure)trajectory = {"task": "Complete payment for a pending order in a mobile app","steps": [{"observation": {"image": "screenshot_00.png","ui_tree": "a11y_00.json"},"reasoning": "The home screen has an entry for pending orders; enter the order list first","action": {"type": "tap", "target": [0.50, 0.22]},"outcome": {"status": "success", "next": "screenshot_01.png"}},# ... intermediate steps omitted{"reasoning": "The payment confirmation page has appeared and the amount matches the order","action": {"type": "confirm"},"outcome": {"status": "terminal", "reward": 1.0}}],"meta": {"total_steps": 9, "rollback_count": 1, "latency_ms": 8420}}
Metrics like rollback_count and total_steps give reinforcement learning a much denser reward signal than “final success or failure” alone. A common industry practice is to layer process rewards on top of verifiable rewards: a small reward when an action hits the correct control, a negative reward for ineffective swipes, and a penalty for excessive steps—thereby improving success rates while suppressing “random clicking and trial-and-error.”
Layer 4: Capability retention and anti-forgetting. Multimodal training can easily damage a model’s original pure-text and coding abilities, something often underestimated in practice. Mitigation measures include: keeping a certain proportion of high-quality pure-text and code corpora in the training data for replay; using a lower learning rate or parameter isolation for language-side parameters; and running a text regression evaluation suite after each round of post-training, treating capability regression as a release blocker rather than an option.
4. Toolchain and Engineering: Turning Model Capability into a Stable Service
Model capability is only the starting point; a stable agent system relies on a complete toolchain.
Inference serving and cost governance. The memory and bandwidth pressure of multimodal inference is significantly higher than that of pure text, usually requiring elastic scaling on a container platform, along with separate batching and caching for the vision encoder. One cost-effective technique is perception caching: if two consecutive observations of the same page differ by fewer pixels than a threshold, reuse the previous encoding result directly instead of re-sending it to the model.
Asynchrony and state management. Long tasks (operation chains of dozens of steps or more) should not be bound to a single request. A sound approach is to enqueue tasks into a message queue, advance execution via a state machine, and persist intermediate artifacts (screenshots, action logs, intermediate conclusions) to object storage, making tasks pausable, resumable, and available for human takeover.
Execution sandbox and rollback. Actions issued by an agent must run in an isolated environment and are forbidden from directly operating production systems. A virtualized device environment paired with pre-action snapshots makes rollback a routine capability rather than an incident-handling measure. High-risk actions (transfers, deletions, external publishing) should have secondary confirmation and whitelist interception.
Observability. It’s recommended to uniformly log each step’s input fingerprint, token consumption, model output, action, and execution result in a logging service, and to track a few core metrics in monitoring and alerting: average steps per task, action hit rate, rollback rate, token cost per task, and timeout rate. Without this observability, a multimodal agent is nearly impossible to operate stably.
5. Evaluation System: How to Tell Whether “Native” Actually Holds Up
Evaluating a multimodal agent by end-to-end task success rate alone is insufficient—it masks the reasons for failure. Three layers of evaluation are recommended:
- Perception layer. Element localization accuracy, small-text recognition accuracy, chart value reading error, and the structural similarity of layout reconstruction. If this layer doesn’t meet the bar, everything above it is empty talk.
- Reasoning layer. Question answering over documents with mixed text and images, cross-page information comparison, and UI state inference—with a focus on whether the model answers based only on information that actually exists in the image. This requires a dedicated anti-hallucination evaluation: construct samples where the corresponding information doesn’t exist in the image, and examine whether the model explicitly refuses to answer rather than fabricating.
- Execution layer. End-to-end task success rate, average steps to completion, proportion of invalid actions, and the ability to recover after a task interruption.
In addition, a pure-text regression evaluation must be retained to ensure the introduction of multimodal capability hasn’t come at the expense of fundamental language and coding abilities. The evaluation sets themselves need contamination prevention and should cover real-world variation dimensions such as different resolutions, light and dark themes, and multilingual interfaces.
6. Typical Use Cases and Common Pitfalls
The value of native visual capability concentrates in scenarios where “information inherently lives in the picture”: converting design mockups into usable frontend code, automated operation and regression testing of mobile apps, field extraction from receipts and contracts, cross-checking multiple metrics on data dashboards, and temporal understanding of video surveillance events.
Four pitfalls are most common in practice. First, the resolution trap—downsampling purely to save money, causing all small text to be lost and success rates to drop off a cliff. Second, uncontrolled context—stuffing every step’s raw screenshot into the context, so cost grows exponentially with the number of steps. Third, lack of symbolic redundancy—relying solely on visual coordinate targeting, which fails immediately upon theme switching or delayed rendering. Fourth, missing evaluation—substituting a few demo cases for systematic evaluation, so problems erupt all at once only after going live.
7. Go-Live Checklist
- Does visual input go through a unified token space rather than an independent “image-to-text” middleware?
- Are dynamic resolution and token budget control implemented, with cache reuse for repeated observations?
- Is action output structured, are coordinates normalized, and do actions carry confidence and fallback strategies?
- Does post-training include trajectory-level data and process rewards, with replay of pure-text capability?
- Does execution happen in an isolated sandbox, with confirmation and whitelists for high-risk actions?
- Are intermediate artifacts persisted, and can long tasks be paused, resumed, and handed over to humans?
- Are there three-layer evaluations (perception, reasoning, execution) plus anti-hallucination evaluation?
- Do logs, metrics, and alerts cover step count, hit rate, rollback rate, and per-task cost?
The difficulty of a native multimodal agent was never “connecting a vision model,” but rather getting perception, reasoning, and execution to correct one another within the same representation. Do these four things well—unify the architecture, layer the training, complete the toolchain, and make evaluation solid—and vision turns from a demo highlight into a reliable production capability. Today, as foundational capabilities such as compute, storage, and elastic containers have become highly commoditized (for example, obtaining general-purpose capabilities like object storage, container platforms, and logging and monitoring on demand through cloud platforms such as Baidu AI Cloud), a team’s differentiated competitiveness comes more from refining the data loop and engineering details than from the choice of a single model.
