Native Multimodal Agents: Architecture, Training, and Engineering Implementation
Bolting a vision model onto a text LLM as a preprocessing step is easy to build but structurally limited. This article explains what it takes to treat vision as a native capability in a multimodal agent. It covers six criteria that separate native designs from bolt-on ones, the architecture choices for unified representation and token budgeting, a four-stage training curriculum with anti-forgetting techniques, the perception–planning–execution–reflection toolchain, a three-axis evaluation framework, layered cloud deployment with cost governance, and the common pitfalls and phased rollout that decide whether an agent holds up in production.
1. Where the Problem Starts: Why “Bolt-On” Multimodal Can’t Carry an Agent
Over the past two years, the mainstream approach has been to treat a mature vision model as a preprocessing module: a screenshot goes in, the vision model outputs a natural-language description, and a text LLM makes decisions from that description. This pipeline is easy to build, but it has three structural flaws.
First, irreversible compression of the observation. The description a vision model produces is natural language written for human readers. It preserves semantics but systematically drops the meta-information an agent actually needs: pixel coordinates of elements, stacking order, clickable regions of controls, the precise spacing between text and icons, and the current scroll position. Once that information is discarded at the encoding stage, no amount of downstream reasoning can recover it.
Second, misaligned objectives between perception and reasoning. The description model doesn’t know what the downstream task is. When the task is “recreate a design as code,” the attributes that matter are size, grid, and color. When the task is “help a user complete a refund on their phone,” the attributes that matter are control state and interactivity. A fixed description style can’t shift its focus per task, which produces the classic failure where “the model answers fluently but the click lands half a control off.”
Third, error cascading and stacked latency. Serial calls amplify perception errors at each level, while multiple network round trips push up end-to-end latency. For an agent that needs dozens of consecutive steps, this latency is unacceptable.
2. What Counts as “Native Multimodal”
Before discussing implementation, it’s worth defining what makes a system treat vision as a native capability. I use six criteria to separate native from bolt-on:
- A unified token space: image patches and text tokens interact within the same self-attention layer, rather than two towers encoding separately and then concatenating;
- A controllable visual resolution strategy: the system can decide dynamically how finely to look, rather than being forced to accept a fixed 224/336 resolution;
- A structured output space: the output is not just text but constrained actions (coordinates, clicks, typing, scrolling, termination), with grammar-constrained decoding to guarantee legality;
- Joint training: vision, text, and action trajectories are mixed and optimized in a single training curriculum, rather than fine-tuned independently per stage;
- An observation feedback loop: at inference time the model can call tools and feed execution results back in as new multimodal observations;
- No regression in pure-text capability: after adding vision, existing abilities such as code generation and long-form reasoning must hold their baseline under regression tests.
These six also serve as the reference table for the architecture and training design below.
3. Architecture: Unified Representation and Token Budget
3.1 Dynamic Resolution and Tiling for Visual Input
High resolution is a hard requirement for UI understanding and document parsing: the text on a button rendered at 12px is nearly indistinguishable on a downscaled global thumbnail. The common industry approach is a dual-path strategy of “global thumbnail + local tiles”—first provide a low-resolution global image so the model builds an overall layout understanding, then cut the image into high-resolution tiles along a row/column grid to fill in detail.
The key engineering problem here is the token budget. Suppose the available context is 128k, text and history occupy 30k, and 1k must be reserved for output; then the amount of budget left for vision determines how many tiles you can cut:
# Illustrative: allocate visual tokens and tile count from the remaining budgetdef allocate_visual_tokens(ctx_limit, text_tokens, reserved_output=1024, per_tile=256):budget = ctx_limit - text_tokens - reserved_outputtiles = max(1, budget // (per_tile + 1)) # +1 reserves a slot for the global thumbnailreturn tiles, per_tile
The recommended approach is progressive refinement: use a low-resolution global image first to judge “where needs a closer look,” then re-cut only the referenced regions at high resolution. This is more token-efficient than blindly cutting a 3×3 grid, and it better matches the human behavior of “glancing first, then leaning in.”
3.2 Position Encoding and Spatiotemporal Alignment
Once image patches enter the sequence, they must carry 2D spatial information, or the model can’t understand “top-left” or produce coordinate outputs. A common approach extends rotary position encoding with 2D (row, column) components, and for video adds a timestamp component so the model knows that frame 37 occurs at 3.7 seconds. This design matters especially for video agents: judging “in which second a popup appears” depends on explicit encoding of the time axis, not implicit inference from frame order.
3.3 Three Routes to Fusion
- Projection alignment: the vision encoder output is mapped into the language space through a lightweight projection layer. Simple structure, stable training, and the current mainstream.
- Cross-attention: visual features are injected as additional K/V into the language layers. Larger capacity but noticeably higher inference cost.
- Early fusion: vision and text tokens are interleaved from the very first layer. Strongest representation but the highest training cost.
From a cost-performance standpoint, projection alignment + high-resolution tiling + sufficient alignment data already covers the vast majority of UI and document scenarios.
3.4 Compressing Video and Long Observations
Video naturally causes token explosion. Three practical layers help: on the sampling side, combine uniform sampling with keyframe triggers (sample extra frames when the picture changes beyond a threshold); on the content side, use inter-frame differencing to keep only the regions that changed; on the semantic side, aggregate frames within a time window into “event summaries,” falling back to raw frames only when precise actions are needed. Combining the three usually compresses the effective tokens of long video down to a trainable scale.
4. Training: A Four-Stage Curriculum with Anti-Forgetting
Stage 0: Image-Text Alignment Pretraining
Data is primarily interleaved image-text documents (web pages, slide decks, scanned documents, annotated datasets). The goal is to align the vision encoder with the language space and give the model basic OCR and chart-reading ability. This stage should deliberately retain a large amount of text corpus so that language ability doesn’t collapse first.
Stage 1: Multi-Task Supervised Fine-Tuning
Task design should cover four capabilities: “see–say–localize–reconstruct”—visual question answering and document parsing (see and say), referring localization and control recognition (localize), and UI-to-code generation (reconstruct). Localization tasks must use coordinate-form answers; this is the foundation for later GUI-operation ability—a model without grounding data can’t click accurately no matter how accurate its descriptions are.
The format of action output must be fixed before training and enforced at inference time with constrained decoding:
{"thought": "Currently on the order detail page; the refund entry is at the bottom-right of the card","action": { "type": "tap", "point": [0.812, 0.436] },"expected": "Page title changes to 'Refund Request'"}
The expected field is a cheap but extremely valuable self-check hook: after execution, compare it against the actual observation, and any mismatch triggers a reflection branch.
Stage 2: Preference Optimization
Use rejection sampling to construct positive/negative sample pairs, where positives are trajectories with “fewer steps, smaller localization error, and eventual task completion.” There’s a common pitfall here: rewarding only “task completion” induces the model to learn shortcut-style destructive operations (for example, closing a popup outright instead of handling it through the proper flow). The preference pairs must explicitly include compliance constraints.
Stage 3: Agent Trajectory Reinforcement Learning
Rely primarily on verifiable rewards: whether the task is accomplished and whether assertions pass are objective signals; process rewards include distance from the localization center point, penalties for invalid actions, and step-count penalties. Failure trajectories collected in real environments are extremely valuable—they carry recovery-path knowledge and are the only source from which a model learns “what to do after a wrong click.”
You must also do text replay: mix a certain proportion (empirically 20%–30%) of pure-text and code corpus into every training batch, or the gains in visual ability will often come with a clear regression in code and reasoning ability.
5. Toolchain: The Perception–Planning–Execution–Reflection Loop
A trained model still needs a reliable execution skeleton. The core loop looks like this:
history = []for step in range(MAX_STEPS):obs = capture() # pixel screenshot + optional accessibility tree/DOMdecision = model.plan(obs, goal, history) # constrained decoding guarantees legal actionsif decision.action.type == "done":return verify(obs, goal)result = executor.run(decision.action, expect=decision.expected)if result.mismatch:history.append(reflect(obs, decision, result)) # record the deviation, enter the recovery branchhistory = compress(history) # sliding window + summaries + archived key screenshotsif step > MAX_STEPS: break
A few engineering details decide success or failure:
Observation fusion. Pixel observations preserve raw information but are expensive to interpret; structured observations (accessibility trees, DOM) are readable but often incomplete or missing fields. Using pixels as primary and structure as auxiliary for cross-checking significantly reduces the misclick rate.
Coordinate conversion. When normalized coordinates are mapped back to device pixels, you must account for screenshot scaling ratio, DPI, and status-bar and safe-area offsets. This kind of 1px-level error is the main source of the “clicked but nothing happened” failures in production, so handle it uniformly in the adaptation layer rather than scattering it across the codebase.
Failure recovery and loop prevention. After each action, judge whether the page changed as expected; if two consecutive steps produce no change, force a rollback or a fresh observation. Retry counts, timeouts, and idempotency need explicit design.
Memory compression. Context grows fast on long tasks. The strategy is to keep the raw observations of the most recent steps, summaries of earlier steps, and references to all key screenshots that were cited (stored in object storage and fetched by reference when needed).
Safety boundaries. For irreversible operations such as payments, deletions, and publishing, force an interruption and request human confirmation; the execution environment needs sandbox isolation and least-privilege permissions, and account, ID-document, and monetary information in screenshots should be masked.
6. Evaluation: Measure Three Capabilities Separately
Don’t evaluate a multimodal agent with a single aggregate score; split it into at least three groups of metrics:
| Capability | Representative Tasks | Key Metrics |
|---|---|---|
| Visual understanding | Document parsing, chart reading, UI question answering | Field-level accuracy, OCR character error rate |
| UI reconstruction | Design-to-code | Structural match, visual similarity, runnability rate |
| End-to-end agent | Multi-step cross-app operation | First-turn success rate, average steps, human-takeover rate, token cost per task |
Two regression items must be held: the pure-text capability baseline (code generation, long-form reasoning) and action legality (constrained-decoding failure rate of 0). Also watch out for evaluation-set leakage and distribution bias—a distribution shift along any dimension of UI text, language, resolution, or OS version will disconnect offline scores from online performance, so online sampling and replay matter more than a one-time offline score push.
7. Cloud Deployment: Layered Architecture and Cost Governance
In a public cloud environment (for example, on platforms like Baidu AI Cloud), a usable multimodal agent service is typically layered like this:
- Object storage holds screenshots, video, and trajectory datasets, managed by reference so that large objects stay out of message bodies;
- A message queue decouples the collection side from the inference side, smoothing peaks and filling troughs while supporting failure replay;
- A container platform runs the target applications and the execution sandbox, elastically starting and stopping per task with natural isolation;
- Logging plus monitoring and alerting build a quality dashboard tracking end-to-end latency P95, action legality rate, token consumption per task, and human-takeover rate;
- On the inference-service side, batching, prefix caching, elastic scaling, and rate-limiting quotas control cost and stability.
The lever for cost optimization is tiered visual budgeting: judge quickly at low resolution first, and upgrade only the decision-point regions to high resolution; cache observations and deduplicate by hash for repeated pages; steps that can be solved deterministically with rules (such as a fixed login flow) don’t need the model at all. In practice, this kind of combination usually compresses per-task token consumption to a fraction of a naive implementation.
On compliance, three things need to be settled in advance: the classification and retention period of screenshot data, the coverage of masking rules, and end-to-end operation auditing (who triggered which irreversible action, and when).
8. Common Pitfalls
- Global resolution too low, so small text and icon states can’t be read—solve resolution first, then talk about model capability;
- Doing image-text alignment without grounding data, so the model describes accurately but clicks off-target;
- Adding visual capability without text replay, so code and reasoning ability quietly regress;
- A reward function that rewards only completion, so the model learns destructive shortcuts;
- Coordinate conversion scattered across the business layer, causing failures to erupt all at once when adapting to new devices;
- No failure recovery or step limit, so long tasks fall into ineffective loops and keep incurring charges;
- A single evaluation set, so offline scores rise but online performance doesn’t.
9. Recommended Rollout Path
I recommend proceeding in three steps: in the POC stage, use 50–100 self-built golden tasks to validate single-step localization accuracy and basic task success rate; in the canary stage, connect to real business and focus on observation fusion, failure recovery, and safety confirmation while building online sampling and replay; only in the scale-up stage invest training resources, use real failure trajectories for preference optimization and trajectory reinforcement learning, and simultaneously establish a cost dashboard and a quality-regression pipeline.
The value of native multimodality lies not in any single metric but in compressing perception, reasoning, and execution into one optimizable pipeline: every improvement to the model translates directly into higher task success rate, instead of staying at the level of “look at a picture and talk about it.”
