Beyond Text and Vision: How the New Generation of Multimodal Agent Models Executes Actions
This article analyzes how the latest multimodal agent models are being upgraded for real-world agentic work. Instead of treating vision and language as two stitched-together systems, these models unify visual and linguistic inputs in a shared semantic space and turn visual information into executable action sequences. The article covers the core limitations of traditional multimodal agent models, the three-stage unified architecture (visual encoding, cross-modal projection, unified backbone modeling), Mixture-of-Experts backbones, native-resolution and arbitrary-aspect-ratio visual encoding, joint text-vision optimization, structured tool calling and result observation, multi-agent parallel orchestration to reduce latency, and the engineering systems needed for production: evaluation, observability, and security. It closes with practical guidance for developers on designing reliable agent applications.
Introduction: From “Understanding Content” to “Completing Actions”
Over the past year, the evaluation of multimodal large models has largely focused on perception-oriented tasks such as image-text understanding, visual question answering, and image captioning. As agent applications move deeper into real business scenarios, however, letting a model “understand an image” or “read a paragraph of text” is no longer enough. What developers care about now is different: Can the model grasp a goal within a complex context? Can it invoke the right tools? Can it revise its plan based on feedback? Can it complete a multi-step task within an acceptable amount of time?
As a result, the technical upgrade of the new generation of multimodal agent models is not simply about strengthening the vision module or adding more tool interfaces. It is a systematic redesign built around the complete loop of “perception—understanding—planning—execution—verification.” Three key changes stand out. First, vision and language are no longer two systems concatenated one after another; they are co-modeled within a unified semantic space. Second, the model does not just recognize visual content; it can convert visual information into executable action sequences. Third, when facing complex tasks, executing everything sequentially with a single agent introduces high latency, so task decomposition, parallel scheduling, and result aggregation mechanisms are needed.
This article examines these changes across architecture design, visual encoding, joint training, tool calling, agent orchestration, and engineering implementation.
1. Core Problems Facing Traditional Multimodal Agent Models
1.1 Vision and language capabilities easily trade off against each other
Early multimodal model approaches often trained a pure-text language model first, then attached a visual encoder and relied on post-hoc fine-tuning to enhance vision capabilities. This method has low engineering cost, but it tends to produce two kinds of problems:
- Degradation of language ability: After large numbers of visual parameters are injected, the model’s performance on complex text reasoning, code generation, and mathematical calculation may decline.
- Shallow visual understanding: Visual input gets compressed into a limited number of visual tokens, and fine-grained text, chart structure, and spatial relationships are easily lost.
In agent scenarios, these problems are amplified. For example, a model may need to read button labels in an interface, numbers in a table, and arrow relationships in a flowchart, then decide the next action based on them. If visual information is already distorted at the encoding stage, subsequent planning and tool calling become hard to do reliably.
1.2 Tool calling and visual observation are disconnected
Agent tasks are usually not one-shot question answering, but a loop of “observe the environment—form a plan—call a tool—get feedback—continue reasoning.” If a model can only understand images but cannot output structured actions, or can only call text interfaces without combining the results of visual observation, a capability gap emerges.
For example, when facing a business dashboard page that contains multiple regions, a model needs to complete the following actions at once:
- Identify the current page state;
- Determine the location of the target entry point;
- Generate structured operation instructions;
- Invoke an executor to perform clicks, input, or queries;
- Judge whether the action succeeded based on the returned result.
If visual encoding, language reasoning, tool calling, and result verification are not modeled together, the model can only perform well on local tasks and struggles with long-chain tasks.
1.3 Single-agent serial execution leads to high latency
Complex agent tasks often contain multiple sub-steps. If a single agent executes them sequentially, where each step depends on the output of the previous one, end-to-end latency grows linearly. In composite tasks combining visual analysis, web operation, code execution, and data queries, this serial waiting significantly affects the experience.
For example, suppose a task requires checking three reports at once, calling two query interfaces, and generating a summary. If each sub-task runs one after another, the total time may be far higher than the theoretical minimum. In this case, parallelizing tasks becomes a necessary capability.
2. Unified Multimodal Architecture: Bringing Vision and Language into the Same Modeling Space
An important direction for the new generation of multimodal agent models is to bring visual, text, and video inputs into a unified architecture for joint modeling, rather than treating vision as an add-on module to the language model.
2.1 A three-stage architecture: visual extraction, cross-modal projection, unified backbone modeling
From an engineering standpoint, this kind of architecture can usually be abstracted into three stages:
Visual feature extraction
Image or video input enters the visual encoder and produces a sequence of visual features. The encoder needs to preserve high-resolution detail and support different aspect ratios and input lengths.Cross-modal projection
A projection layer maps visual features into the language model’s embedding space, so that visual tokens can participate in attention computation alongside text tokens within the same representation space.Unified backbone modeling
The unified backbone handles the mixed sequence of text, image, and video tokens and carries out semantic understanding, reasoning, code generation, action planning, and result judgment.
The following simplified pseudocode illustrates the idea:
visual_features = visual_encoder(image_or_video)visual_tokens = projection_layer(visual_features)text_tokens = text_encoder(user_prompt)model_input = concatenate(text_tokens, visual_tokens)hidden_states = unified_backbone(model_input)output = language_head(hidden_states)
The core value of this design is that visual information is no longer just an “external input” but enters the attention mechanism together with language tokens. The model can compare text descriptions, image details, and structured instructions within the same context, which improves reasoning stability in complex scenarios.
2.2 Mixture-of-Experts backbones change how parameters are used
Within the unified backbone, the Mixture-of-Experts (MoE) architecture is widely used to improve inference efficiency for large-parameter models. The idea is that different input modes or semantic regions can activate different expert modules, avoiding the need for all parameters to participate in computation at every step.
In multimodal agent scenarios, this pattern is well suited:
- Text reasoning tasks may activate more of the language- and logic-related parameters;
- Image recognition tasks may activate more of the vision-semantic parameters;
- Tool calling and structured output may rely on specific formatting and action-planning capabilities;
- Long-context tasks need to stably retain key information.
By selecting expert combinations through a routing mechanism, the model can keep its overall parameter scale while minimizing the compute cost of a single pass. This matters especially for high-frequency agent calls and long-chain reasoning.
2.3 What unified modeling means for agent capabilities
A unified multimodal architecture is not just about improving image-captioning accuracy; it gives the model the following capabilities:
- Cross-modal coreference resolution: understanding “click the third red button in the top-right corner” requires combining spatial position, color, and interface structure at once.
- Chart and table reasoning: reading a trend from a bar chart and then making a decision based on a text goal.
- Video event understanding: recognizing operation steps, phase changes, or anomalous segments.
- Visual result reflection: judging whether an action succeeded based on the screenshot, log, or state returned by a tool.
- Multimodal instruction following: handling image constraints, text requirements, and output-format restrictions together.
Together, these capabilities form the foundation that lets an agent move from “understanding content” to “completing tasks.”
3. Native-Resolution Visual Encoding: Reducing Information Loss
The quality of visual understanding often depends on how much effective detail is preserved in the input. Traditional visual preprocessing commonly resizes images to a fixed size, which is common in object detection or classification, but causes clear problems in agent scenarios.
3.1 Fixed resolution causes loss of key information
The visual inputs an agent faces vary greatly, including:
- High-resolution webpage screenshots;
- Long scanned document pages;
- Complex charts;
- Small-font text in user interfaces;
- Key frames from surveillance video;
- Information panels containing multiple regions.
If these are forced to scale down to a fixed size, small fonts, thin lines, and local buttons can become blurry, and the model struggles to recognize them accurately. For scenarios that require precise operation of interface elements, this loss directly affects action generation.
3.2 Variable resolution and block-packing strategies
A more reasonable approach is to support native or near-native resolution input. One concrete method is block packing:
- Divide the image into multiple local blocks;
- Encode each block separately;
- Organize the block features into a sequence following spatial order;
- Combine position encodings to restore spatial relationships.
This avoids compressing the entire image into a single low-resolution representation and instead balances detail preservation against sequence length.
A simplified view:
image -> split into patchespatches -> patch encoderpatches -> sequence with spatial positionsequence -> visual tokens
3.3 Why arbitrary aspect-ratio support matters
Images in real business settings are rarely in standard proportions. Interface screenshots, scanned documents, photographed charts, and video frames often have different aspect ratios. If a model can only accept a fixed input shape, the preprocessing stage must crop or scale, which easily causes content loss or distortion.
The value of native aspect-ratio support includes:
- Reducing information loss caused by cropping;
- Preserving table column structure and paragraph layout;
- Improving the localization accuracy of interface elements;
- Improving the correspondence between coordinates, labels, and legends in charts;
- Lowering the preprocessing complexity on the business side.
3.4 Unified video modeling and key-frame selection
Video understanding cannot simply be treated as stacking multiple images together. Video contains a time dimension and needs to handle several problems:
- How to select key frames;
- How to represent temporal changes;
- How to align the action sequence with text instructions;
- How to control the number of tokens in long videos;
- How to avoid redundancy caused by repetitive information between frames.
Common approaches include key-frame extraction, temporal position encoding, cross-frame feature aggregation, and visual-token compression. In agent scenarios, video understanding is often used for tasks such as tutorial video analysis, understanding of screen-recorded operations, and surveillance event recognition. The model needs to do more than recognize “what is in the frame”; it must also judge “what to do next” or “whether the previous step has been completed.”
4. Joint Optimization of Text and Vision: Achieving Two-Way Enhancement
The key change in the new generation of multimodal models is not just adding visual input, but using joint training so that language and vision reinforce each other.
4.1 From post-hoc adaptation to end-to-end joint optimization
Traditional approaches usually split into two steps:
- Train text capabilities first;
- Then add a visual module for adaptation.
This pattern easily produces modality conflicts. Visual training can shift the representation distribution of the language model, causing originally strong language abilities to decline. Conversely, if language ability is over-protected, fine-grained visual understanding fails to develop fully.
The goal of end-to-end joint optimization is to account for the following simultaneously during training:
- Text understanding and generation;
- Recognition of visual details;
- Image-text alignment;
- Structured output;
- Tool-calling formats;
- Multi-turn task planning.
Through a unified loss function and mixed multi-task data, the model can learn to establish a stable mapping between the language space and the visual space.
4.2 Vision enhances language reasoning
Visual information is not only used for visual question answering. In many tasks, an image is part of the reasoning conditions. For example:
- Judging a current path from a circuit diagram;
- Identifying an anomalous node from a flowchart;
- Determining the operable region from an interface layout;
- Predicting business changes from a chart trend;
- Summarizing a table in a scanned document.
Once visual tokens enter the backbone alongside text tokens, the model can use visual details as contextual evidence in reasoning, rather than only producing surface-level descriptions.
4.3 Language enhances visual localization
Language capability in turn affects visual understanding. The model needs to understand the user’s goal in order to decide which parts of an image to focus on.
For example:
- “Find the region where the order status is abnormal”;
- “Locate the fields involved in a configuration error”;
- “Select a screenshot region suitable as the next input”;
- “Find, among several charts, the one related to a drop in the metric.”
These tasks require the model to translate natural-language constraints into visual attention, and then form an executable operation. The clearer the language instruction, the more stable the visual localization; the more accurate the visual result, the more reliable the language plan.
4.4 Agent benefits after joint optimization
The direct benefits of joint optimization include:
- More stable image-text alignment;
- Higher accuracy in recognizing interface elements;
- Better memory of visual context;
- Stronger structured-output ability;
- More reliable generation of tool-calling parameters;
- Smoother multi-turn task correction.
This shifts the model from “passively answering” to “actively executing,” which is central to the agent capability upgrade.
5. Tool Capability Upgrades: From Generating Content to Producing Actions
The tool capability of a multimodal agent model is the bridge connecting model understanding with real systems. Tool capability upgrades usually show up in three areas: tool descriptions, parameter generation, and result feedback.
5.1 Tool descriptions should be model-readable
Tool interfaces should not only serve traditional programmatic calls; they should also be easy for a model to understand. A tool description suitable for agent use typically includes:
- The tool’s purpose;
- The meaning of input parameters;
- Parameter types and constraints;
- Return-value format;
- Failure scenarios;
- Permission scope;
- Preconditions for use.
Example structure:
tool name: query_business_recordpurpose: query business records by user IDinput parameters:user_id: unique user identifierdate_range: query date rangereturned fields:status: business statusamount: amounterror_code: failure reasonconstraints:only authorized users may be queriedreturn at most 100 records per call
This kind of description helps the model judge “when to use it, how to use it, and what to do after a failure” in complex tasks.
5.2 Structured action output improves reliability
When an agent performs tool calls, its output should not be just a natural-language suggestion; it should be as structured as possible. Structured output is easy for the executor to parse and easy to validate.
Typical actions can include:
- Calling a query interface;
- Reading a file;
- Writing a configuration;
- Starting a task;
- Submitting a form;
- Returning an intermediate result;
- Requesting human confirmation.
In terms of architecture design, the model output can be constrained to a fixed schema:
action_type: call_tooltool_name: query_statusarguments:task_id: abc123
For high-risk actions, a confirmation mechanism should also be introduced:
if action.risk_level == high:require human_approval()
5.3 Observation results drive the next round of planning
The core of agent capability is not a single call but multiple rounds of observation and correction. After a tool returns, the model needs to judge:
- Whether the call succeeded;
- Whether the returned content is complete;
- Whether there is a permission error;
- Whether the parameters need to be changed;
- Whether additional tools are needed;
- Whether the task can be ended.
A simplified flow:
goal = receive_user_goal()state = load_context()while not is_finished(state):plan = model.plan(goal, state)action = model.choose_action(plan)result = tool_executor.run(action)state = model.observe(state, result)return model.summarize(state)
This loop is what distinguishes an agent from an ordinary conversational model.
6. Multi-Agent Parallel Orchestration: Easing Latency for Complex Tasks
As the number of task steps grows, single-agent serial execution becomes a performance bottleneck. The new generation of models is beginning to introduce multi-agent collaboration or parallel orchestration, decomposing complex tasks into multiple sub-tasks that can run in parallel.
6.1 The limitations of single-agent serial execution
The single-agent mode suits tasks with clear logic and strong step dependencies, but it falls short in scenarios such as:
- Analyzing multiple documents at once;
- Querying multiple data sources at once;
- Checking multiple page states in parallel;
- Evaluating multiple candidate approaches across branches;
- Batch visual recognition;
- Joint retrieval across multiple tools.
If all sub-tasks run sequentially, every step must wait for the previous one to finish, and overall latency rises noticeably.
6.2 The basic structure of multi-agent orchestration
A typical multi-agent orchestration framework can include the following roles:
- Planner agent: understands the overall goal, decomposes the task, and determines dependencies.
- Executor agents: handle specific sub-tasks such as visual recognition, code execution, and data queries.
- Aggregator agent: merges multiple execution results, resolves conflicts, and produces the final output.
- Verifier agent: checks result completeness, format correctness, and business compliance.
Simplified pseudocode:
task_graph = planner.decompose(user_goal)independent_tasks = select_independent_nodes(task_graph)results = parallel_execute(independent_tasks)merged_state = aggregator.merge(results)final_result = verifier.check(merged_state)if not final_result.passed:planner.replan(failed_nodes)
6.3 Parallelism does not always apply
While parallel orchestration can reduce latency, task dependencies and consistency must be considered. Tasks suited to parallelism typically share these characteristics:
- Sub-tasks have no strong dependencies on one another;
- Sub-tasks can be retried independently;
- Sub-task results can be merged structurally;
- Sub-tasks do not cause resource conflicts;
- A sub-task failure can be rolled back locally.
Tasks that should not be blindly parallelized include:
- High-risk operations that write to the same resource;
- Business approvals that require strict ordering;
- Steps with strong causal dependency between them;
- Cases where parallel results could overwrite data;
- Scenarios with complex permission boundaries that cannot be isolated.
So the key to multi-agent orchestration is not “the more the better,” but “whether the task graph is reasonable, whether scheduling is reliable, and whether results can be verified.”
6.4 Important takeaways for developers
When implementing parallel agents, it is recommended to prioritize the following designs:
- Clearly define task nodes and their dependencies;
- Set timeouts and retry strategies for each sub-task;
- Add idempotency control to tool calls;
- Preserve local state for failed nodes;
- Introduce human confirmation for high-risk actions;
- Apply unified validation to output results.
This can reduce latency while avoiding the uncontrollable risks that parallelism can introduce.
7. Engineering Implementation: Evaluation, Observability, and Security
For a multimodal agent model to enter production, you cannot look at model capability alone; you also need to build a complete engineering system.
7.1 Evaluation dimensions should cover the full chain
A reasonable agent evaluation system usually includes:
- Visual understanding: image recognition, interface-element localization, chart reading.
- Language reasoning: long-context understanding, instruction following, multi-turn dialogue.
- Code and math: logical computation, code generation, error correction.
- Tool calling: parameter correctness, call success rate, exception handling.
- Task completion: end-to-end success rate, average number of steps, human intervention rate.
- Latency and cost: time to first step, total elapsed time, resource consumption.
- Security: permission control, sensitive-information filtering, blocking of out-of-scope operations.
Evaluation should not rely only on static question answering; it should also include real task chains. For example:
input: a screenshot containing an order pagegoal: query anomalous orders and generate handling recommendationsrequirement: do not access other users' data beyond your authorization
This kind of task can test visual understanding, tool calling, permission judgment, and result generation at the same time.
7.2 Standard cloud capabilities are enough to support engineering implementation
When deploying a multimodal agent in the cloud, you can build a complete pipeline using general-purpose cloud capabilities, for example:
- Object storage: storing images, videos, logs, and model artifacts.
- Message queues: decoupling asynchronous tasks and tool calls.
- Container platforms: deploying model services, executors, and schedulers.
- Logging services: recording tool calls, model outputs, and exception information.
- Monitoring and alerting: observing latency, error rate, resource usage, and task success rate.
These capabilities are not unique to any single proprietary platform; they are common building blocks of a modern cloud architecture. By combining standardized components, you can build an elastic, observable, and scalable environment for running agents.
7.3 Observability determines whether an agent can iterate stably
Debugging an agent system is harder than debugging an ordinary model service, because its output triggers real actions. It is therefore necessary to record the complete chain:
- User input;
- Visual preprocessing results;
- Model plan;
- Tool-call parameters;
- Tool return results;
- Model retry reasons;
- Final output;
- Human feedback.
Without this observability, it is very hard to pinpoint whether a problem comes from visual recognition, language planning, or tool execution.
7.4 Security boundaries must be set up front
Once an agent gains the ability to call tools, security design must be brought to the front. At a minimum, it should include:
- The principle of least privilege;
- Tool whitelists;
- Second confirmation for sensitive operations;
- Content review of inputs and outputs;
- Sandboxed execution environments;
- Call-rate limiting;
- Rollback mechanisms for exceptions.
Especially in scenarios involving file writes, configuration changes, business submissions, and calls to external interfaces, model output should not be treated as directly executable instructions. It should instead pass through a policy engine or executor for validation.
8. Practical Advice for Developers
8.1 Optimize the task structure, not just the prompt
Unstable agent results are often not due to a weak prompt but to an unclear task structure. It is recommended to first define:
- What the input is;
- What the output is;
- What the intermediate steps are;
- What the acceptance criteria for each step are;
- How to retry on failure;
- Which steps need human confirmation.
8.2 Visual input quality sets the ceiling
If the visual input is too blurry, poorly cropped, or too low in resolution, even a strong model downstream will struggle to make up for it. Pay attention to:
- Whether the screenshot is complete;
- Whether the text is clear;
- Whether the table structure is preserved;
- Whether the page proportions are natural;
- Whether key regions are obscured.
8.3 Context management matters more than simply enlarging the window
Agent tasks often produce a large number of intermediate results. Stuffing all of them into the context increases latency and cost. You can use:
- Summaries of intermediate results;
- Extraction of key fields;
- State retained across rounds;
- Deduplication of historical tool calls;
- Compression of low-relevance observations.
8.4 Tool interfaces should be designed to be agent-friendly
Tool returns should not be overly verbose or ambiguous. It is recommended to:
- Return structured fields;
- Define clear error codes;
- Avoid meaningless logs;
- Desensitize sensitive fields;
- Provide explainable failure reasons.
8.5 Build a small-scale regression evaluation set
After each change to the model, prompt, tool description, or scheduling strategy, you should run a regression against a fixed evaluation set. The set should cover:
- Common tasks;
- Boundary tasks;
- Failure-recovery tasks;
- Permission-restricted tasks;
- Multi-turn interaction tasks.
Only in this way can you avoid a situation where a local optimization causes an overall regression.
9. Summary
The core of the upgrade to the new generation of multimodal agent models is not the enhancement of any single capability point, but the construction of a closed-loop system that can simultaneously handle vision, language, tool calling, and task planning. Its key highlights can be summarized as:
- A unified multimodal architecture that brings vision and language into the same semantic space;
- Native-resolution and variable-input visual encoding that reduces information loss caused by preprocessing;
- Joint text-vision optimization that achieves two-way capability enhancement;
- Structured tool calling and result observation that improve real-task execution ability;
- Multi-agent parallel orchestration that eases the latency of complex tasks;
- Engineering-based evaluation, observability, and security mechanisms that ensure production readiness.
For developers, understanding these technical points helps in designing agent applications more clearly: the focus should not only be on whether the model “can answer,” but also on whether the model “can act reliably in a complex environment.” As visual encoding, tool protocols, task scheduling, and security governance continue to mature, multimodal agents will deliver greater value in scenarios such as document processing, business-process automation, content-production assistance, data analysis, and interactive system operation.
