Co-designing Visual Understanding and Tool Calling for Multimodal Agents
Multimodal agents must seamlessly integrate visual perception with tool execution to perform complex, multi-step tasks. However, traditional architectures often face bottlenecks such as modality conflicts, information loss during image preprocessing, and sequential tool-calling latency. This article explores a natively unified vision-language architecture, staged training strategies, and parallel orchestration techniques to optimize the collaborative design of visual understanding and tool calling in multimodal agents.
1. Background
Multimodal agents are systems capable of simultaneously understanding images, videos, and text, while leveraging tools to complete multi-step tasks. Typical scenarios include reading UI screenshots to execute software operations, performing data analysis based on charts, and locating events in videos before calling a retrieval API. These tasks impose two main requirements on the system: visual information must be accurately perceived, and tool calls must occur at the correct time and in the correct sequence.
In engineering practice, these two requirements constrain each other. Visual perception quality depends on the alignment between the visual encoder and the language model. Tool calling quality depends on the model’s instruction-following and planning capabilities. Meanwhile, the overall task latency depends on whether the execution process can be parallelized. Optimizing only one aspect will still leave the overall performance limited by the weakest link.
2. Bottlenecks of Traditional Approaches
A common industry practice is to first train a text-only language model, then attach a pre-trained visual encoder, and align them using a small amount of adaptation data. While this approach has a short development cycle, it presents three main issues:
First, modality conflict. Visual encoders primarily learn pixel-level statistical patterns, whereas language models learn symbolic sequence patterns. When stitched together later, their representation spaces can only be勉强 aligned through a projection layer, making language capabilities prone to degradation during visual adaptation.
Second, preprocessing loss. Many visual input pipelines scale or crop images to a fixed size. Once the aspect ratio is distorted, critical information such as fine text, table lines, and UI controls may be lost.
Third, video branching. If a separate module is created for video, image capabilities struggle to transfer naturally to temporal understanding, which also increases maintenance costs.
3. Natively Unified Vision-Language Architecture
For agent-oriented multimodal models, a more robust approach is to unify the representation of vision and language at the architectural foundation. A typical pipeline can be broken down into four steps: vision encoding, cross-modal projection, unified language backbone modeling, and task output.
3.1 Vision Encoding
The visual encoder receives an image or video and outputs a set of uniformly formatted visual features. The key is that the input should not be forcibly normalized to a fixed resolution.
3.2 Native Resolution Input
By splitting images into fixed-size patches, images of different dimensions will generate patch sequences of varying lengths. By concatenating these sequences into a 1D sequence in batches and recording the boundaries of each image, inputs of arbitrary aspect ratios can be processed in a single forward pass. Attention calculation is performed only within the same image, and boundary information is used to mask cross-image attention.
Here is a schematic code snippet. Real-world systems typically scale proportionally and pad to an integer multiple of the patch size rather than directly stretching the image.
import numpy as npdef patchify(image: np.ndarray, patch: int = 14) -> np.ndarray:"""Split an H x W x C image into an (N, patch*patch*C) patch sequence. H and W must be multiples of the patch size."""h, w, c = image.shapeassert h % patch == 0 and w % patch == 0, "Dimensions must align to integer multiples of the patch size"x = image.reshape(h // patch, patch, w // patch, patch, c)x = x.transpose(0, 2, 1, 3, 4)return x.reshape(-1, patch * patch * c)def pack_images(images: list[np.ndarray], patch: int = 14):"""Concatenate multiple images of varying sizes into a 1D sequence, returning the start and end positions of each image in the sequence."""seqs, bounds, start = [], [], 0for img in images:p = patchify(img, patch)seqs.append(p)bounds.append((start, start + p.shape[0]))start += p.shape[0]return np.concatenate(seqs, axis=0), bounds
3.3 Spatiotemporal Unified Modeling for Video
Video can be viewed as an extension of images along the time dimension. By organizing consecutive frames into spatiotemporal blocks, intra-frame patches and inter-frame patches are packed together, and the same set of encoder parameters handles attention modeling across both spatial and temporal dimensions. This allows images and videos to share weights, enabling spatial knowledge learned from image data to transfer directly to video understanding without designing a separate video branch.
Note that the number of frames in a spatiotemporal block determines the sequence length, and increasing the frame count leads to a roughly linear increase in computation. Therefore, real-world systems must balance temporal resolution with inference costs and select sampling strategies based on task requirements.
3.4 Cross-modal Projection and Unified Sequence
The encoder output is projected into the language model’s embedding space via a multilayer perceptron (MLP). It is then interleaved with text tokens according to their original positions and fed into the same language backbone. During end-to-end training, loss signals can backpropagate to both the projection layer and the visual encoder simultaneously, allowing the two modalities to adapt to each other.
4. Training Strategy: Decoupled and Joint Optimization
A common challenge in multimodal training is balancing data ratios and optimization pacing. A pragmatic approach is to proceed in stages:
- First, train the visual encoder independently to obtain stable representations across images and videos.
- Freeze most parameters or use a lower learning rate to train only the projection layer, gradually bringing the two embedding spaces closer.
- Conduct joint training on a mixture of image-text, video, and text-only data, while retaining an evaluation baseline for language capabilities.
- Perform post-training tailored for agent tasks, incorporating samples of tool calling and multi-step decision-making.
The value of decoupled training lies in reducing debugging difficulty. If a certain metric drops, you can pinpoint the specific stage rather than blindly tuning parameters across the entire large model.
5. Tool Calling and Parallel Orchestration
5.1 Sequential Execution Latency
Many agent frameworks adopt a single-threaded loop: plan, call a tool, observe the result, and plan again. If a task contains ten independent sub-steps, the total latency of sequential execution equals the sum of each step’s duration. For operations dominated by I/O wait times—such as retrieval, web scraping, and file checking—parallelization yields the most significant benefits.
5.2 Task Decomposition and Dependency Analysis
The prerequisite for parallelism is correctly identifying dependencies. Tasks can be represented as a Directed Acyclic Graph (DAG), where nodes are sub-tasks and edges represent data dependencies. After topological sorting, nodes at the same level can execute concurrently. The challenge is that when sub-tasks are dynamically generated by the model, the dependency relationships themselves must be inferred and validated.
5.3 Example: Layered Parallel Scheduling
The following code demonstrates the basic structure of intra-layer parallelism and inter-layer sequential execution. The tool functions are placeholder implementations and should be replaced with actual service calls in real-world scenarios.
import asyncioasync def run_tool(name: str, arg: str) -> str:# Placeholder implementation: replace with real tool or service calls in actual scenariosawait asyncio.sleep(0.1)return f"{name}({arg}) done"async def run_layer(tasks: list[tuple[str, str]]) -> list[str]:return await asyncio.gather(*(run_tool(n, a) for n, a in tasks))async def orchestrate(layers: list[list[tuple[str, str]]]) -> list[str]:results: list[str] = []for layer in layers: # Parallel within layers, sequential across layers to ensure dependency orderresults.extend(await run_layer(layer))return resultsif __name__ == "__main__":plan = [[("fetch_page", "a"), ("fetch_page", "b"), ("fetch_page", "c")],[("summarize", "a+b+c")],]print(asyncio.run(orchestrate(plan)))
5.4 Engineering Best Practices
- Concurrency Limits: Use semaphores to limit the number of tools executing simultaneously, preventing downstream service overload.
- Timeouts and Retries: Set timeouts for each sub-task and distinguish between retryable and non-retryable errors.
- Structured Results: Sub-task outputs should adopt a unified structure to facilitate summarization and reasoning by the main agent.
- Observability: Record the start time, end time, and status of each node to help identify bottlenecks.
6. Evaluation and Deployment
Evaluating multimodal agents should not rely solely on a single benchmark score. It is recommended to establish metrics across three layers. The perception layer focuses on fine-grained text recognition, chart reading, and video event localization; the decision layer focuses on tool selection.
