Multimodal Agent Architecture: Choosing Between Late and Native Vision-Language Fusion, and Parallel Task Orchestration
This article explains when a multimodal agent is worth its added cost, how late fusion and native fusion differ in training and engineering, how native-resolution visual encoding trades detail against latency and memory, and how to plan parallel subagent execution using task dependency graphs and clearly scoped roles.
1. Background: Why Agents Need Vision and Parallel Execution
Large language models are moving from chat assistants to agents that complete tasks. An agent must read text, but it also needs to interpret screenshots, tables, documents, and user interfaces. It must call tools and break work into multiple steps. Vision is therefore a core capability, on par with language reasoning and tool use.
This article discusses one family of technical approaches for open-source multimodal agent models. It focuses on two questions: how vision and language are optimized together during training, and how multiple subagents execute tasks in parallel. The article does not refer to any specific model, and it does not cite speedup multiples or rankings. Such figures often lack baselines, test sets, and measurement conditions, so quoting them directly can mislead decisions. To verify specific metrics, consult the papers, technical reports, and open-source repositories of the relevant model.
When a Multimodal Agent Is Not Needed
Many projects do not need vision. If a task only processes structured data, plain-text contracts, or system logs, a text model with a well-designed tool interface is usually sufficient. Vision adds training cost, inference latency, and deployment complexity. Its benefits should be weighed against these costs.
Selection Checklist: Answer These Five Questions First
| Criterion | Leans toward late fusion | Leans toward native fusion |
|---|---|---|
| Existing text model | A stable, capable text model is already in place | Unified training is needed from the pretraining stage |
| Vision demand | Occasional image use, such as recognizing a few pictures | Frequent work with charts, interfaces, and documents, requiring fine-grained reading or localization |
| Multimodal data budget | Limited data and compute, so only alignment fine-tuning is possible | Sufficient image-text and video data and training compute |
| Latency constraints | Latency-sensitive, so visual input must be strictly limited | Able to accept the inference overhead of more visual tokens |
| Compliance requirements | The vision module must be separately replaceable, upgradable, or auditable | The source and usage scope of image data can be fully traced |
A column leading on one row does not settle the decision. Hard constraints such as compliance requirements should be met first, and then the other criteria should be compared.
2. Choosing an Architecture: Late Fusion vs. Native Fusion
The traditional approach happens in two steps. First, train a text-only large language model. Once its language ability is stable, attach a vision encoder and fine-tune alignment on a batch of multimodal data. This approach is common and simple to implement, but it has three problems:
- Modality conflict: Gradient updates from the vision component can disturb language abilities that have already converged. The model may get better at seeing images while getting worse at reasoning.
- Capability imbalance: Vision often serves as a supplement and struggles to support operations agents commonly need, such as reading chart values or locating interface elements.
- Data and budget limits: Data volume and compute in the late alignment stage are limited, so the ceiling for vision capability is largely set at that step.
Native fusion puts vision and language into the same training process. A typical pipeline has three stages. A vision encoder extracts visual features. A lightweight projection layer maps those features into the language model’s embedding space. The visual tokens are then concatenated with text tokens and fed together into the language backbone. Both modalities are represented in one semantic space, so each can reinforce the other.
A minimal example shows what concatenation means. Suppose a user uploads a table screenshot and asks, “What is the sales figure in the third row?” The system first splits the screenshot into visual tokens and splits the question into text tokens. The two are arranged in order into one sequence. The language backbone computes attention over the entire sequence and then generates the answer. The backbone receives only a series of vectors. It distinguishes image content from text content by relying on representations learned during training.
The following table compares the two routes:
| Dimension | Late fusion | Native fusion |
|---|---|---|
| Training stage | The language model is trained first, then vision is attached | Vision and language are jointly optimized during pretraining |
| Modality conflict | Vision gradients may interfere with language ability | Joint training and data mixing manage conflicts |
| Data mixing | Multimodal data is usually used only for fine-tuning | Text, image, and video data are mixed during pretraining |
| Engineering complexity | Lower, with changes concentrated in the add-on module | Higher, requiring coordinated training of data, encoder, and backbone |
| Data and compute threshold | Lower, so work can start with a modest fine-tuning dataset | Higher, requiring large-scale multimodal data and a longer training cycle |
| Debugging complexity | Lower, with most issues in the add-on module | Higher, with issues that may stem from data mixing, the encoder, or the backbone |
| Suitable scenarios | A mature text model exists and vision needs are light | Vision-dense tasks such as charts, interfaces, and documents are core |
“Higher” and “lower” are relative comparisons, not absolute values. Actual costs depend on data scale, model size, and team experience.
3. What Native-Resolution Visual Encoding Is
Many vision models resize input images to a fixed size, often a square. Resizing compresses detail. Small text in tables, icons in interfaces, and footnotes in documents may become unreadable after shrinking, so the model has to guess to fill in the gaps.
Native-resolution encoding keeps the original aspect ratio and resolution. The encoder divides the image into fixed-size patches, and each patch becomes one or more visual tokens. Larger images produce more patches and more visual tokens. The model can see more detail, but computation and memory usage grow with the token count.
Simple arithmetic shows the gap. With 14×14 pixel patches, a 1920×1080 screenshot yields about 137 columns and 77 rows of patches, or roughly 10,000 patches in total. An image resized to 448×448 yields 32×32 patches, or about 1,000 in total. The difference is roughly tenfold, which directly affects attention computation and time to first response. These figures only illustrate the effect of an assumed patch size. They do not describe the actual settings of any specific model.
Screenshot, table, and interface tasks depend heavily on detail. Locating a button requires seeing its edges and text. Reading a table requires aligning rows and columns. These tasks usually benefit most from native resolution.
Weigh three trade-offs when using this approach:
- Higher resolution captures more detail, but it also raises latency and memory cost.
- A common practice is to set a maximum resolution. Images above that limit are resized or split into tiles first.
- Tiling can break structures that span tiles, such as a table line crossing two tiles. Confirm that the encoding scheme preserves positional relationships between tiles.
The limits of this approach also need to be clear. For tasks centered on holistic understanding of natural photos, such as identifying the type of scene, higher resolution offers limited gains. For high-volume workloads with strict latency requirements, measure how the visual token count affects response time before setting the resolution limit. If the input image is blurry, raising the resolution only amplifies noise. It cannot replace a clear original input.
4. Considerations for Dynamic Parallel Task Orchestration
Complex tasks often contain multiple steps. Some of these steps do not depend on each other and can run at the same time. For example, an agent may need to compare two reports. One subagent reads the first report, another reads the second, and a third subagent summarizes the differences between them. Parallelism shortens total run time, but it also introduces new problems.
Draw the Task Dependency Graph First
Before splitting a task, draw each step as a node and each data dependency as an edge. Only nodes with no dependency between them can run in parallel. For example, “calculate the summary from the table in step one” must wait until step one finishes. The dependency graph also reveals the critical path, the longest chain of dependencies, which sets the lower bound on total run time.
Define Each Subagent’s Role Clearly
Each subagent should have clear inputs, outputs, tools, and permissions. The narrower the role, the easier it is to test and replace independently. When several subagents share the same writable resource, such as the same file or the same data record
