Architectural Evolution of Open-Source Native Multimodal Agent Models

1. Background: From Text-First to Native Multimodal Agents

In recent years, Large Language Models (LLMs) have made significant progress in code generation, mathematical reasoning, and natural language understanding. However, for agent applications in real-world business scenarios, text-only capabilities are no longer sufficient. Enterprises expect models to directly understand visual information such as screenshots, scanned documents, surveillance feeds, and user interfaces, while also executing multi-step decision-making within complex workflows.

The traditional approach involves training a pure text LLM first and then plugging in a visual encoder for adaptation. While simple to implement, this “language-first, vision-second” pipeline introduces two major issues:

1. Modality Imbalance: The model exhibits strong language skills but weak visual comprehension, leading to significant biases in mixed image-text tasks.
2. Training-Inference Disconnect: The objectives of the visual module and the language backbone are misaligned, making late-stage adaptation insufficient for deep cross-modal semantic alignment.

Consequently, a new generation of open-source multimodal agent models is emerging, designed to natively fuse text and vision at the foundational architectural level. This allows both types of information to be jointly modeled within the same semantic space. This shift represents a transition in design goals from simply “seeing and answering” to “observing, planning, and acting.”

2. Overall Architecture: A Unified Multimodal Pipeline

A typical native multimodal agent model can be abstracted into a three-layer structure:

Input Layer: Text + Image + Video
        ↓
Vision Encoding Layer: Unified visual feature extraction
        ↓
Cross-Modal Projection Layer: Aligning visual features to the language embedding space
        ↓
Language Backbone Layer: Unified modeling and inference via Mixture of Experts (MoE)

The key to this architecture is that visual signals are not simply appended after text tokens. Instead, they undergo independent and complete visual encoding before being mapped by a projection layer into the same vector dimension as the language embeddings. Finally, both text and visual tokens enter the language backbone, achieving cross-modal interaction during attention computation. This design enables the model to learn fine-grained correspondences between image regions and text fragments early in the training phase, rather than being forced to align them during inference.

3. Native Visual Encoder: Unified Representation for Images and Videos

In multimodal agent scenarios, input formats are highly diverse. An input could be a high-resolution product screenshot or a continuous screen recording. Designing separate encoders for images and videos not only increases parameter count and engineering complexity but also reduces the transfer efficiency of visual knowledge between the two domains.

To address this, mainstream technical solutions are adopting a Unified Visual Encoder. Its core principles can be summarized in three points:

3.1 Native Resolution Support

Traditional vision models typically require input images to be resized to fixed dimensions, such as 224×224 or 336×336. While effective for natural image classification, this approach causes severe information loss in agent scenarios. Small-font buttons in UIs, dense numbers in tables, and distant objects in surveillance feeds lose critical details once compressed.

A more robust approach uses a patch-packaging strategy. The image is divided into fixed-size patches, which are then flattened into a 1D sequence according to spatial order. Whether the original image is a wide panoramic view or a vertical screenshot, the model encodes it while preserving the original aspect ratio without forced stretching or cropping. This effectively retains the legibility of small targets and high-density text.

3.2 Shared Weights for Images and Videos

Video can be viewed as a collection of continuous image frames, but adjacent frames contain massive redundancy. Encoding every frame independently not only incurs heavy computational overhead but also fails to effectively model temporal action changes.

A unified encoder typically groups consecutive frames into spatiotemporal units (e.g., grouping 4 frames together). The spatial and temporal blocks within each unit are jointly packaged and fed into the same encoder. Consequently, image tasks become a special case of video tasks where the “frame count is 1.” Because weights are fully shared, the model’s ability to recognize edges, textures, text, and objects learned from image data naturally transfers to video inputs, eliminating the need for additional temporal branches.

3.3 Multi-Scale Spatiotemporal Attention

The unified encoder’s attention mechanism must cover both spatial and temporal dimensions. Spatially, the model focuses on relationships between different regions within the same frame; temporally, it tracks changes in the same region across consecutive frames. Through shared weights and joint attention computation, the encoder learns spatiotemporal patterns crucial for agent operations, such as button highlighting, progress bar movement, or popup appearances.

4. Cross-Modal Projection: Bringing Vision into the Language Space

The feature dimension output by the visual encoder usually differs from the hidden dimension of the language model. Therefore, a projection module is required to map visual features linearly or non-linearly into the language embedding space. Although seemingly straightforward, this module significantly impacts training stability.

If the projection layer lacks expressive power, visual information may be compressed into overly coarse representations before entering the language backbone, rendering the model unable to distinguish fine-grained visual differences. Conversely, an overly complex projection layer introduces extra parameters and training difficulties. In practice, a Multi-Layer Perceptron (MLP) is typically used, paired with appropriate normalization strategies during large-scale training to ensure that the numerical distributions of visual and text tokens are as consistent as possible.

5. Language Backbone: Cross-Modal Inference via Mixture of Experts

The backbone network handles the final reasoning and generation tasks. In open-source large models, the Mixture of Experts (MoE) architecture is widely adopted. Its defining feature is maintaining a massive total parameter count while only activating a subset of expert modules during each forward pass, balancing capability with efficiency.

In a native multimodal design, the MoE backbone receives both text and visual tokens from the beginning of training. This means the expert routing strategy must learn to allocate tokens appropriately based on their modality source and content characteristics. For instance, some experts might specialize in code symbols, while others lean toward processing spatial relationships in images. Since the number of visual tokens is usually far greater than text tokens, engineering efforts must carefully control the compression ratio of visual tokens to prevent overwhelming the backbone’s computational capacity.

6. Training Strategy: Decoupled Encoders and End-to-End Optimization

Directly performing end-to-end training on large-scale multimodal models often leads to instability and slow convergence. A common practice is to adopt a staged strategy:

1. Visual Encoder Pre-training: Train the visual encoder using large-scale image, video, and image-text pair data to establish foundational visual comprehension.
2. Cross-Modal Alignment Training: Freeze or partially unfreeze the visual encoder and focus on training the projection layer to achieve initial alignment between visual features and the language space.
3. Joint Fine-tuning: Unfreeze all parameters and perform end-to-end optimization using high-quality multimodal instruction data.

This decoupled training pipeline mitigates conflicts arising from different optimization paces across modalities, allowing the model to achieve finer cross-modal synergy in its final stage.

7. Agent Clusters: From Serial Execution to Dynamic Parallelism

Beyond multimodal comprehension, agent models face a prominent challenge: execution latency in complex tasks. In traditional single-agent modes, tasks are decomposed into multiple steps executed sequentially, where each step must wait for the previous one’s output. For example, when processing a report with multiple tables, the model might read the first page, then the second, and finally summarize. While logically clear, this serial approach causes latency to grow linearly with the number of steps.

An innovative solution is Dynamic Parallel Orchestration of Agent Clusters. Its core concepts include:

• Decomposing a large task into multiple parallelizable sub-tasks.
• Assigning a lightweight execution agent to each sub-task.
• Using a coordination node to dynamically monitor the progress and dependencies of all sub-tasks.
• Aggregating the results of all branches into the main model for final integration.

This approach resembles Directed Acyclic Graph (DAG) scheduling in software engineering: nodes without dependencies execute in parallel, while waiting only occurs when data dependencies exist. Compared to fixed serial pipelines, this mechanism can reduce end-to-end latency several times over, as massive independent local observations and reasoning processes are handled simultaneously rather than queued.

For example, when analyzing a multi-page contract, clause extraction on different pages can occur in parallel, with final aggregation to the main model for risk assessment. In software UI operations, sub-tasks like page state analysis, log parsing, and input generation can dynamically run in parallel based on their dependencies.

8. Evaluation and Practical Considerations

From an engineering perspective, evaluating whether a native multimodal agent model is truly production-ready requires looking beyond single benchmark scores. Focus should be placed on the following dimensions:

1. Cross-Modal Consistency: Can the model provide consistent judgments for the same information presented in both text and image formats?
2. High-Resolution Robustness: Under native resolution input, can the model still recognize small-font text, tiny icons, and low-contrast elements?
3. Long-Task Stability: During multi-step agent tasks, does the model suffer from forgetting, repetition, or error accumulation as steps increase?
4. Parallel Scheduling Efficiency: After decomposing complex tasks, is the coordination node’s scheduling strategy stable enough to reduce wait times without compromising result quality?

For enterprise developers, deployment and inference costs are equally critical. The number of visual tokens, video frame sampling strategies, and MoE activation counts directly impact GPU resource consumption. Before business deployment, it is recommended to benchmark throughput and latency across pure text, single-image, multi-image, and video inputs based on specific use cases.

9. Conclusion

Native multimodal agent models are fundamentally changing the traditional “late-stage visual adaptation” paradigm. Through unified visual encoding, cross-modal projection, and joint modeling from the onset of training, these models achieve more stable performance across mixed image, video, and text tasks. Concurrently, dynamic parallel agent orchestration frameworks offer new engineering solutions to the latency challenges of complex tasks. For enterprise business scenarios, the true value of these models lies not in isolated benchmark scores, but in their ability to execute a complete closed loop—from observation to understanding and action—within real-world workflows.