Architecture Upgrades in Native Multimodal LLMs: Visual Encoding and Agent Clusters

Architecture Upgrades in Native Multimodal LLMs: Visual Encoding and Agent Clusters

In recent years, the development of Artificial General Intelligence (AGI) has been evolving from simple text-based dialogue toward general-purpose agents capable of executing complex tasks. However, traditional multimodal models often follow an architecture paradigm where “language is primary, and vision is adapted later.” This patchwork approach easily leads to a semantic gap between visual and language modalities. Furthermore, when executing complex tasks, the sequential execution model of a single agent is highly susceptible to latency bottlenecks. This article delves into the core technical highlights of a new generation of native multimodal agent architectures, exploring how native visual encoding and parallel orchestration of agent clusters achieve a leap in general-purpose agent capabilities.

1. Architectural Philosophy: From Late Adaptation to Native Fusion

Traditional visually-adapted models typically pre-train a powerful text-only Large Language Model (LLM) first. A visual encoder is then attached externally, and a projection layer maps visual features into the language model’s input space. Although this “text first, vision later” approach is simple to engineer, it has inherent flaws: the language model does not establish a foundational understanding of visual concepts during the pre-training phase. This leads to a trade-off in multimodal tasks where “language capabilities are strong, but visual grounding is weak.”

The new generation of architecture disrupts this paradigm entirely, adopting an end-to-end native fusion pipeline consisting of “visual feature extraction - cross-modal projection - unified modeling by the language backbone.” Specifically, image and video inputs are first processed by a native visual encoder to extract unified features. These features are then aligned to the language model’s embedding space via a Multi-Layer Perceptron (MLP) projection layer. Finally, they are fed into a Mixture of Experts (MoE) language backbone alongside text tokens for joint modeling. This design ensures that visual and language modalities are uniformly represented within the same semantic space, eliminating modal conflicts at the foundational architectural level and achieving bidirectional enhancement.

2. Core Component: Breakthroughs in the Native 3D Visual Encoder

The visual encoder acts as the “eyes” of a multimodal model to perceive the physical world. The new architecture introduces a native 3D visual encoder—a Vision Transformer based on joint spatiotemporal modeling—specifically designed for the unified understanding of images and videos. Its core technical highlights are reflected in two main dimensions:

2.1 Native Resolution and Variable Input Support

In traditional visual processing pipelines, images usually need to be cropped or resized to a fixed resolution to fit the encoder. This preprocessing not only loses crucial local details (such as small text in document screenshots or distant targets) but also introduces redundant background padding, wasting computational resources.

The new visual encoder introduces a dynamic patching strategy based on Patch Packing. Its core logic is as follows:

1. Dynamic Patching: Raw images of any aspect ratio and resolution are directly split into multiple visual patches.
2. 1D Flattening: These patches are flattened into a one-dimensional sequence without forced resizing to a uniform size.
3. Attention Computation: Full attention modeling is performed on the entire sequence within the Transformer.

The following pseudocode illustrates this dynamic processing logic:

def dynamic_patch_packing(image):
    # Split into patches at native resolution without forced resizing
    patches = extract_patches(image, dynamic_size=True)
    # Flatten into a 1D sequence, preserving original aspect ratio information
    sequence = flatten(patches)
    # Add positional encoding and input into the visual encoder
    return vision_encoder(sequence + positional_encoding(sequence))

This design natively supports inputs of any resolution, maximally preserving original visual information. This is crucial for high-precision scenarios such as document understanding and fine-grained image analysis.

2.2 Spatiotemporal Modeling with Full Weight Sharing for Images and Videos

For video understanding tasks, traditional solutions often require designing independent 3D visual modules or temporal fusion branches. This not only increases the model’s parameter count but also hinders the smooth transfer of image understanding capabilities to the video domain.

The new architecture innovatively extends the “Patch Packing” concept from the spatial dimension to the temporal dimension. It treats continuous 4-frame video segments as spatiotemporal volumes, jointly flattening and packing 2D patches from multiple frames. By fully sharing the weights of a single visual encoder, it simultaneously performs attention modeling across both spatial and temporal dimensions. This design eliminates the need for additional temporal modules, reducing architectural complexity. More importantly, it achieves an extreme transfer of image pre-training knowledge to video understanding tasks, significantly improving the model’s ability to capture temporal dynamics in videos.

3. Agent Clusters: A Parallel Orchestration Framework Breaking Single-Chain Latency

In complex task execution scenarios (such as code generation or computer control), traditional single-agent models often need to sequentially execute a loop of “think - call tool - observe result - think again.” This serial execution mode leads to high task completion latency, making it difficult to meet the demands of enterprise-grade real-time applications.

To solve this bottleneck, the new architecture introduces a dynamic parallel orchestration framework for agent clusters. The core idea is to decompose a complex single task into multiple executable sub-tasks that can run in parallel, dynamically coordinated by multiple agents within a cluster.

3.1 Parallel Orchestration Mechanism

The key to the agent cluster framework is the graph-based construction of task dependencies. Upon receiving a complex instruction, the master agent generates an execution plan in the form of a Directed Acyclic Graph (DAG):

• Parallel Independent Nodes: Sub-tasks with no dependencies (e.g., querying two different APIs simultaneously) are assigned to different sub-agents for parallel processing.
• Serial Dependent Nodes: Sub-tasks with data dependencies are executed strictly following topological sorting.

3.2 Dual Benefits of Latency and Performance

Through this dynamic parallel orchestration, the model fully utilizes the concurrency capabilities of the compute cluster. Empirical data shows that in complex tasks involving multi-step tool calls, this framework can reduce end-to-end execution latency by up to 4.5 times. Meanwhile, because sub-agents can focus on specific sub-domains, the accuracy and robustness of task execution are also significantly improved.

4. Engineering Training Pipeline and Enterprise Deployment Recommendations

An excellent architecture requires a robust engineering training pipeline. This multimodal architecture adopts a decoupled encoder training strategy: the visual encoder is first pre-trained independently on a large scale to equip it with strong foundational visual feature extraction capabilities, and then jointly fine-tuned with the language backbone. This decoupling strategy ensures the efficiency and stability of large-scale multimodal training.

For enterprise applications, deploying such complex multimodal agents typically relies on mature cloud-native infrastructure:

1. Model Inference Service: Deploy MoE architecture models using container platforms, utilizing elastic scaling capabilities to handle high-concurrency requests.
2. Agent Tool Calling: Coordinate asynchronous interactions between agents and external tools using message queues and API gateways.
3. Data Storage and Logging: Store multimodal input data in object storage. Concurrently, use logging services to record every inference and tool call chain step of the agent cluster, facilitating subsequent auditing and performance tuning.

5. Conclusion

Through two core technological innovations—“native fusion of vision and language” and “parallel orchestration of agent clusters”—the new generation of native multimodal agent architectures has successfully broken through the capability ceiling of traditional multimodal models. The native 3D visual encoder resolves the challenges of information loss and modal transfer, while the dynamic parallel agent cluster completely restructures the execution efficiency of complex tasks. This architectural evolution not only points the way forward for the development of general-purpose agents but also provides a highly valuable technical blueprint for enterprise deployment in high-value scenarios such as automated office work, complex code generation, and computer vision analysis.