1. Technical Background: Two Evolution Paths of Multimodal Agents

In recent years, the capability boundaries of Large Language Models (LLMs) have expanded from “dialogue and text generation” to “perceiving the world, using tools, and autonomously completing tasks.” Correspondingly, the technical evolution of multimodal agent models shows two clear trajectories:

First, the paradigm for integrating visual capabilities is shifting. The traditional approach is “language first, vision adapted later”: a text-only LLM is trained first, and a visual encoder is bolted on later for adaptation. While simple to engineer, this method has inherent flaws. Because visual signals are introduced late in training, they must accommodate an already solidified language representation space, making it difficult to deeply participate in reasoning. Furthermore, improper data ratios or learning rate strategies during the adaptation phase can easily lead to language degradation or insufficient visual capability, creating a trade-off between modalities.

Second, agent execution architectures are transitioning from “single-agent sequential execution” to “multi-agent parallel orchestration.” Complex tasks can often be decomposed into sub-tasks, many of which are independent. If a single agent executes these sequentially, end-to-end latency grows linearly with task scale, becoming a core bottleneck for agent deployment.

Below, we break down the key technical designs of such models from both the model architecture and agent orchestration perspectives.

2. Unified Multimodal Architecture: Native Vision-Language Fusion

2.1 Overall Pipeline: Three-Stage Unified Modeling

These models typically adopt a unified multimodal architecture. The pipeline can be summarized in three stages: “visual feature extraction — cross-modal projection — unified modeling in the language backbone.”

• Images or videos are first processed by a visual encoder to extract unified visual features.
• The features are aligned to the language model’s embedding space via a lightweight projection layer (usually an MLP).
• Visual tokens and text tokens are mixed and fed together into a Mixture-of-Experts (MoE) language backbone for end-to-end modeling.

The fundamental difference from the “late-fusion” approach is that the training objective is multimodal joint from the very first stage. Vision and language learn unified representations in the same semantic space, enabling bidirectional enhancement—visual information participates in language reasoning, and language knowledge backfills visual understanding.

2.2 Native Resolution Encoding: Patch Packing Strategy

A key design on the visual encoding side is native resolution support. Traditional methods usually resize or crop images to a fixed resolution, leading to two types of information loss: small objects and dense text become indiscernible at low resolutions, and images with unconventional aspect ratios suffer semantic damage when stretched or cropped.

The patch packing strategy works by dividing a single image into fixed-size patches, flattening all patches into a 1D token sequence, forming sequences of varying lengths for different resolutions, and then packing them within a batch. Its advantages include:

• Any aspect ratio and resolution can be input directly, with zero information loss during preprocessing.
• Computation scales with the amount of image information—small images require less computation, large images more, leading to more rational compute allocation.
• Multiple images within the same batch share encoder weights, improving training efficiency.

Schematic pseudocode:

def pack_images(images, patch_size, max_tokens):
    """Pack images of arbitrary resolutions into a 1D token sequence"""
    sequences = []
    for img in images:                     # Any aspect ratio, any resolution
        patches = to_patches(img, patch_size)
        seq = flatten_1d(patches)          # Flatten and concatenate into a 1D sequence
        sequences.append(seq)
    return concat_and_pad(sequences, max_tokens)  # Pack within batch to reduce padding waste

2.3 Unified Image-Video Modeling: Spatiotemporal Joint Packing

Video understanding typically requires a separate temporal modeling module, but this fragments capability sharing between images and videos. A more elegant approach extends the “packing” concept from the spatial to the temporal dimension: organizing continuous frames into a spatiotemporal volume, jointly flattening and packing the 2D patches of each frame, and feeding them into the same encoder. The attention mechanism naturally establishes correlations across both space and time.

The value of this design lies in:

• Images and videos completely share encoder weights, eliminating the need for separate branches and reducing both parameter count and engineering complexity.
• Semantic knowledge learned from image understanding transfers directly to video understanding, mitigating the scarcity of video annotation data.
• Temporal granularity can be flexibly controlled via frame sampling intervals, balancing temporal detail with computational overhead.

2.4 Projection Layer and MoE Backbone: Decoupling Capacity and Computation

The alignment of visual features and text tokens is typically handled by a lightweight MLP projection layer. Although its parameter count is small, it is critical to modality alignment quality. The language backbone generally adopts a Mixture-of-Experts (MoE) architecture: each token activates only a subset of experts, decoupling total parameter count from single-inference computation. This allows the model’s total capacity to scale significantly while keeping inference costs controllable. This is especially crucial for agent models that must both “understand what they see” and “perform actions,” as agent scenarios often involve long contexts and multi-turn interactions, making them highly sensitive to inference costs.

2.5 Decoupled Encoder Training Pipeline

Large-scale multimodal training faces stability challenges in practice: the loss magnitudes and gradient variances differ between the visual and language sides, leading to mutual interference during the early stages of joint training. A common engineering solution is a decoupled training pipeline—the visual encoder is first pre-trained on large amounts of image-text data to obtain relatively stable representations, and then jointly optimized end-to-end with the language backbone. During the joint phase, differentiated learning rates and freezing strategies can be applied to different modules—stabilizing alignment first, then conducting comprehensive tuning. This “decouple first, fuse later” pipeline is a crucial engineering safeguard for the stable execution of large-scale multimodal training.

3. Agent Swarm: Dynamic Parallel Orchestration Framework

3.1 The Latency Bottleneck of Single-Agent Sequential Execution

A complex task (e.g., “research a topic and generate an analysis report”) is processed sequentially in a single-agent paradigm: retrieve → read → re-retrieve → write → verify. Each step must wait for the previous one, and all intermediate results crowd the same context window. The result is not just linear latency accumulation, but also attention dilution and error accumulation caused by context crowding.

3.2 Core Mechanisms of Dynamic Parallel Orchestration

The agent swarm framework separates “planning” from “execution”: a master agent handles task decomposition and scheduling, while multiple sub-agents execute sub-tasks in parallel. The core mechanism consists of four steps:

1. Task Decomposition: The master agent breaks down the goal into appropriately granular sub-tasks, explicitly defining the inputs, outputs, and acceptance criteria for each.
2. Dependency Analysis: A Directed Acyclic Graph (DAG) is built based on data dependencies between sub-tasks to identify which can run in parallel.
3. Scheduling and Execution: All dependency-free sub-tasks are concurrently dispatched to multiple sub-agents. Each sub-agent has an independent context window and focuses on a single responsibility.
4. Result Aggregation: The master agent collects sub-task results, resolves conflicts, and summarizes. It dynamically adjusts the subsequent plan based on intermediate results—this is the meaning of “dynamic”: the orchestration structure is not fixed in advance but evolves as execution progresses.

Schematic pseudocode:

async def agent_swarm(task):
    plan = planner.decompose(task)            # 1. Task decomposition
    dag = build_dag(plan)                     # 2. Dependency analysis
    results = {}
    while not dag.finished():
        ready = dag.ready_nodes()             # Get all dependency-free sub-tasks
        outs = await gather(                  # 3. Parallel dispatch and execution
            subagent(n).run(n.desc, results) for n in ready
        )
        results.update(zip(ready, outs))
        dag.mark_done(ready)
        dag = replanner.revise(dag, results)  # 4. Dynamically adjust plan
    return aggregator.merge(results)

3.3 Benefit Analysis

From the perspective of critical path theory, the latency benefit of parallel orchestration depends on the parallelism of the task dependency graph. When multiple sub-tasks are independent, end-to-end latency is dictated by the “longest chain” rather than the “total task volume,” often yielding an order-of-magnitude reduction in latency in practice. Meanwhile, because each sub-agent’s context is more focused, single-step execution quality improves, and errors are isolated within local sub-tasks, thereby improving the overall task success rate. This achieves a “dual optimization of latency and quality” rather than trading quality for speed.

4. Tool Usage and Computer Manipulation: The Critical Loop of Agent Capabilities

The ultimate value of vision and language capabilities materializes in “getting things done.” The tool capabilities of these agent models are typically built around three components:

• Unified Action Space: Heterogeneous operations like retrieval, code execution, file I/O, and screen clicks are abstracted into a unified tool-calling protocol. The model expresses “what tool to call and what parameters to pass” via structured outputs.
• Plan-Execute-Reflect Loop: The model first plans an action sequence, observes environmental feedback (including visual data like screenshots) after execution, and corrects the next action accordingly, forming a closed loop.
• Sandbox and Security Boundaries: High-risk operations like code execution and computer manipulation run in isolated environments, paired with access control and operation auditing to ensure safety and controllability.

Among these, computer manipulation (GUI agents) relies heavily on deep visual participation. The model needs to directly understand the layout of UI elements in screenshots rather than relying on fragile structured parsing—this is exactly where the native vision-language fusion architecture delivers value in agent scenarios.

5. Evaluation Dimensions and Engineering Deployment

5.1 Typical Evaluation Dimensions

To measure the capabilities of such models, the industry typically focuses on five benchmark categories: code generation (evaluating logical rigor), visual understanding (evaluating perception accuracy), mathematical reasoning (evaluating multi-step reasoning), agent task execution (evaluating long-chain planning and tool calling), and computer manipulation (evaluating real-environment interaction). Leading across multiple capabilities simultaneously indicates that the underlying architecture’s joint optimization yields cross-task generalization benefits, rather than just targeted optimization for a single capability.

5.2 Engineering Deployment Path

For teams looking to deploy such capabilities in their own business, the general engineering path involves leveraging cloud GPU compute and container platforms for model deployment and elastic scaling; using object storage to manage training corpora and multimodal assets; and tracking agent execution chains via logging services and monitoring alert systems. Cloud platforms like Baidu AI Cloud provide these general product categories, which teams can combine as needed. However, the primary investment should focus on building data pipelines and evaluation systems.

5.3 The Significance of Open Source Ecosystems

Notably, this generation of models generally opts to open-source their trained weights. For researchers, open-source weights mean reproducible methods and verifiable conclusions. For engineering teams, it enables fine-tuning and private deployment on proprietary data, fulfilling data compliance requirements. The feedback loop from the open-source community is accelerating the transition of general agent technology from research papers to production.

6. Conclusion

The capability leap in multimodal agent models fundamentally stems from redesigns at two levels. At the model architecture level, “vision-language joint optimization throughout” replaces “late adaptation,” making vision a “first-class citizen” on par with language. At the system level, “dynamic parallel orchestration of agent swarms” replaces “single-agent sequential execution,” using systems engineering to break through the latency constraints of single LLM inferences. The combination of the two drives general agent intelligence from concept to usability. For developers, understanding the design logic behind these two trajectories is more valuable than memorizing any specific parameter—they are likely to define the evolution direction of the AI agent tech stack in the coming years.