From Post-Hoc Vision Adaptation to Native Joint Optimization

For a long time, mainstream multimodal large models have relied on a two-stage adaptation paradigm: first train a text-only language model, then attach a visual encoder later and fine-tune it with a small amount of image-text alignment data. This approach is cheap and fast to iterate, but it has three clear drawbacks:

1. Modal capability trade-offs. The vision module acts as a late patch, often at the expense of language performance. A typical symptom is that text reasoning scores drop after image input is enabled, or the model performs mediocrely on image-text tasks.
2. Alignment information bottleneck. The projection layer only serves as a translator. Since the language backbone never saw visual signals during pretraining, the cross-modal semantic space fails to achieve deep fusion.
3. Weak video modeling. The language backbone lacks prior knowledge of temporal visual signals. Video understanding usually requires extra branches or large-scale continued training, which is costly and delivers limited gains.

The core idea of next-generation multimodal agent models is to move vision from a late-stage adaptation step to the very beginning of pretraining, enabling joint optimization of text and vision throughout the entire pipeline. Based on public technical reports and open-source practice, this native joint training paradigm delivers bidirectional gains: visual understanding tasks benefit from the reasoning power of the language backbone, while text reasoning tasks gain richer world knowledge from visual signals. On public benchmarks covering code generation, visual understanding, mathematical reasoning, agent execution, and computer control, such models have reached the leading level of the open-source camp, with some metrics comparable to closed-source flagship models. This provides a solid, reproducible foundation for agent research.

Unified Multimodal Architecture: Three Core Components

The overall pipeline follows a three-stage design: visual feature extraction, cross-modal projection, and unified modeling in the language backbone. Image and video inputs are first processed by a native 3D visual encoder to extract unified visual features. These features are then aligned to the language model’s embedding space via an MLP projection layer. Finally, visual tokens and text tokens are concatenated and fed into a MoE language backbone for end-to-end modeling, achieving unified representation of both modalities in the same semantic space.

Native-Resolution Visual Encoder: Patch Packing

Traditional visual encoders require fixed input resolutions, forcing arbitrary images to be cropped or resized first. This introduces two types of irreversible loss:

• Aspect ratio distortion: Stretching and scaling breaks geometric structure, which is especially harmful for layout-sensitive inputs such as charts, documents, and UI screenshots.
• Resolution bottleneck: When large images are squeezed to small resolutions, fine-grained information such as small text, thin lines, and icons is lost entirely.

The common industry fix is patch packing: images are split into patches and flattened into a one-dimensional sequence. Patches from multiple samples can be packed into the same batch, with attention masks preventing cross-sample interference. The benefits include:

1. Native support for arbitrary aspect ratios and resolutions without destructive preprocessing.
2. Samples with different visual overheads share one batch, significantly improving training throughput.
3. Unified serialization of images lays the foundation for joint image-video modeling.

Unified Image-Video Modeling: Spatiotemporal Packing with Full Weight Sharing

If images and videos use separate encoders, parameter count doubles and knowledge learned on the image side does not transfer well to the video side. A more elegant approach treats video as “images with a time dimension”:

• Organize several consecutive frames (e.g., 4 frames) into a spatiotemporal block.
• Flatten 2D patches from all frames jointly into the same sequence.
• Use exactly the same encoder weights as images, applying attention modeling uniformly across spatial and temporal dimensions.

This “one set of weights, two modalities” design allows image understanding to transfer naturally to video understanding, eliminating the need for video-specific modules or branches. It also reduces parameter count and training complexity while maximizing cross-modal knowledge reuse.

Projection Layer and MoE Language Backbone

The projection layer’s job—mapping visual features to the language embedding space—may seem simple, but design choices matter. Using a lightweight MLP rather than a complex cross-attention module reduces coupling friction between modalities and makes joint training more stable. The language backbone commonly adopts a Mixture-of-Experts (MoE) structure, using sparse activation to balance capacity and compute: total parameter count determines knowledge capacity, while each forward pass activates only a subset of experts. This balances large capacity with low inference cost, which is especially critical for agent scenarios requiring long contexts and multi-turn tool calls.

Decoupled Encoder Training Pipeline

A key engineering challenge in large-scale multimodal training is that the visual encoder and language backbone converge at different rates. Training them jointly from start to finish often leads to gradient conflicts and instability. A practical solution is a decoupled training pipeline:

1. Stage one (separate pretraining): The visual encoder is pretrained on large-scale image-text and video data, while the language backbone performs text pretraining in parallel.
2. Stage two (alignment adaptation): The visual encoder is frozen or fine-tuned with a low learning rate, with the focus on training the projection layer and the language backbone for cross-modal alignment.
3. Stage three (joint fine-tuning): Full-parameter joint training with a modality-mixed data ratio and loss weighting to prevent visual data from degrading language capabilities.

The essence of decoupling is “build each component first, then fuse and advance together.” It ensures both efficiency and stability in large-scale joint training, making it a key engineering step for moving multimodal training from the lab to production scale.

Agent Parallel Orchestration: From Serial Loops to Cluster Scheduling

The Latency Bottleneck of Serial Execution

A traditional single agent follows a “perceive-plan-act-observe” loop where all steps are strictly serial. Consider a task like “research and generate a visualized report.” The chain might be: retrieve sources → read and summarize → write code → debug and run → produce charts. Each step waits for the previous one, and total latency is the sum of all steps. Yet many subtasks—such as multiple parallel web searches or cleaning several datasets—have no interdependencies. Serial execution wastes time on avoidable waiting.

Agent Clusters: Dynamic Parallel Orchestration

The parallel orchestration framework introduces three roles: planner, executor, and aggregator.

• Planner: Decomposes the overall goal into subtasks and produces an explicit dependency graph (DAG).
• Executor: Multiple heterogeneous agent instances concurrently claim subtasks that have no dependencies, each equipped with tool sets (search, code execution, browser control, etc.).
• Aggregator: Collects results in topological order, handles retries on failure, and produces the final merged output.

An illustrative task decomposition:

{
  "goal": "Research and generate a data visualization report",
  "subtasks": [
    { "id": "t1", "action": "web_research", "deps": [] },
    { "id": "t2", "action": "data_clean",   "deps": [] },
    { "id": "t3", "action": "code_gen",     "deps": ["t1", "t2"] },
    { "id": "t4", "action": "render_chart", "deps": ["t3"] }
  ]
}

The corresponding scheduling pseudocode:

async def run_orchestrated(goal: str):
    dag = planner.decompose(goal)           # Dynamic task decomposition
    ready = [t for t in dag if not t.deps]  # Subtasks without deps run concurrently
    results = await scheduler.execute(
        dag, initial=ready, max_concurrency=4
    )
    return aggregator.merge(results)        # Result aggregation and cross-validation

Public technical reports show that this dynamic parallel orchestration can reduce end-to-end latency for complex tasks by up to approximately 4.5x. More importantly, parallel execution often improves task quality as well: multiple subtask paths can cross-validate each other, and the aggregation stage can arbitrate conflicting conclusions, reducing the risk of hallucination propagation along a single chain.

Engineering Considerations for Parallel Orchestration

• Dependency granularity: Overly fine decomposition increases scheduling overhead; overly coarse decomposition limits parallelism. Aim for the smallest independently deliverable unit.
• Concurrency limits: Tool-side rate limits and token costs require throttling via token buckets or semaphores.
• Failure isolation: A single subtask failure should not collapse the whole workflow. Use a three-stage strategy: timeout, retry, and graceful degradation.
• Result consistency: For operations with side effects (file writes, environment changes), use locking or serialization.

Tool Calling and Computer Control

The other half of agent capability comes from “knowing how to use tools.” Function calling allows the model to invoke external interfaces with structured parameters. Computer control goes further, requiring the model to understand screen screenshots, locate UI elements, and generate keyboard and mouse action sequences. These two directions precisely validate the value of vision-language joint training: tasks like screen understanding, chart reading, and document parsing require both visual perception and strong reasoning. Single-modal models struggle here, which is exactly where the unified multimodal architecture excels.

Evaluation Dimensions and Deployment Recommendations

When evaluating such multimodal agent models, we recommend covering five dimensions: code generation, visual understanding, mathematical reasoning, agent execution, and computer control. Pay attention to the balance of capabilities across tasks rather than any single leaderboard score.

Practical deployment recommendations:

1. Avoid destructive preprocessing of visual input; preserve native resolution and aspect ratio whenever possible.
2. For workflows with clear parallelization opportunities (batch retrieval, multi-document processing), prefer an orchestration framework over a single serial loop.
3. For deployment validation, consider general cloud capabilities—for example, container platforms for hosting inference services, object storage for managing multimodal data, and monitoring and alerting systems for tracking latency and success rates.
4. Set cost and concurrency guardrails for parallel orchestration to avoid “higher parallelism, higher bills.”

Conclusion

Two clear threads define the current evolution of multimodal agent technology: moving from post-hoc vision adaptation to native joint optimization, and from single-agent serial loops to cluster-based parallel orchestration. The first addresses the model’s ability to “see clearly and reason deeply”; the second addresses “running fast and operating reliably.” For developers, understanding the design trade-offs behind these two threads—the information fidelity of patch packing, the knowledge transfer of spatiotemporal packing, the stability of decoupled training, and the latency gains of DAG orchestration—offers more long-term value than chasing a single model score. It also better equips you to build efficient, controllable agent systems in real-world applications.