Multimodal AI Agents Explained: Native Vision Modeling and Parallel Orchestration

    This article breaks down the technical upgrade of multimodal AI agents, contrasting the long-standing adapter-based architecture with an emerging native approach in which vision and language are jointly optimized from the start of pretraining. It walks through a unified multimodal architecture made up of a vision encoder, a cross-modal projector, and a language backbone, then examines key capabilities: native-resolution vision encoding via patch packing, unified spatiotemporal modeling of images and videos, a sparse Mixture-of-Experts backbone, and decoupled encoder training. It also covers building tool capabilities that move an agent from understanding to action, plus parallel agent orchestration that rewrites serial workflows as a DAG to cut end-to-end latency. The article closes with evaluation dimensions and practical engineering recommendations, offering readers a reusable framework for the selection, design, and optimization of multimodal agent systems.

    1. Introduction: From “Adapter-Based” to “Native”

    Recently, a team open-sourced a multimodal agent large model whose core goal is to push the boundaries of “general-purpose agent intelligence.” The approach rests on two main ideas:

    • Unified modeling: Vision and language are jointly optimized throughout, rather than first training a pure text model and bolting on a vision encoder later.
    • Parallel execution: An agent-cluster-style dynamic orchestration compresses the execution latency of complex tasks.

    Beyond the model release itself, the more valuable takeaway is what this technical route implies for multimodal agent engineering practice. Drawing on “source materials plus general technical knowledge,” this article abstracts a reusable technical framework for multimodal agents, intended as a reference for engineers when making decisions about selection, design, and optimization.

    1.1 Comparing Two Architectural Paradigms

    For a long time, mainstream multimodal models have largely adopted an “adapter-based” architecture, trained in three stages:

    1. First train a language model on large-scale pure-text corpora;
    2. Freeze the language model and align a vision encoder using image-text paired data;
    3. Fine-tune with multimodal instruction data.

    The engineering value of this route is unquestionable. The language model first builds solid knowledge and reasoning ability, the visual modality is added later, training costs stay controllable, and engineering maturity is high. But as model capabilities extend into agent scenarios, its inherent weaknesses gradually appear:

    • Weak cross-modal synergy: Visual representations are shaped by the limited data in the alignment stage and cannot be optimized in reverse through the massive training of the language model. Tasks such as fine-grained visual reasoning and chart parsing tend to “see but fail to use” the information.
    • Semantic space misalignment: When visual features are projected into the language embedding space, alignment error exists and requires large amounts of alignment data to compensate; at inference this shows up as missed image details or hallucination.
    • Preprocessing information loss: To fit fixed-length input, images are typically cropped or resized. Small text and edge controls in long images, long documents, and UI screenshots therefore become distorted.
    • Cross-modal interference: Late-stage parameter updates can easily damage existing language-model capabilities, producing a “vision improves, language regresses” trade-off.

    The “native” architecture takes the opposite approach: it treats vision as a first-class input modality on par with text, jointly optimized with text from the very start of pretraining. General practice shows the benefits are bidirectional:

    • Vision reinforces language: Multimodal data such as images, text, and video enrich the model’s perception of the real world, supplementing knowledge of spatial relations, physical common sense, and event causality.
    • Language reinforces vision: Strong language priors provide semantic constraints for visual understanding, helping the model give more accurate answers in scenarios such as coreference resolution and implicit-intent understanding.

    1.2 From “Understanding and Generation” to “Perception and Action”

    The capability structure of a single chat assistant can be roughly decomposed into three parts: language ability, visual ability, and tool calling / task execution. Multimodal agents further require the model to:

    1. See accurately: Accurately extract details from high-resolution images, long documents, and videos, including text, numbers, icons, and spatial layout;
    2. Reason clearly: Combine visual evidence with textual constraints to perform reasoning, planning, and decision-making;
    3. Act reliably: Call external tools such as search, code execution, file processing, and UI operation in a structured way, and iteratively correct based on execution feedback.

    The three are tightly interlinked: a clear weakness in any one constrains overall performance. Therefore, upgrading a multimodal agent cannot stop at “swapping in a stronger language model” or “plugging in a new vision model,” but must advance systematically across three dimensions: modal fusion, inference efficiency, and task orchestration.

    2. Overview of the Unified Multimodal Architecture

    A multimodal model designed for native fusion typically consists of three major components: a vision encoder, a cross-modal projector, and a language backbone. An engineering decoupled encoder training pipeline works alongside them to ensure large-scale multimodal training stays stable and efficient. The overall pipeline is:

    1. Image / Video
    2. │
    3. ▼
    4. ┌─────────────────────────────────────┐
    5. │ Vision encoder (native resolution, │ ← trained independently,
    6. │ unified spatiotemporal) │ decoupled gradients
    7. └─────────┬─────────────────────────────┘
    8. │ visual feature sequence
    9. ▼
    10. ┌─────────────────────────────────────┐
    11. │ Cross-modal projector (aligns to │
    12. │ the embedding space) │
    13. └─────────┬─────────────────────────────┘
    14. │ visual token embeddings
    15. Text input ─┤
    16. ▼
    17. ┌─────────────────────────────────────┐
    18. │ Language backbone (autoregressive) │
    19. └─────────┬─────────────────────────────┘
    20. ▼
    21. Text output / structured tool calls / action operations

    2.1 Design Point 1: Separating Feature Extraction from Semantic Modeling

    The vision encoder operates only within the visual domain, responsible for converting pixels into compact features that contain information such as shape, texture, and spatial structure. The language backbone faces only a unified token sequence and does not need to perceive the modality source. The two are connected by the projector, with clear boundaries of responsibility.

    This separation brings at least four benefits:

    • The vision encoder can reuse mature visual-representation training objectives and data pipelines and iterate independently;
    • The language backbone continues to benefit from the knowledge and reasoning ability of text corpora;
    • Multimodal training can be organized as “feature caching + streaming reads,” reducing peak demand for online compute;
    • When debugging, problems can be localized by modality and by module.

    2.2 Design Point 2: Unified Spatiotemporal Representation

    Modeling images and videos with two separate approaches both increases system complexity and weakens cross-task transfer ability. A sensible approach is to let the two share the same feature space:

    • Images are treated as a single-frame video sequence;
    • Videos are treated as a multi-frame sequence with timestamps;
    • Position encoding includes three dimensions—time, height, and width—so the model can distinguish content at the same position at different moments;
    • Redundancy between frames is handled through unified spatial and spatiotemporal compression mechanisms.

    The following sections expand along this main thread, covering native-resolution vision encoding, the sparse backbone, decoupled training, and parallel orchestration.

    3. Key Capability Analysis

    3.1 Native-Resolution Vision Encoding: No More “One Size Fits All”

    Fixed-resolution preprocessing has gradually become a bottleneck in the large-model era. The mainstream approach adopts a sequence-packing (patch packing) strategy that splits an image into patches and flattens them into a sequence input:

    1. On-demand partitioning: Dynamically determine the patch grid based on the original dimensions, keeping the aspect ratio unchanged, with no resizing or cropping;
    2. Variable-length packing: Pack multiple differently sized inputs (images or video frames) into the same sequence, carrying boundary markers to avoid interference between samples;
    3. Position encoding restoration: Each patch carries three-dimensional position information, letting the model accurately recover its spatiotemporal coordinates in the original image or video;
    4. Visual token control: Through mechanisms such as spatial and spatiotemporal merging, balance information retention against context cost.

    The schematic logic is as follows:

    1. def pack_visual_inputs(samples, grid_size, max_tokens):
    2. """Pack images/video frames of different resolutions into a unified visual sequence."""
    3. seq, segments = [], [] # seq: visual token sequence; segments: token range per sample
    4. for s in samples:
    5. # 1. Dynamically determine the patch grid from the original size, preserving aspect ratio
    6. h_patches = max(1, round(s.height / grid_size))
    7. w_patches = max(1, round(s.width / grid_size))
    8. # 2. Compute the number of visual tokens (incl. spatiotemporal merging); subsample frames if needed
    9. tokens = visual_budget(h_patches, w_patches, s.num_frames, max_tokens)
    10. feats = visual_encoder.encode(s, tokens) # native-resolution encoding
    11. feats = merge_spatiotemporal(feats, tokens) # spatiotemporal merging to control sequence length
    12. start = len(seq)
    13. seq.extend(project(feats)) # project into the language embedding space
    14. # 3. Record the [start, end) interval for attention-mask sample isolation
    15. segments.append((start, len(seq)))
    16. return seq, segments

    This design brings three benefits:

    • Small text, dense tables, and UI icons in fine-grained visual tasks are preserved;
    • Long documents and high-resolution screenshots no longer sacrifice clarity;
    • Variable-length packing reduces padding waste and improves training and inference throughput.

    Practical deployment still requires trade-offs:

    • Higher resolution means longer sequences, and attention cost rises accordingly;
    • Use visual-token budgets, on-demand resizing, and caching mechanisms to strike a balance between accuracy and cost.

    3.2 Unified Spatiotemporal Modeling of Images and Videos

    The core of a native 3D vision encoder is to let images and videos share the same modeling logic:

    • Images: the time dimension degenerates to a single frame, and spatial position encoding is reused directly;
    • Videos: position encoding covers both time and space, and inter-frame content can be further compressed to reduce sequence length;
    • Joint training: images, interleaved image-text documents, short videos, and long videos can be mixed and sampled within the same training run, strengthening the model’s understanding across time scales.

    A schematic implementation is as follows:

    1. def build_3d_position_ids(frame_idx, h_patches, w_patches):
    2. """Generate (t, h, w) 3D position indices for patches in a video."""
    3. return [
    4. (frame_idx, i // w_patches, i % w_patches)
    5. for i in range(h_patches * w_patches)
    6. ]

    In practice, video modeling also requires attention to the following details:

    • Sampling strategy: Equal-interval frame sampling is simple to implement but may miss key moments; introducing change-detection-based adaptive sampling can allocate the frame budget more sensibly.
    • Audio-visual coordination: Speech and sound effects in a video carry important semantics and can enter the backbone together with visual features, but training data must ensure audio-visual synchronization and content consistency.
    • Temporal grounding: Long-video understanding often requires answering “at what second did an event occur,” so frame-level time identifiers should be aligned consistently with position encoding, avoiding vague time descriptions from the model.

    3.3 Sparse Language Backbone: Balancing Performance and Efficiency

    Modern multimodal-agent language backbones commonly adopt a Mixture-of-Experts (MoE) architecture whose core idea is sparse activation:

    • Gated routing: For each token, a routing network selects a small number of experts to participate in the computation;
    • Capacity differences: The total parameter count can be very large, but the number of parameters actually activated per token stays small;
    • Load balancing: An auxiliary loss encourages tokens to distribute evenly across experts, avoiding some experts being overloaded while others sit idle.

    For multimodal agents, the value of this design is that the model must simultaneously carry multiple capabilities—language knowledge, visual reasoning, code generation, and task planning—and insufficient parameter capacity easily leads to “mutual exclusion among multiple tasks.” A sparse backbone expands capacity while keeping inference cost relatively controllable, providing a foundation for multiple capabilities to coexist.

    Three points deserve attention in engineering implementation:

    1. Load-balancing monitoring: If one expert is selected by too many tokens, training efficiency drops and inference latency rises.
    2. Memory and communication overhead: Expert parameters must be scheduled across devices in distributed training, and the expert-parallelism strategy directly affects throughput.
    3. Routing distribution of multimodal tokens: Whether the routing distribution of image tokens and text tokens differs systematically can serve as a diagnostic signal for judging whether modal fusion is sufficient.

    3.4 Decoupled Encoder Training: Letting Vision and Language “Train Separately”

    The challenge that native fusion brings is that the vision encoder and the language backbone become coupled during training, and their gradients interfere with each other. If the learning rate is set improperly, problems can arise—the language backbone may be “pulled off course” by the vision task, or the visual representation may be repeatedly updated with large swings before it has stabilized.

    A common engineering solution is decoupled encoder training:

    • Stage 1: Independent pretraining of the vision encoder. Train the encoder separately on visual data such as images, text, and video to learn general visual representations.
    • Stage 2: Projector alignment. Freeze both backbones and train only the projector, mapping visual features into the language embedding space.
    • Stage 3: Incremental training of the language backbone. While preserving text capabilities, gradually mix multimodal data into training.
    • Stage 4: Instruction and agent-data fine-tuning. Use data such as tool-calling traces and UI-operation records to strengthen task-execution ability.

    Each stage uses differentiated learning rates, gradient clipping, and loss weights so that vision and language optimize at their own pace. Combined with data-mixture management and continuous regression evaluation (for example, comparing text benchmarks before and after multimodal training), capability regressions can be detected early. Schematic:

    1. def train_decoupled(vision, projector, llm, stages):
    2. """Staged decoupled training: each stage updates only the specified modules with differentiated optimization settings."""
    3. for st in stages:
    4. set_trainable(vision, st.name in ("vision_pretrain", "joint_finetune"))
    5. set_trainable(projector, st.name in ("align", "joint_finetune"))
    6. set_trainable(llm, st.name in ("joint_finetune", "agent_sft"))
    7. optim = build_optimizer(
    8. lr=st.lr, # differentiated learning rate per stage/module
    9. params=[p for m in (vision, projector, llm) if is_trainable(m) for p in m.parameters()],
    10. )
    11. for batch in st.dataloader:
    12. loss = compute_loss(vision, projector, llm, batch, st)
    13. optim.step(loss)

    The significance of this approach is that a multimodal model’s performance does not come from any single “stronger” module, but from the coordinated evolution of all modules under the right order and the right constraints.

    4. Tool Capabilities: From “Understanding” to “Acting”

    The value of a multimodal agent ultimately shows in whether it can complete tasks. Building tool capability can be broken into five parts.

    4.1 Tool Interface Design

    Each tool should include a clear name, purpose description, parameter structure (type, required or not, value range), and return format. The more precise the parameter definitions, the higher the accuracy with which the model selects tools and fills in parameters. Recommendations:

    • Keep tool granularity moderate; avoid “one tool does everything”;
    • Return structured results, avoiding overly long text that drowns out key information;
    • Provide examples for typical parameters.

    4.2 Multimodal Tool Calling

    Visual tasks often need dedicated capabilities, with common approaches including:

    • Document parsing;
    • Image cropping and resizing;
    • Optical character recognition (OCR);
    • UI element localization.

    These capabilities are registered with the agent as standardized tools, and the model decides autonomously when to call them based on the task. For example, facing a complex report, the agent can first decide that “the table needs to be read precisely,” then call a structured-parsing tool to obtain row and column data, and combine it with visual understanding to complete the analysis.

    4.3 Structured Output Constraints

    A tool call is essentially generating a piece of structured data that an external system can parse; in practice, techniques such as constrained decoding can be used to guarantee valid formatting. Before execution, parameter validation and permission checks must also be completed, and operations involving sensitive actions should require human confirmation.

    4.4 Execution Feedback Loop

    After a tool returns, the agent should replan in light of the result:

    • If a call fails or a result is abnormal, analyze the cause, adjust parameters, and retry;
    • If the task has been split into multiple sub-goals, decide the next action based on intermediate results;
    • If the retry limit is reached, clearly explain the failure reason and completed steps to the user.

    4.5 Multimodal Observation

    In scenarios such as computer control (UI operation), the agent takes screenshots as observation input and outputs actions such as clicks, typing, and scrolling. High-resolution vision encoding preserves details such as buttons, menus, and text labels, which is a prerequisite for completing operations accurately. Engineering implementation requires attention to:

    • Mapping visual coordinates to coordinates the system can execute;
    • Setting wait and retry strategies for uncertain states such as pop-ups and asynchronous loading;
    • Adding a confirmation step for high-risk actions.

    5. Parallel Agent Orchestration: Breaking the Serial-Latency Bottleneck

    5.1 The Latency Dilemma of Serial Execution

    A traditional single agent uses a “think–act–observe” loop in which every reasoning step and tool call happens only after the previous one completes. If a task needs n steps, total time is roughly the sum of each step’s time:

    1. Serial: reason₁ → tool₁ → reason₂ → tool₂ → ... → reasonₙ → toolₙ
    2. └────────────────────── total latency = sum of per-step times ──────────────────────┘

    In reality, the number of steps often far exceeds expectations: in UI operation, a single page-load failure can trigger a retry; multi-document analysis means reading document by document and waiting proportionally longer; and across multiple tool-calling rounds, network and retrieval latency accumulate into the main overhead. More importantly, many steps have no real dependency between them, so serial execution actively forgoes the benefits of concurrency.

    5.2 The Basic Idea of Parallel Agent Orchestration

    The core of parallel orchestration is to rewrite a sequential process as a directed acyclic graph (DAG) to maximize concurrency:

    1. Task decomposition: A planning agent breaks the goal into parallelizable and serial subtasks and annotates dependencies;
    2. Parallel execution: Subtasks without dependencies are dispatched to different agent instances that run simultaneously, each configurable with a different toolset and context;
    3. Result aggregation: Aggregate the outputs of each instance, deduplicate and resolve conflicts, and form a unified result;
    4. Dynamic replanning: Adjust the subsequent plan based on intermediate results—for example, after parallel search uncovers a key clue, add a dedicated analysis;
    5. Resource scheduling: Allocate compute resources and timeout budgets by task type to avoid a single slow task dragging down the whole.

    Schematic:

    1. def run_agent_swarm(goal, planner, tools, max_rounds=5):
    2. """Parallel agent orchestration: run dependency-based tasks concurrently per round, replanning dynamically from intermediate results."""
    3. plan = planner.decompose(goal) # generate the task DAG
    4. results = {}
    5. for _ in range(max_rounds):
    6. ready = [n for n in plan.nodes
    7. if n.id not in results and all(dep in results for dep in n.deps)]
    8. if not ready:
    9. break
    10. # Tasks without dependencies within the same round run concurrently
    11. with ThreadPool(max_workers=plan.max_parallel) as pool:
    12. futures = {n.id: pool.submit(n.run, tools, results) for n in ready}
    13. for nid, fut in futures.items():
    14. results[nid] = fut.result(timeout=plan.node_timeout)
    15. plan = planner.replan(goal, results) # adjust the plan from intermediate results
    16. return planner.merge(results)

    5.3 The Theoretical Bound of Latency Optimization

    Task latency can be approximated as:

    Total latency ≈ critical-path time + scheduling overhead + result-merging time

    • Critical path: When dependencies cannot be eliminated, it can be shortened by optimizing single-step inference speed, compressing context, and caching repeated computations;
    • Scheduling overhead: Splitting tasks too finely raises the share of scheduling overhead, so granularity must be set sensibly;
    • Result merging: The merging stage itself requires a round of inference, and its input grows with the number of subtasks.

    Therefore, “parallel is always faster” does not hold. The sensible judgment is: parallelism yields the most significant gains when the serial chain contains many mutually independent steps; if the steps are highly coupled, optimization should shift to single-step efficiency.

    5.4 Engineering Practice Points

    While improving speed, parallel agent orchestration also brings new engineering challenges:

    • Context consistency: Parallel instances each hold local context, and information conflicts can arise during merging, so clear priority rules and conflict-detection mechanisms are needed;
    • Cost and resource control: Parallelism means more concurrent calls, so set global budget caps, task timeouts, and concurrency limits;
    • Idempotency and side effects: Subtasks involving write operations must guarantee idempotency or be serialized to avoid duplicate submissions;
    • Observability: Record each subtask’s input, output, latency, and failure reason to help locate bottlenecks;
    • Fallback strategy: When parallelism fails, the system should fall back to serial execution to ensure the task ultimately completes.

    6. Evaluation and Engineering Recommendations

    6.1 Evaluation Dimensions

    Evaluating a multimodal agent should not rely on a single leaderboard; it is better to measure it across the following dimensions:

    Dimension Focus Typical Tasks
    Code generation Correctness, executability, engineering standards Algorithm implementation, scripting, code fixes
    Visual understanding Detail recognition, spatial relations, document parsing Chart Q&A, UI understanding, document information extraction
    Mathematical reasoning Multi-step derivation, rigor Competition problems, word problems, symbolic computation
    Agent execution Task completion rate, tool-call accuracy Multi-step tool calling, information retrieval and integration
    Computer control Operation success rate, step efficiency UI operation, form filling, cross-app workflows

    Three points also deserve attention during evaluation:

    • Distinguish “capability ceiling” from “actual usability”: input quality, network conditions, and tool stability in real scenarios all affect final performance;
    • Pay attention to latency and cost metrics: at equal capability, response time and call cost often determine whether a solution can scale;
    • Build your own evaluation set: public leaderboards deviate from business scenarios, so construct regression test sets based on real tasks and continuously track changes from model iterations.

    6.2 Engineering Recommendations

    1. Define capability boundaries: Multimodal agents suit tasks with a long “perception–understanding–decision–action” chain; for simple Q&A, use a general-purpose model directly and avoid the extra cost of architectural complexity.
    2. Build a high-quality data pipeline: Multimodal data requires collection, cleaning, labeling, and version management; image resolution, video frame rate, and image-text alignment quality all directly affect training results.
    3. Design the toolset sensibly: Clear tool descriptions and stable return structures are prerequisites for reliable agent execution.
    4. Establish a layered evaluation system: Cover from unit-test level (single-step reasoning, single tool call) to end-to-end level (complete task flow) layer by layer.
    5. Make cloud deployment elastic: Using general capabilities from cloud platforms such as Baidu AI Cloud—container services, object storage, message queues, logging services, and monitoring/alerting—makes it possible to achieve elastic scaling of inference services, massive storage of training data, decoupled scheduling of asynchronous tasks, and full-link observability.

    7. Summary

    From the perspective of technical evolution, the upgrade path of multimodal agents can be summarized along three main lines:

    1. Modal fusion moves from “concatenation” to “native”: A unified architecture lets vision and language be jointly optimized in the same semantic space, reducing modal misalignment and mutual capability damage;
    2. Vision encoding moves from “fixed” to “adaptive”: Native resolution and unified spatiotemporal modeling let the model preserve detail perception in the face of high-resolution images and long videos;
    3. Task execution moves from “serial” to “parallel”: Agent-cluster orchestration, through task decomposition, concurrent execution, and dynamic replanning, significantly compresses the end-to-end latency of complex tasks.

    For development teams, understanding these design ideas matters more than chasing a single metric. Choosing the right architecture for the business scenario, building a high-quality data and tool ecosystem, and establishing a scientific evaluation and operations system are the keys to truly putting multimodal agents into production and continuously creating value.