Native Multimodal Agents: Visual Encoding and Parallel Orchestration

    This article examines four technical directions for general-purpose multimodal agents: unified visual representation, joint vision-language training, tool-calling agents, and multi-agent parallel orchestration. It explains the design rationale behind each approach, outlines a basic implementation of dependency-graph scheduling, and offers practical guidance for evaluation and production deployment.

    Multimodal large models are moving from language-first systems with late visual adaptation toward native joint modeling. Agents are also moving from single-step sequential execution toward parallel orchestration. This article reviews four key directions: unified visual representation, joint vision-language training, tool-calling agents, and multi-agent parallel orchestration. Readers can learn the design logic behind each approach, understand a basic implementation of dependency-graph scheduling, and get practical guidance for evaluation and deployment.

    1. Background: From Adapted Multimodality to Native Multimodality

    Early multimodal systems often followed a “train the language model first, then attach a visual encoder” route. This approach has low engineering cost, but it tends to produce two problems. First, language and visual capabilities compete for parameter capacity during later training, so improving one may degrade the other. Second, visual features and the language embedding space are not sufficiently aligned, making the model unstable on tasks such as chart understanding, fine-grained localization, and temporal video reasoning.

    The native multimodal approach lets text and visual data participate in optimization during pretraining. Visual signals and text tokens are modeled in the same semantic space, and the two modalities provide supervision signals to each other. The cost is more complex training-data organization and higher compute-scheduling requirements, but the benefit is an overall improvement in cross-modal reasoning.

    2. Design Points for a Unified Multimodal Architecture

    A typical unified multimodal architecture usually contains three modules.

    1. Visual encoder: Converts an image or video into a set of visual feature vectors.
    2. Cross-modal projection layer: Maps visual features into the embedding dimension of the language model so that the two can be concatenated directly.
    3. Language backbone: Receives a mixed sequence of visual tokens and text tokens and performs end-to-end modeling.

    This pipeline can be summarized as visual feature extraction, cross-modal projection, and unified modeling in the language backbone. The design of the projection layer directly affects alignment quality. An overly simple linear mapping may lose spatial structure information, while an overly complex mapping can increase training instability. A common practice is to use a two-layer MLP with normalization and a suitable initialization strategy.

    2.1 Native-Resolution Input

    Traditional vision models often resize or crop images to a fixed size, such as 224×224. This preprocessing loses detail and is especially disadvantageous for high-information-density images such as documents, tables, and UI screenshots.

    The core idea of native-resolution input is to split an image into fixed-size patches and then flatten and concatenate patches from different images into a one-dimensional sequence. Positional encoding is adapted to the actual two-dimensional grid. This allows the model to handle inputs with arbitrary aspect ratios and reduces information loss caused by resizing.

    The engineering difficulty of this approach lies in batching. Different images have different numbers of patches, so variable-length sequence packing and attention masks are needed to ensure correct computation.

    2.2 Unified Representation for Images and Video

    Treating an image as a single-frame video is a common unification strategy. A further approach treats several consecutive frames as a spatiotemporal volume. Each frame is still split into 2D patches, and patches from multiple frames are packed jointly along the temporal dimension before being passed to the same encoder for spatiotemporal attention.

    This design has two benefits. First, there is no need to design a separate branch for video, which reduces model complexity. Second, spatial understanding learned during the image stage can transfer to video tasks, reducing annotation requirements for video data. Note that as the number of frames increases, the sequence becomes longer and attention computation grows rapidly. Therefore, frame-sampling strategy and temporal window length must be balanced against the specific task.

    2.3 Decoupled Encoder Training

    The visual encoder and the language backbone often learn at different rhythms. If they are fully coupled from the beginning, gradient conflicts may cause training oscillation. A common decoupled training approach proceeds in stages:

    • In the first stage, train or pretrain the visual encoder separately so that it learns stable representations from large-scale image-text pairs and video data.
    • In the second stage, freeze the encoder or update it with a low learning rate, train the projection layer, and align it with the language space.
    • In the third stage, jointly fine-tune the entire model and gradually increase the proportion of visual data in the mixed corpus.

    Staged training increases process-management overhead, but it significantly reduces instability risk during training and makes it easier to identify which module is causing a problem.

    3. Engineering Value of Visual Understanding

    The value of visual capability appears in concrete scenarios. The following are several typical applications.

    • UI understanding and automated operation: The model reads screenshots, identifies buttons, input fields, and status messages, and then decides the next action.
    • Document and table parsing: The model preserves layout structure and converts scanned documents into editable structured data.
    • Chart analysis: The model reads axes, legends, and numeric labels and answers trend and comparison questions.
    • Video content understanding: The model extracts a timeline of key events and supports retrieval and summarization.

    These scenarios have different requirements for resolution, detail preservation, and temporal modeling. Therefore, when evaluating visual capabilities, test sets should be split by scenario rather than relying only on an aggregate score.

    4. Tool-Calling Agents

    What distinguishes an agent from an ordinary question-answering model is that it can plan steps, call external tools, and adjust its strategy based on returned results. A basic execution loop is as follows:

    1. Understand the user goal and decompose it into subtasks.
    2. Select tools and generate structured parameters.
    3. Execute tool calls and obtain results.
    4. Evaluate whether the results satisfy the subtask requirements and decide whether to continue, retry, or report.

    Tool interface design has a major impact on reliability. The following principles are recommended.

    • Use explicit types and enum constraints for parameters to reduce the chance of invalid parameters.
    • Return a uniform structure from each tool, including status code, data body, and error information.
    • Add confirmation steps for operations with side effects, such as deletion, sending, or payment.
    • Set upper limits for timeouts and retries to prevent loops from consuming resources.

    5. Multi-Agent Parallel Orchestration

    The main bottleneck of single-agent sequential execution is latency. In a workflow with ten subtasks, if each subtask must wait for the previous one to finish, total time is the sum of all step durations. If subtasks have no dependencies, running them in parallel can greatly reduce total time.

    The key to parallel orchestration is the task dependency graph. Each node represents a subtask, and each directed edge represents a data or ordering dependency. The scheduler aims to run independent nodes simultaneously while satisfying dependencies.

    5.1 Scheduling Principle

    The scheduling process can be divided into three steps.

    1. Decompose the user goal into subtasks and label the input and output of each subtask.
    2. Build a dependency graph and check for circular dependencies.
    3. Launch ready nodes in batches according to topological order, and release downstream nodes after their dependencies complete.

    5.2 Example Code

    The following code demonstrates dependency-graph parallel execution based on asyncio. It is intended only to illustrate the idea and omits error handling and timeout control.

    1. import asyncio
    2. async def run_task(name, deps_results):
    3. # Replace this with an actual subtask call, such as model inference or a tool request
    4. await asyncio.sleep(0.1)
    5. return f"{name} done with {list(deps_results.keys())}"
    6. async def run_graph(graph, runners):
    7. # graph: {task_name: [dependency_task_names]}
    8. futures = {}
    9. results = {}
    10. async def execute(name):
    11. deps = graph[name]
    12. for dep in deps:
    13. if dep not in futures:
    14. futures[dep] = asyncio.ensure_future(execute(dep))
    15. dep_results = {d: await futures[d] for d in deps}
    16. return await runners[name](name, dep_results)
    17. for name in graph:
    18. if name not in futures:
    19. futures[name] = asyncio.ensure_future(execute(name))
    20. for name in graph:
    21. results[name] = await futures[name]
    22. return results
    23. graph = {
    24. "fetch_data_a": [],
    25. "fetch_data_b": [],
    26. "analyze_data": ["fetch_data_a", "fetch_data_b"],
    27. "generate_report": ["analyze_data"],
    28. }
    29. runners = {name: run_task for name in graph}
    30. print(asyncio.run(run_graph(graph, runners)))

    In this example, fetch_data_a and fetch_data_b run at the same time, and analyze_data starts only after both are complete. If each fetch step takes several seconds, the total time of parallel execution will be close to the duration of a single step rather than the sum of both.

    5.3 Risks of Parallel Orchestration

    Parallelism brings speedups but also new problems.

    • Resource contention: Multiple subtasks accessing the same API or database at the same time may trigger rate limits or lock contention. A concurrency limit should be set.
    • Result consistency: If parallel branches modify the same state, merge rules must be explicit to avoid later writes overwriting earlier ones.
    • Error propagation: When an upstream task fails, downstream nodes should skip, degrade, or terminate according to policy rather than silently producing incorrect results.
    • Observability: Logs from parallel workflows can easily interleave. Assign a unique identifier to each node and record start, end, and duration.

    The reduction in latency depends on the dependency structure. The longer the dependency chain and the fewer the parallel branches, the more limited the benefit. Therefore, before introducing parallel orchestration, analyze the critical path length of the task graph.

    6. Evaluation Methods and Interpretation

    Public benchmarks can reflect a model’s general level, but they should be used cautiously for selection.

    • Task coverage: Benchmarks usually cover code, vision, math, and agent tasks. Focus on the subset closest to your own business.
    • Evaluation conditions: Prompt format, tool permissions, sampling parameters, and retry counts all affect results. Confirm that conditions are consistent when comparing.
    • Data contamination: If training data overlaps with the test set, scores will be inflated. A private evaluation set built in-house better reflects real performance.
    • Cost dimension: In addition to accuracy, measure inference time per task, number of calls, and resource consumption.

    It is advisable to build a small but focused evaluation set for the business, keeping twenty to thirty typical samples per scenario, including failure cases, and run regression tests after every model or workflow change.

    7. Deployment Practices

    Deploying multimodal agents in production usually requires the following general capabilities.

    • Object storage: Store original images, videos, and intermediate artifacts, with lifecycle policies to control cost.
    • Message queue: Receive asynchronous tasks, smooth peak loads, and support retry of failed tasks.
    • Container platform: Elastically scale inference services and orchestration services based on load, and isolate runtime environments for different tasks.
    • Logging, monitoring, and alerting: Record every tool call and model output, monitor latency, error rate, and concurrency, and alert promptly when metrics are abnormal.

    Implementation can proceed in the following order.

    1. Select one high-value scenario with clear boundaries, such as document parsing or UI inspection.
    2. Build a minimum viable workflow and record complete inputs, outputs, and timing.
    3. Use logs to identify failure-heavy stages and prioritize improvements to prompts, tool interfaces, and data preprocessing.
    4. After the workflow is stable, introduce parallel orchestration and gradually expand the task scope.

    For security, follow the principle of least privilege. Tools called by the agent should expose only the necessary read and write permissions, and operations involving sensitive data should require human confirmation or audit trails. If model output directly drives external systems, it must pass structural validation.

    8. Conclusion

    Native multimodal architectures address the problem of visual and language capabilities constraining each other, and unified visual encoding lets images and video share representations. Tool calling moves models from answering questions to executing tasks, while parallel orchestration directly targets latency bottlenecks. Together, these three elements form a major technical route for general-purpose agents.

    In deployment, attention should focus on verifiable metrics: task success rate, end-to-end latency, error recovery capability, and unit cost. Validating in a small-scope scenario first and then expanding gradually is a reliable way to control risk.