Native Multimodal Agents: Vision Encoding and Dynamic Parallel Orchestration

1. Background: From Late-Stage Modality Adaptation to Native Multimodal Fusion

Over the past two years, the dominant approach to building multimodal large models has followed a two-stage paradigm: “language first, vision adapted later.” A pure text language model is trained first, then a vision encoder is attached and lightly aligned. This approach is simple to implement but suffers from two structural flaws:

• Modality conflict: When the vision encoder is connected late in the process, the parameters of the language backbone have already converged, and visual signals tend to be treated as “foreign noise,” leading to unstable performance on visual understanding tasks.
• Trade-off between capabilities: Making the model “see images” often requires compromising language ability, resulting in degraded text reasoning performance.

To address these issues, the industry has begun exploring native multimodal architectures—where vision and language are jointly optimized within the same semantic space from day one of training. At the same time, sequential execution in single-agent systems has exposed a clear latency bottleneck in complex tasks: a task may require calling multiple tools and multiple rounds of reasoning in sequence, and the time cost of serial execution grows linearly with the number of steps.

This article focuses on these two technical threads: first, the architectural design of native multimodality; second, a dynamic parallel orchestration framework for complex tasks. It does not cover specific commercial products, only reusable engineering methods and technical principles.

2. Overall Architecture: Three Core Components

The end-to-end pipeline of a native multimodal agent can be summarized as “visual feature extraction → cross-modal projection → unified language backbone modeling.” It consists of three core components:

1. Native vision encoder: Extracts unified visual features from images and videos.
2. Cross-modal projector: Aligns visual features to the language model’s embedding space.
3. MoE language backbone: Uses a mixture-of-experts architecture to carry end-to-end modeling of both text and visual tokens.

The data flow is shown below (pseudocode):

# Pseudocode: native multimodal forward pass
visual_features = vision_encoder(images, videos)   # native resolution, no cropping
visual_tokens   = projector(visual_features)       # align to language embedding space
text_tokens     = tokenizer(text)
inputs          = concat(visual_tokens, text_tokens)
outputs         = moe_backbone(inputs)             # end-to-end joint modeling

The key point: visual tokens and text tokens enter the same sequence from the input layer and are processed by the same attention mechanism, rather than having visual information “injected” at an intermediate layer. This design allows the two modalities to reinforce each other during training rather than interfere with each other.

3. Design Considerations for Native-Resolution Vision Encoders

The vision encoder sets the upper bound on multimodal understanding quality. Traditional approaches typically resize input images to a fixed resolution (e.g., 224×224 or 336×336), which introduces two problems: loss of detail in high-resolution images and distortion of images with non-standard aspect ratios.

3.1 Patch Packing and Variable Input

Drawing on the idea of patch packing, images of arbitrary resolution can be split into patches and flattened into a 1D sequence without forced resizing:

# Pseudocode: variable-resolution patch packing
patches = split_into_patches(image, patch_size=14)  # arbitrary aspect ratio
flat_seq = flatten(patches)                          # 1D sequence
# Different images yield different sequence lengths, handled by attention masks

The benefits are:

• No preprocessing loss: Raw pixel information is fully preserved, which notably benefits fine-grained scenarios such as documents, tables, and UI screenshots.
• Arbitrary aspect ratio support: No padding to a square is needed, reducing wasted computation.
• Unified image and video modeling: Video can be treated as a temporally stacked patch sequence, sharing the same encoder as images and reducing architectural complexity.

3.2 Decoupling the Engineering Training Pipeline

Native-resolution input introduces problems such as large fluctuations in sequence length and unstable memory usage. A common engineering practice is to decouple encoder training: the training cadence of the vision encoder and the language backbone is separated. The encoder first converges fully on vision tasks, then connects to the language backbone with a relatively stable feature distribution. This reduces the difficulty of joint training and avoids the impact of gradient fluctuations on the vision side affecting language capabilities.

4. Dynamic Parallel Orchestration: Breaking the Sequential Execution Latency Bottleneck

Multimodal understanding is only the first step. A truly general-purpose agent also needs to call tools, perform multi-step reasoning, and operate interfaces. As task complexity increases, the latency of sequential single-agent execution becomes the primary bottleneck.

4.1 Problem Formulation

Suppose a task requires N steps with an average duration of t per step; the total serial latency is approximately N×t. If data dependencies exist between steps, full parallelism is impossible. But in practice, many steps are weakly dependent or separable, for example:

• Retrieving information from multiple sources simultaneously;
• Calling multiple tools in parallel and aggregating results;
• Independently analyzing the same material from different perspectives.

4.2 Design of a Parallel Orchestration Framework

The core idea of dynamic parallel orchestration is: a scheduler identifies parallelizable subtasks at runtime, dynamically spawns sub-agents to execute them, and finally aggregates the results.

# Pseudocode: dynamic parallel orchestration
def solve(task):
    subtasks = planner.decompose(task)          # dynamic decomposition
    parallel_groups = scheduler.group(subtasks) # identify dependencies
    results = []
    for group in parallel_groups:
        # execute in parallel within the same group
        group_results = parallel_execute(group)
        results.extend(group_results)
    return aggregator.merge(results)

Key design points include:

• Dependency graph construction: Explicitly model input/output dependencies between subtasks to avoid incorrect parallelism.
• Dynamic granularity adjustment: Do not split when the task is simple, to avoid scheduling overhead outweighing the benefit.
• Result aggregation: Parallel subtask outputs require consistency and conflict resolution mechanisms; otherwise, new sources of error are introduced.

4.3 Benefits and Costs

The benefit of parallel orchestration is a significant reduction in latency, theoretically approaching the critical path length rather than the total number of steps. The cost is increased scheduling complexity, higher peak resource usage, and a strong dependence on the quality of subtask partitioning. For production deployment, consider:

• First measure the split benefit ratio on small-scale tasks;
• Set a concurrency limit on the scheduler to prevent resource exhaustion;
• Keep a serial fallback path so tasks can still complete if parallel execution fails.

5. Evaluation Dimensions and Engineering Best Practices

When evaluating a multimodal agent system, it is advisable to build your own benchmarks across the following dimensions rather than relying on a single metric:

Dimension	Focus	Suggested Approach
Visual understanding	Fine-grained recognition, document/table parsing	Build a custom high-resolution test set
Code generation	Multi-language, long context	Cover real repository-level tasks
Mathematical reasoning	Stability of multi-step derivation	Controlled comparison of serial vs. parallel
Agent execution	Tool call success rate	Track end-to-end task completion rate
Computer control	UI element localization accuracy	Record real interaction traces for replay

A few lessons from engineering practice:

1. Align first, then jointly train: After the cross-modal projector is aligned with sufficient data, unfreeze the language backbone for joint training—convergence is more stable.
2. Balance resolution and cost: Native resolution is not always better; set an upper limit based on the business scenario to avoid excessively long sequences.
3. Parallel orchestration needs observability: Instrument the scheduler to record decomposition decisions, concurrency levels, and critical path length for tuning.
4. A fallback mechanism is essential: Any dynamic decision can fail; a serial fallback path is a prerequisite for production readiness.

6. Summary

Native multimodal architecture solves the modality conflict caused by “late-stage vision adaptation,” allowing vision and language to reinforce each other within the same semantic space. Dynamic parallel orchestration breaks the latency bottleneck of sequential single-agent execution at the execution layer. Together, they provide dual support for general-purpose agents in terms of both understanding capability and execution efficiency.

For developers, the key to deployment is not chasing a single metric, but building an evaluation system that matches your own business needs, understanding the trade-offs behind architectural choices, and leaving room in the engineering for parallelism, fallback, and observability. As open-source weights and training methods continue to open up, the technical barrier to general-purpose agents is declining, and true differentiation will come from scenario understanding and engineering refinement.