Technical Analysis: Upgrading Vision and Tool Capabilities for Multimodal Agentic LLMs

1. The Paradigm Shift: Why “Text-First, Vision-Adapted-Later” is Unsustainable

The prevailing approach to building multimodal large models has historically been “text-first, vision-later”: training a strong language backbone on pure text corpora, then freezing or fine-tuning it while attaching a visual encoder and an adapter layer to “translate” images for the language model. While this route facilitates asset reuse and controls training costs, its structural flaws become increasingly apparent with the rise of agentic tasks:

1. Representation Ceiling: Visual signals are projected into a language representation space that has never “seen” visual concepts, inherently causing semantic alignment loss. The upper limit of fine-grained visual understanding (e.g., document tables, UI screenshots, geometric shapes) is locked in during the backbone shaping phase.
2. Modality Competition: Slight imbalances in learning rates or data ratios during adapter training can lead to a trade-off—visual capabilities improve while language capabilities regress, and vice versa.
3. Fragmented Task Capabilities: Agents require deep coupling of visual perception and tool invocation (view screenshot -> locate element -> call tool -> read result -> re-decide). Under the two-stage paradigm, this pipeline is both long and fragile.

The native multimodal approach offers a different solution: text and visual signals share optimization objectives from the pre-training phase, jointly shaping within the same representation space to achieve bidirectional enhancement. Visual signals provide “grounded” semantics for language representation, while language priors constrain the structuring of visual features. Recent public benchmarks indicate that models in this direction have achieved competitive performance against closed-source flagship models in code generation, visual understanding, mathematical reasoning, agent execution, and computer control, even surpassing them in certain dimensions.

2. Unified Architecture: Three Core Components and End-to-End Data Pipeline

A typical unified multimodal agent architecture consists of three core components:

• Vision Encoder: Transforms images and videos into a unified sequence of visual tokens, natively supporting variable resolution inputs.
• MLP Projection Layer: Aligns visual features to the embedding space of the language model, acting as a bridge for cross-modal semantic alignment.
• MoE (Mixture of Experts) Language Backbone: Carries large parameter capacity through sparse activation, handling unified autoregressive modeling for both visual and text tokens.

The end-to-end data pipeline is as follows:

Image / Video ──► Vision Encoder ──► Patch sequence (1D flattened)
Pure Text      ──► Tokenizer & Embedding ──► Text token sequence
                 │                  │
                 └──► MLP Projection Layer ◄──┘   Visual features align to language embedding space
                          │
                          ▼
              MoE Language Backbone (End-to-end joint modeling)

The key to this design is the “unified semantic space”: visual tokens and text tokens complete representation space alignment before entering the backbone, requiring no additional conversion during inference. During training, gradients can backpropagate end-to-end to the vision encoder, allowing the two modalities to shape each other during optimization rather than unilaterally adapting.

3. Native Resolution Vision Encoding: Core Details of Patch Packing

Traditional vision encoders typically require fixed-resolution inputs, meaning arbitrary-sized images must be scaled or center-cropped. While acceptable for natural photos, this is disastrous for agent scenarios: text becomes illegible when ultra-wide screenshots are compressed, key regions of long document charts are lost to cropping, and the relative positional information of UI elements is destroyed.

The strategy commonly known as patch packing offers a solution with three main points:

1. Chunking and Flattening: Images are divided into patches and directly concatenated into a 1D sequence, rather than forming a 2D grid. Combined with position encoding carrying 2D coordinate information, the model can still recover spatial structures.
2. Native Aspect Ratio: Images of any resolution and aspect ratio can be input without preprocessing, eliminating information loss from scaling and cropping at the source.
3. Variable-Length Batch Packing: Samples of different sizes within the same batch are packed by total token count, keeping computation approximately constant and GPU utilization stable.

The illustrative packing logic is as follows:

def pack_batch(samples, max_tokens):
    """Pack variable-length samples into batches with approximately constant token counts"""
    buffer, batches = [], []
    for s in samples:                      # s.tokens: number of patches for this sample
        if cur_tokens(buffer) + s.tokens > max_tokens:
            batches.append(flush(buffer))  # Note: pad sample boundary masks
        buffer.append(s)
    if buffer:
        batches.append(flush(buffer))
    return batches

Two engineering pitfalls are worth emphasizing: First, sample boundaries must be handled correctly during packing to block cross-sample information leakage via attention masks. Second, position encoding must simultaneously encode “intra-sample 2D coordinates” and “intra-sequence 1D positions,” otherwise the model may confuse spatial attribution. Furthermore, video can be treated as a sequence of frames; after frame-by-frame patching, they share the same representation space as image tokens, realizing unified image-text-video modeling—which is the foundation for agent tasks like screen recording replay and operation trajectory understanding.

4. Decoupled Training Pipeline: A Stabilizer for Large-Scale Multimodal Training

Joint training does not mean throwing everything into a single pot. The vision encoder and language backbone differ vastly in optimal learning rates, data ratios, and training objectives. Direct full-parameter joint training easily leads to oscillation or even divergence. Practice proves that a decoupled encoder training pipeline is a stabilizer for large-scale multimodal training:

• Phase 1: Independently train or adapt the vision encoder to produce high-quality general visual representations.
• Phase 2: Constrain the encoder with a low learning rate, focusing on training the cross-modal alignment capabilities of the projection layer and backbone.
• Phase 3: Full-parameter end-to-end joint optimization, allowing language and vision to mutually shape and approach the capability ceiling.

This progressive route of “build representation first, then align, then jointly optimize” balances the capability ceiling of joint optimization with the stability and reproducibility of thousand-GPU-scale training. In large-scale multimodal training, stability itself is throughput.

5. Parallel Agent Orchestration: Breaking the Sequential Execution Bottleneck

Vision and tool capabilities solve the problem of “understanding and invoking,” while execution efficiency depends on orchestration. When a single agent executes complex tasks, tool calls are inherently serial: read document -> retrieve -> calculate -> write code -> retrieve again… Latency accumulates linearly with task chain length. Worse, early information in a long context is easily diluted, affecting subsequent decision quality.

The solution offered by dynamic parallel orchestration (understood as an “agent cluster” style scheduling) involves three steps:

1. Task Decomposition: The main control model breaks down complex tasks into a sub-task graph with dependencies (DAG).
2. Parallel Dispatch: Dependency-free sub-tasks are simultaneously assigned to multiple agent instances, each independently calling tools to execute.
3. Result Aggregation: Branch results are uniformly aggregated, and the main control model performs conflict resolution and final decision-making.

def orchestrate(task):
    dag = planner.decompose(task)                # Decompose into sub-task DAG
    pending = set()
    results = {}
    with AgentPool(max_workers=N) as pool:       # Parallel agent instance pool
        pending |= {pool.submit(run_agent, t) for t in dag.ready_nodes()}
        while pending:
            done, _ = wait(pending, return_when=FIRST_COMPLETED)
            for fut in done:
                pending.discard(fut)
                node, res = fut.node, fut.result()
                results[node.id] = res
                pending |= {pool.submit(run_agent, nx)
                            for nx in dag.unlock(node, results)}
    return aggregator.merge(task, results)       # Conflict resolution and aggregation

This pattern brings dual benefits: First, latency—dependency-free branches execute in parallel, and industry practices show end-to-end latency can be reduced several times under specific task combinations. Second, quality—each branch maintains a focused independent context, significantly mitigating information dilution and improving sub-task completion rates.

Engineering trade-offs cannot be ignored:

• Parallelism vs. Consistency: Sub-tasks with implicit dependencies must avoid conflicts through explicit dependency graphs or iterative validation.
• Context Management: The aggregation phase must summarize and compress branch results to prevent main control context explosion.
• Failure Handling: Single branch failures should support retry and degradation without blocking the execution of the entire graph.

6. Evaluation Dimensions: How to Scientifically Measure Multimodal Agents

A comprehensive evaluation should cover at least five dimensions: code generation (function-level to repository-level), visual understanding (chart parsing, UI element localization), mathematical and symbolic reasoning, agent execution (tool call success rate, multi-step task completion rate), and computer control (GUI operation sequence correctness). Simultaneously, be wary of benchmark contamination and “leaderboard chasing.” A more reliable approach is to build an in-house business replay set, using real task distributions to verify long-term stable performance, and focusing on analyzing long-tail failure modes rather than just looking at average scores.

7. Engineering Implementation: Development Paths and Infrastructure Recommendations

1. Data First: The ratio and quality filtering of multimodal data determine the capability ceiling. It is recommended to establish a quality grading pipeline for image-text and video-text pairs rather than blindly accumulating volume.
2. Training Infrastructure: The three phases of decoupled training demand high elasticity, fault recovery, and experiment tracking. Cloud-native general capabilities can accelerate this setup. For instance, general services like Baidu AI Cloud’s container platforms, object storage, logging services, and monitoring alerts can be utilized for training cluster scheduling, data storage, and experiment metric tracking, avoiding reinventing the wheel.
3. Inference Optimization: The number of visual tokens is typically much larger than text tokens, making it a primary source of inference cost. Focus on token compression, KV Cache reuse, and dynamic batching.
4. Agent Engineering: The “trio” of a tool registry center, sandbox execution environments, and evaluation guardrails is the prerequisite for the safe deployment of parallel orchestration. Tool calls from any parallel branch should be executed in a controlled environment and fully logged.

8. Conclusion

The collaborative upgrade of vision and tool capabilities is essentially a synergy across three levels: at the architectural level, replacing late-stage adaptation with a unified semantic space; at the training level, ensuring the stability of joint optimization through a decoupled progressive pipeline; and at the execution level, replacing sequential calls with dynamic parallel orchestration. For developers, understanding this technical logic provides value beyond mere model selection. Whether evaluating candidate models, planning proprietary development, or building an agent engineering system, “native fusion + parallel orchestration” is becoming an unavoidable baseline design. It is recommended to build a small-scale evaluation set based on real business tasks to first validate the benefits of modal synergy and parallel execution in your specific scenario, and then decide on the scale of investment, ensuring that technological upgrades genuinely translate into business efficiency.