Multimodal Agent Models: Native Vision Encoding and Parallel Orchestration in Practice

    Multimodal agent models are moving beyond the traditional "language first, vision later" approach, which struggles with imbalanced capabilities, locked resolution, and serial-execution latency. This article walks through an alternative design that jointly optimizes vision and language end to end — a native-resolution vision encoder with patch packing and shared image/video weights, an MoE language backbone with a lightweight projection layer, a decoupled four-stage training pipeline, and DAG-based parallel orchestration with verification and conflict control. It also covers evaluation dimensions, deployment and cost-control practices, and common pitfalls. Developers gain two transferable methods: decoupled multimodal training and parallelism-safe task orchestration.

    1. Where the problem starts: why “language first, vision later” hits a wall

    For the past few years, the mainstream approach to multimodal models has been two-stage: first train a pure text LLM, then attach a pretrained vision encoder through a projection layer and fine-tune it with a small amount of image-text alignment data. This path is cost-effective engineering-wise, but it carries three structural flaws.

    First, imbalanced capability allocation. Late-stage adaptation means the vision side only gets a limited fine-tuning budget. The model tends to stay strong on language tasks but performs mediocrely on fine-grained visual understanding, chart reading, and UI element localization. Conversely, if you drastically adjust parameters to shore up vision, you damage the original language and reasoning abilities — a classic case of robbing Peter to pay Paul.

    Second, resolution is artificially locked. To keep the number of visual tokens controllable, adaptation-based approaches almost always crop or scale inputs to a fixed size. For content with extreme aspect ratios — long screenshots, document pages, code screenshots, UI interfaces — the preprocessing itself causes irreversible information loss.

    Third, sequential execution creates a latency wall. A real agent task isn’t a single Q&A round, but a multi-turn loop of “observe → plan → call tools → check results → re-plan.” When a single agent proceeds serially, total latency equals the sum of each step’s latency, and a task with dozens of tool calls easily slips into minute-level waiting.

    The evolution direction of general multimodal agent models is precisely to tear down both walls at once: end-to-end joint optimization of vision and language at the architecture level, and turning serial execution into parallel orchestration at the scheduling level.

    2. The architecture backbone: vision feature extraction → cross-modal projection → unified backbone modeling

    The overall structure of this kind of model can be abstracted into three components, with a clear pipeline:

    1. Image/video input
    2. ↓
    3. Native-resolution vision encoder (extracts unified visual features)
    4. ↓
    5. MLP projection layer (aligns to the language model's embedding space)
    6. ↓
    7. Sparse expert (MoE) language backbone ←——— text tokens
    8. ↓
    9. Unified end-to-end modeling and output

    Three key design points stand out.

    1) Keep the projection layer lightweight. Use only an MLP for dimension alignment, rather than introducing large cross-attention modules. The thinner the projection layer, the more easily gradients flow back from the language backbone to the vision side, so the visual representation is genuinely shaped by task objectives instead of staying at the level of “generic image features.”

    2) Let the sparse backbone carry the multimodal load. A language backbone built on a Mixture-of-Experts (MoE) structure can expand parameter capacity without significantly increasing per-token compute. This is especially critical for multimodal workloads: the number of visual tokens is usually far larger than text, and sparse activation effectively controls inference cost.

    3) Let vision and text share the same semantic space. After projection, visual features are concatenated with text tokens into the same sequence and processed by the same attention and expert networks. As a result, visual information isn’t just “additional context” but participates in deep reasoning just like language, and cross-modal alignment is markedly better than post-hoc concatenation.

    Native resolution and patch packing

    The first core feature of the vision encoder is abandoning fixed resolution. Borrowing from the patch-packing idea: split the image into patches, preserve 2D coordinate information through positional encoding, then flatten and concatenate them into a 1D sequence sent directly into the Transformer. Pseudocode:

    1. def pack_image(image, patch_size):
    2. # No resize; preserve the original aspect ratio
    3. patches, coords = split_to_patches(image, patch_size)
    4. seq = flatten(patches) # [N_patches, D]
    5. pos = encode_2d_rope(coords) # Preserve 2D spatial positions
    6. seq = apply_rope(seq, pos)
    7. return seq, len(seq)

    This brings two direct benefits: any aspect ratio can be fed natively, so details in documents and UI screenshots are no longer blurred away by scaling; and sequence length is determined by true resolution, so it can be dynamically adjusted by budget rather than being cut by a fixed tier.

    Images and videos fully share weights

    The second core feature extends the above idea to the time dimension: treat several consecutive frames (for example, 4 frames) as a spatiotemporal volume, jointly flatten and pack the 2D patches of multiple frames, and use the same encoder to perform spatial-temporal attention modeling.

    There is no separate branch or extra module designed for video here, and the significance lies in knowledge transfer: image data is usually far larger than annotated video data, so shared weights mean massive image-understanding capability can transfer directly to video tasks without cold-starting the temporal dimension. The engineering cost is that sequence length grows faster, so it must be paired with sampling strategies and a token budget cap.

    3. The training pipeline: decoupled encoder training

    The engineering challenge of joint training is stability. If the vision encoder and the language backbone are fully updated at the same time, training can easily diverge or produce a “seesaw” effect between modalities. A more mainstream approach is to split training into stages:

    1. Pretrain the vision encoder independently: train it thoroughly on general vision data to obtain strong general visual representations, with the language backbone frozen.
    2. Alignment stage: train only the projection layer so that visual features fall into the language embedding space; both the backbone and the encoder can stay frozen, making this the lowest-cost stage.
    3. Joint fine-tuning: unfreeze the encoder and backbone, and perform end-to-end joint optimization on mixed corpora of images, text, video, documents, UI, and code screenshots, using a smaller learning rate and gradient clipping.
    4. Post-training alignment: apply preference- or RL-style alignment on agent trajectory data (tool-call sequences, interface operation traces, executable result verification).

    The value of this decoupled process is that each stage has a single objective, is monitorable, and can be rolled back; when something goes wrong, you can quickly pinpoint whether it is encoder representation degradation or insufficient alignment, rather than staring at a single tangled loss curve with no idea what to do.

    4. Parallel orchestration: from single-agent serial to agent clusters

    Model capability improvements only solve “getting each step right,” while the end-to-end experience also depends on “how many steps run at once.” The traditional single agent is strictly serial: read the page → think one step → click once → read the page again. Yet many tasks have a naturally parallel structure internally.

    The core of a dynamic parallel orchestration framework is to model the task as a directed acyclic graph (DAG) rather than a linear list:

    • Task decomposition: break the goal into subtasks, explicitly marking the dependency relationships among them;
    • Dependency analysis: subtasks with no dependencies (for example, searching multiple information sources in parallel or validating multiple candidate solutions at once) are dispatched to different work units;
    • Dynamic scheduling: re-plan at runtime based on returned results, retry failed branches, prune successful branches, and dynamically insert newly discovered subtasks;
    • Result aggregation: a summarizing role compares, votes on, or resolves conflicts among multiple results and outputs a consistent conclusion.
    1. # Pseudocode: a dependency-graph-based parallel orchestration skeleton
    2. def run_swarm(goal, executor):
    3. graph = decompose(goal) # Generate the subtask DAG
    4. while not graph.done():
    5. ready = [n for n in graph.ready_nodes()] # Nodes with in-degree 0
    6. results = parallel_map(executor.run, ready) # Execute in parallel
    7. for node, res in zip(ready, results):
    8. graph.apply(node, res) # Fill in results, release successor nodes
    9. graph.replan() # Dynamically adjust based on intermediate results
    10. return aggregate(graph.collect())

    The key constraint here is that correctness cannot be sacrificed for parallelism. Parallelism means different branches may observe inconsistent views of the world (for example, multiple branches operating the same interface at once), so in practice you typically need to: assign isolated execution environments to branches (separate containers or sessions), serialize or lock writes to shared resources, and introduce verification based on executable results at the aggregation stage (does it compile, do tests pass, does the page state reach the goal), rather than relying purely on the model’s self-report.

    5. Capability evaluation: which dimensions to look at and how to read the scores

    For evaluating multimodal agents, a single leaderboard is no longer enough. It is advisable to cross-validate across capability dimensions:

    Dimension What to look at
    Code generation Unit-test pass rate, cross-file change consistency, maintainability
    Visual understanding Fine-grained Q&A on documents, tables, and charts; OCR robustness; high-resolution screenshots
    Math & logical reasoning Multi-step derivation correctness, long-context consistency
    Agent execution Task completion rate, average number of steps, tool-call error rate
    Computer / interface control Element localization accuracy, state recovery, exception handling

    A pragmatic reminder: there is a transfer gap between being “industry-leading” on public benchmarks and performing well in your own business scenarios. Before deployment, run a small-scale evaluation with your own task set, focusing on task success rate, end-to-end P95 latency, and per-task cost, rather than only general leaderboards.

    6. Engineering deployment: deployment forms and cost control

    To host this kind of model in your own environment, proceed in the following order.

    1. Serve inference as a service: host inference services on a container platform with elastic scaling to handle the long-tail load of agent tasks.
    2. Multimodal data management: store large objects such as images, videos, and task trajectories in object storage, and have inference services pass them by reference rather than inline, reducing request size.
    3. Long-task orchestration and asynchrony: decouple multi-step agent tasks into asynchronous jobs through message queues to avoid holding synchronous connections for a long time; the frontend returns intermediate progress via streaming.
    4. Observability: combine logging services with monitoring and alerting to capture each step’s input token count, output token count, tool-call latency, and failure reasons. Visual tokens are often the main cost driver, so they must be instrumented separately.
    5. Cost-control measures: set token budget caps for visual inputs (lower the frame sampling rate, crop regions of interest on demand); route short requests to smaller model tiers; cache results for repeated calls; enable quantized inference to reduce memory footprint.

    On Baidu AI Cloud, the container platform, object storage, message queue, logging service, and monitoring/alerting mentioned above are all general-purpose capabilities that can be combined to build a complete agent inference and observability pipeline, without binding to any proprietary interface.

    7. Common pitfalls

    • Blindly increasing input resolution. The growth in visual tokens from higher resolution is usually superlinear; beyond what the task actually needs, the benefits decay quickly while cost rises linearly.
    • Overlooking the equivalence of visual preprocessing. If the patch splitting and positional encoding used at training time differ from those used at inference time, they silently degrade results in ways that are hard to attribute directly from metrics.
    • Treating parallelism as the default. Parallel orchestration only pays off when subtasks are genuinely independent; forcing parallelism onto a strongly dependent serial chain only adds scheduling overhead and result conflicts.
    • Looking only at task success rate. A success rate without latency and cost constraints often cannot be delivered in a business context.

    8. Summary

    The progress of general multimodal agent models is essentially about removing two artificial constraints: on the architecture side, removing the “language first, vision later” stage split so that the two modalities are optimized end to end in the same semantic space; on the scheduling side, removing the “only think one step at a time” serial limit, using dependency graphs and dynamic parallelism to raise task throughput. For developers, the real payoff is not a number on some leaderboard, but mastering two transferable design methods — a decoupled multimodal training pipeline, and parallel task orchestration with verification and conflict control. Applying these two in your own scenarios is more valuable in the long run than chasing any single model version.