Native Multimodal Agent Architecture and Dynamic Parallel Orchestration in Practice

    This article examines a native multimodal agent that jointly trains text and vision from the ground up instead of bolting a vision encoder onto a pretrained language model. It walks through the three-stage architecture, the unified vision encoder built on patch packing and spatio-temporal tubes, the decoupled training pipeline that stabilizes large-scale training, and a dynamic parallel orchestration scheme that turns sequential tool-calling loops into a task graph. It also covers how to evaluate such models across five dimensions and the engineering guardrails needed before deployment. Readers will learn why the language-first, vision-later approach hits a ceiling, and which linked metrics—visual input fidelity, training and fine-tuning controllability, and agent execution latency and cost—to validate when choosing a model.

    1. Starting Point: Why the “Language First, Vision Later” Approach Hits a Ceiling

    For the past two or three years, mainstream multimodal models have mostly taken the same engineering shortcut: pretrain a pure-text LLM to convergence, then freeze or lightly unfreeze the backbone, attach a vision encoder, and use a small amount of image-text alignment data to “weld” the two worlds together. This route is good enough for tasks like image captioning and document QA. But when the task escalates from “understanding a single image” to “operating continuously on screen and maintaining judgment across multiple tool-calling rounds,” its structural flaws surface in concentrated form.

    They can be grouped into three categories:

    1. Modality conflict and capability regression. Gradients from the vision adaptation stage keep washing over the language-side parameters, so the model’s language reasoning and long-context ability regress noticeably after multimodal fine-tuning. Conversely, if the backbone is strictly frozen, visual information struggles to reach the deep reasoning layers, creating a stalemate of “seeing clearly but not reasoning deeply.”
    2. Information loss from preprocessing. To fit a fixed input size, high-resolution screenshots, long documents, and charts must be cropped or scaled down. Small fonts, dense tables, and controls at the edges of an interface—the signals most critical for agent tasks—are the first to be lost.
    3. The separate-branch cost of video. If images and videos use two separate encoding pipelines, the understanding learned on images cannot transfer smoothly to video, and branch maintenance, weight storage, and training data mixing all double.

    A recent open-source multimodal agent model offers a different answer in its technical report. Instead of late-stage adaptation, it places text and vision under the same optimization objective from the very bottom of the architecture and jointly trains them throughout, and it reconstructs agent execution from “single-agent sequential execution” into “multi-agent dynamic parallel orchestration.” Its reported result: for complex tasks, end-to-end execution latency can drop to as low as about a quarter (roughly 4.5x speedup), while task completion quality improves at the same time. The engineering logic behind this set of conclusions is the more worthwhile part to break down.

    2. Overall Architecture: A Three-Stage Pipeline and Bidirectional Enhancement Between the Two Modalities

    The overall shape of the approach can be summarized as three core components plus one data flow:

    1. Image/video ──▶ Native 3D vision encoder ──▶ MLP projection layer ──┐
    2. ├──▶ MoE language backbone ──▶ Text/tool calls
    3. Text ─────────▶ Tokenization & embedding ──────────────────────────┘

    The pipeline follows a three-stage flow of “visual feature extraction → cross-modal projection → unified backbone modeling.” The visual input is encoded into a sequence of vectors isomorphic to language tokens, aligned through a projection layer to the language model’s embedding space, then concatenated with text tokens and fed into an MoE (mixture-of-experts) backbone for end-to-end modeling. Vision and language therefore share the same semantic space and the same attention mechanism, rather than being weakly coupled through a glue layer.

    “Bidirectional enhancement” is the key design orientation. Joint optimization means that backpropagation from vision tasks participates in shaping the backbone, so the model draws on the vision channel more fully in reasoning that requires fine-grained visual localization. At the same time, the language side’s reasoning and tool-calling ability inversely constrain how the vision encoder forms representations, pushing it to extract features that are “useful for decision-making” rather than optimizing reconstruction or classification metrics alone. This is also the fundamental difference in the capability curve between native fusion and late-stage adaptation—the former is co-evolution, the latter is division of labor.

    There is also an engineering choice that cannot be ignored: the vision encoder’s parameters do not enter the daily inference hot-path optimization of the backbone; instead they are trained decoupled from the backbone (see Section 4). This is not a compromise in the architecture but a trade to buy stability under ultra-large-scale training.

    3. Unified Vision Encoder: Native Resolution and Unified Spatio-Temporal Modeling

    The design goal of the vision encoder is “one set of weights, one pipeline for both images and videos,” resting on two core properties.

    3.1 Native-resolution input based on patch packing

    The traditional approach resizes images to a fixed side length, or splits them into sub-images on a fixed grid and encodes each one. The former loses information; the latter breaks global context and markedly lengthens the sequence. The approach here is patch packing:

    • Split the image into patches and apply 2D positional encoding;
    • Flatten and concatenate the patches into a 1D sequence, with a block-diagonal attention mask ensuring that patches from different images cannot see each other;
    • Pack the sequence into groups dynamically according to a memory budget, avoiding padding waste.

    Illustrative logic:

    1. def pack_patches(images, position_ids, max_seq_len):
    2. """Variable-length patch packing: no resize, no padding waste"""
    3. packs, cur, cur_len, cur_mask_blocks = [], [], 0, []
    4. for img, pos in zip(images, position_ids):
    5. n = img.num_patches
    6. if cur_len + n > max_seq_len and cur:
    7. packs.append((flatten(cur), build_block_mask(cur_mask_blocks)))
    8. cur, cur_len, cur_mask_blocks = [], 0, []
    9. cur.append(embed(img) + encode_2d_positions(pos))
    10. cur_mask_blocks.append(n)
    11. cur_len += n
    12. if cur:
    13. packs.append((flatten(cur), build_block_mask(cur_mask_blocks)))
    14. return packs

    The benefit here is more than just “not losing information.” Preserving any aspect ratio means screenshots need not be stretched or distorted, and the positional relationships of UI controls are maintained. Meanwhile, dynamic grouped packing keeps GPU compute utilization high under variable-length input, avoiding the “small images waste resources, large images get truncated” dilemma of fixed-resolution schemes.

    3.2 Unified spatio-temporal modeling with fully shared image-video weights

    The video-side design extends the patch-packing idea along the time dimension: treat every 4 consecutive frames as one spatio-temporal tube, jointly flatten and pack the 2D patches of multiple frames, and complete spatial-temporal attention modeling in a single pass within a fully shared encoder.

    The value of this design is threefold:

    • Zero extra branches. There is no need to design a separate encoder or adaptation module for video; weights, inference code, and deployment artifacts are all reused, and operational complexity does not grow with the modality.
    • Maximized knowledge transfer. The gains from image understanding transfer directly to video, so a training mix heavy on images is not wasted.
    • Temporal and spatial modeling decoupled. The 3D positional encoding explicitly distinguishes temporal from spatial components, letting the model decide for itself when cross-frame comparison is needed (e.g., detecting interface changes) and when single-frame understanding suffices (e.g., reading static text).

    The cost is equally clear: spatio-temporal tube packing makes the token count grow linearly with time, and video understanding is fundamentally a context-length problem. In practice, this usually needs to be controlled with token compression, keyframe sampling, or hierarchical pooling to keep the sequence size in check.

    4. Decoupled Encoder Training: The Key to Large-Scale Training Stability

    Joint optimization sounds elegant, but on a thousand-GPU cluster it becomes a stability problem: the vision encoder and the language backbone differ enormously in loss scale, learning-rate needs, and convergence speed. Forcing them to train in the same batch at the same learning rate commonly results in the vision branch oscillating or catastrophic forgetting in the backbone. This approach’s solution is training-pipeline decoupling, with typical stages as follows:

    Stage Training scope Goal Key control points
    Stage 1 Vision encoder (optionally with projection layer) pretrained separately Obtain strong general vision representations Mixed image-text contrastive + fine-grained localization tasks
    Stage 2 Projection layer dominant, backbone lightly involved Align the embedding space Freezing strategy, modality mixing ratio
    Stage 3 Encoder + projection + backbone trained jointly Bidirectional enhancement Layered learning rates, gradient clipping, modality curriculum
    Stage 4 Long-context and video extension Temporal and long-document ability Frame curriculum, progressive sequence-length growth

    Two engineering details here are easily overlooked. First, modality-mix scheduling: if pure-text data makes up too high a share, the model degrades into “image captioning”; if pure image-text data makes up too high a share, language and coding ability decline. In practice a curriculum-style mix is used—leaning toward text and image-text alignment early, then gradually raising the proportion of interleaved documents, UI screenshots, and video—letting the model grow visual reasoning while retaining its language ability. Second, layered learning rates: the vision encoder and the backbone use different peak learning rates and different warmup lengths, and the projection layer is typically the first to unfreeze and the first to decay. This is the most direct and effective means of suppressing modality conflict.

    5. From Sequential to Parallel: Dynamic Orchestration of Agent Clusters

    Improving multimodal capability only solves “understanding what it sees”; the real bottleneck for an agent is “finishing the work.” Single-agent sequential execution follows a serial loop of “observe—think—call tool—observe again,” and a task spanning apps and files may require anywhere from a dozen to several dozen rounds of tool calls. Latency accumulates linearly with the number of rounds, and every round reloads the full context, so the compute cost of long contexts is paid over and over.

    The idea of dynamic parallelism is to turn the task from a single chain into a graph:

    1. Task: Diagnose the login failure and fix it
    2. │
    3. ┌─────┼─────────┬──────────┐
    4. ▼ ▼ ▼ ▼
    5. Logs Config Dependency UI ← parallel sub-agents
    6. review version reproduce
    7. └─────┼─────────┴──────────┘
    8. ▼
    9. Evidence aggregation & conflict resolution
    10. ▼
    11. Code change → regression verification (parallelizable again)

    The orchestration framework has to solve four things:

    1. Task decomposition and dependency inference. Break a natural-language goal into a subtask DAG with explicit dependency edges; dependency-free nodes enter the same batch to run in parallel.
    2. Dynamic scheduling. Sub-agent roles differentiate (retrieval-type, coding-type, verification-type, vision-observation-type) and are assigned dynamically according to current available concurrency and budget, so long-tail slow tasks do not block the aggregation of completed branches.
    3. Result aggregation and conflict resolution. Parallel branches may produce contradictory conclusions, so a dedicated aggregator role is needed to weight evidence and check consistency, rather than simply concatenating outputs.
    4. Failure isolation and retry. A branch failure should not pollute global state; tool calls should require idempotency keys to avoid duplicate side effects.

    Illustrative scheduling logic:

    1. def run_swarm(goal, budget):
    2. dag = planner.decompose(goal) # subtasks + dependency edges
    3. results = {}
    4. while pending(dag):
    5. batch = [n for n in dag if ready(n, results) and budget.ok(n)]
    6. outs = parallel_map(assign_role(n) for n in concurrent_limit(batch))
    7. for node, out in zip(batch, outs):
    8. results[node] = verifier.check(node, out) # branch-level verification
    9. return aggregator.resolve_conflicts(goal, results)

    We have to view its costs objectively: parallelism amplifies token consumption, insufficient shared context between sub-agents leads to duplicated work, and conflict resolution itself is an inference cost. So the applicable boundary of this framework is scenarios where tasks are decomposable, branches are loosely coupled, and there are naturally parallel evidence sources (codebase troubleshooting, multi-source document comparison, multi-page UI verification). For narrowly sequential tasks with strong order dependencies, a serial single agent is actually cheaper.

    6. How to Evaluate These Models: Five Dimensions and Verification Methods

    A vendor’s published benchmark scores can only serve as an initial filter. Before deployment, it is advisable to build your own evaluation set covering five dimensions:

    • Code generation and repo-level understanding: cross-file changes and judging dependency impact;
    • Visual understanding: OCR and localization accuracy on high-resolution screenshots, dense charts, and small-font documents;
    • Math and logical reasoning: stability of multi-step derivations;
    • Agent execution: success rate and number of rounds across multiple tool calls;
    • Computer use: correctness of screen-element localization and action sequences, and failure-recovery ability.

    The evaluation should record four metrics at once—success rate, end-to-end latency, token cost, and failure-recovery ability—and distinguish “first-round pass rate” from “pass rate after allowing retries,” since in agent scenarios the gap between the two is often larger.

    7. Deployment Path and Engineering Guardrails

    The greatest value of open weights is controllability: data stays within your domain, you can deploy privately, and you can fine-tune for your domain. Common general cloud engineering practices that pair with this typically include:

    • Inference serving hosted on a container platform, auto-scaling by concurrency, with the vision encoder and language backbone deployable as separate pools;
    • Multimodal inputs (screenshots, video, documents) placed in object storage, with the service side passing only references and summaries to avoid large objects crossing the gateway;
    • Batch offline evaluation and data feedback routed through a message queue to smooth peaks, with task status and trajectories written to a logging service;
    • Latency, failure rate, and token cost wired into monitoring and alerting, with thresholds set on P95/P99 rather than averages.

    Four guardrails cannot be omitted: screen- and computer-use capabilities must enforce least privilege and an action allowlist, with high-risk actions requiring human confirmation; tool calls in parallel orchestration need idempotency + timeout + circuit breaking; visual input must be treated as untrusted content, watching for prompt injection hidden in screenshots or documents; and long-context reasoning needs a token budget cap to prevent costs from spiraling.

    8. Summary

    The technical mainline of a native multimodal agent is actually quite clear: at the architecture layer, fuse modalities natively rather than stitching them together late; at the data layer, use patch packing and spatio-temporal tube packing to preserve information fidelity and modality reuse; at the training layer, trade a decoupled pipeline for stability; and at the execution layer, use dynamically parallel cluster orchestration to compress serial latency into an acceptable range. The takeaway for developers: when selecting a model, do not look only at single-benchmark scores, but validate “visual input fidelity, training and fine-tuning controllability, and agent execution latency and cost” as a set of linked metrics; build your evaluation set, observability, and permission guardrails first, then talk about realizing the benefits of a model upgrade.