Native Multimodal Agent Architecture and Parallel Agent Swarm Orchestration: A Technical Deep Dive

    As AI competition shifts from single-turn question answering to end-to-end task completion, two technical threads have become central: native multimodal architectures that fuse vision and language from the start of training, and multi-agent parallel orchestration that breaks the latency wall of serial single-agent execution. This article walks through the structural limits of bolted-on multimodal designs, the three-component architecture of an open-weight multimodal agent model (native 3D vision encoder, MLP projector, MoE language backbone), decoupled encoder training stages, patch-packing-based native-resolution encoding with fully shared image/video weights, why single agents hit a latency wall on DAG-shaped tasks, and how dynamic parallel agent swarms work in practice — including dependency-aware scheduling, context isolation, result aggregation, cost and governance concerns, post-training data mix, evaluation dimensions, and deployment patterns.

    1. Background: Two Technical Threads Behind Multimodal Agents

    Recent developments show that the frontier of general AI is moving away from “single-turn question answering” toward “completing real tasks end to end” — what can be called general agentic intelligence. Models in this category must understand visual inputs such as screenshots, charts, documents, and video, while also planning autonomously, calling tools, and operating software. Two mutually reinforcing technical threads have emerged around this goal:

    • Native multimodal architectures: fuse vision and language early in training to overcome the capability gaps of bolted-on multimodal designs.
    • Multi-agent parallel orchestration: let multiple agents decompose and execute tasks cooperatively, addressing the latency bottleneck of serial single-agent execution.

    The sections below first examine the limitations of the conventional paradigm, then analyze both core mechanisms along with their supporting engineering practices.

    2. From “Language-First, Vision as Adapter” to Native Unified Modeling

    2.1 Three Structural Limits of Bolted-On Multimodal

    Conventional vision-adaptation approaches usually proceed in two steps:

    1. Train the language backbone on large volumes of pure text.
    2. Attach a vision encoder and a projection layer, then run alignment training on image-text data — commonly with the language backbone frozen.

    This route is engineering-simple and reuses mature language models, but it carries three structural shortcomings:

    • Language and vision trade off against each other: alignment data is far smaller in both volume and diversity than pure-text data. With a frozen backbone, the language side cannot benefit from visual information to improve spatial and physical common sense; with an unfrozen backbone trained jointly, language capability tends to degrade — catastrophic forgetting.
    • Modality conflict: visual tokens and text tokens differ substantially in embedding distribution, and deep visual features only enter the language space after passing through multiple adaptation layers. The result is that language priors “overpower” visual evidence, showing up as models that can see but cannot read accurately dense information such as charts, receipts, and UI screenshots.
    • Images and video are handled separately: the two modalities typically use different preprocessing pipelines and separate branch structures, so image capability transfers poorly to video understanding.

    2.2 Two Requirements of Native Joint Optimization

    The core of native multimodal design is to achieve fusion at the architecture level rather than in post-processing. Text and vision are jointly optimized under the same objective from early in training, so gradient signals flow in both directions:

    • Large-scale text knowledge feeds back into visual semantic understanding;
    • Visual information supplements physical-world and interface state, which in turn strengthens text reasoning and code generation.

    Two design requirements come with this in practice:

    1. Visual token budget control: once an image passes through the vision encoder it usually expands into hundreds to thousands of tokens; stacked on top of multi-turn dialogue and tool outputs, the context grows quickly.
    2. Attention modeling efficiency: both the inference cost of long contexts and detail fidelity at arbitrary resolutions must be handled.

    3. Overall Architecture: Three Components and Decoupled Training

    An open-weight multimodal agent foundation model — one whose stated core goal is pushing forward the boundary of general agentic intelligence — is composed of three components:

    1. Vision encoder: a native 3D visual encoding structure built for unified modeling of images and video (a native 3D vision encoder), supporting input at arbitrary resolutions.
    2. Cross-modal projector: an MLP structure that maps visual features into the language model’s embedding space.
    3. Language backbone: a mixture-of-experts (MoE) sparse language model serving as the unified modeling core.

    The overall pipeline is “visual feature extraction → cross-modal projection → unified modeling in the language backbone.” Images and video are first encoded by the vision encoder, then aligned by the projector, and finally concatenated with text tokens and fed into the language backbone, where everything is modeled end to end in a single semantic space.

    3.1 Why the Projection Layer Uses an MLP

    A single linear projection is sufficient for dimensional alignment, but nonlinear transformation is more effective at adapting the feature distributions of different modalities to each other. A common parameter-efficient approach is “concatenate first, then compress”:

    1. Concatenate 2×2 spatially adjacent visual tokens along the channel dimension.
    2. Project each group into a single language token using a two-layer MLP with gated activation (for example SwiGLU).

    Compared with per-token projection, this compresses the number of visual tokens while preserving detail, directly reducing the context overhead of long-document and video scenarios.

    3.2 Why the Language Backbone Uses an MoE Sparse Architecture

    The MoE routing layer selects the Top-k experts (commonly 2) to participate in computing each token. Total parameter count can scale to the trillion range, while the parameters activated per token remain a small fraction of that.

    This matters especially for multimodal workloads: visual tokens usually far outnumber text tokens, and the prefill cost of visual tokens is high. Sparse activation expands model capacity while keeping compute in check. The tradeoff is heavy memory consumption and a tendency toward load imbalance across the router, which requires expert parallelism plus an auxiliary load-balancing loss.

    3.3 Decoupled Encoder Training Pipeline

    The model is also paired with an engineering-oriented decoupled encoder training pipeline, summarized as “strengthen perception separately, then align, then fuse agent capabilities”:

    1. Train the vision encoder independently: use objectives and parallelism strategies decoupled from the language backbone to improve visual representation quality on its own, avoiding interference from language gradients during the early formation of visual features.
    2. Modality alignment: jointly train the projector to map visual semantics into the language space.
    3. Joint optimization: unified modeling of the language backbone and the vision side, reinforcing each other.
    4. Agent post-training: strengthen tool calling and long-horizon planning through supervised fine-tuning, preference optimization, or reinforcement learning with execution feedback.

    From a distributed-training standpoint, decoupling brings two further benefits:

    • Each stage can adopt a parallelism strategy better matched to its workload: data parallelism suits encoder training, tensor parallelism suits the dense phases of the large model, and expert parallelism suits the MoE backbone.
    • Checkpoints of the vision encoder and the language backbone are independent, so failures can be rolled back separately, lowering the cost of interruptions in large-scale training.

    Illustrative flow (not a real API):

    1. # Decoupled training pipeline (illustrative)
    2. stage1 = train_encoder(data=image_text_pairs, objective=encoder_pretrain_loss)
    3. stage2 = train_projector(
    4. encoder=stage1.encoder, lm=base_lm, data=image_text_pairs,
    5. trainable={"projector"}, # Train the projector only
    6. )
    7. stage3 = joint_train(
    8. encoder=stage2.encoder, projector=stage2.projector, lm=base_lm,
    9. data=mix(image_text_pairs, video_instructions, ocr_charts),
    10. trainable={"encoder", "projector", "lm"}, # Joint text-vision optimization
    11. replay_ratio=0.3, # Replay pure text to curb language forgetting
    12. )
    13. stage4 = post_train(model=stage3, data=agent_trajectories) # SFT + RL

    4. The Vision Side: Native Resolution and Unified Image-Video Encoding

    4.1 Native-Resolution Input Based on Patch Packing

    This encoding structure draws on patch-packing approaches from prior research (methods in the NaViT family, for example). Conventional preprocessing loses information noticeably:

    • Images are first resized to a square, then cropped or padded.
    • All images in a batch are normalized to a fixed resolution.

    For receipt recognition, dense documents, and GUI screenshots, this leads to blurred small text, distorted layout relationships, and broken aspect ratios.

    The new approach works as follows:

    1. Split the image using a uniform patch size (commonly 14×14 pixels).
    2. Flatten each patch into a 1D token sequence, paired with a resolution-agnostic position encoding.
    3. Concatenate the sequences of differently sized images in the batch head-to-tail into one long sequence for the encoder.

    Visual token budget (illustrative):

    Input resolution Patch grid Visual tokens
    224×224 16×16 256
    448×448 32×32 1024
    896×1792 (long image) 64×128 8192

    Three points deserve particular attention in engineering:

    • Attention isolation: tokens from different images use block-diagonal masks to prevent cross-sample attention contamination, while attention kernels that handle variable-length sequences are used within the sequence itself.
    • Packing throughput: variable-length packing improves hardware utilization, but when combined with variable-length attention kernels, load-balancing scheduling directly affects training throughput.
    • Aspect ratio first: image resizing should be constrained by the token budget while preserving the original aspect ratio, avoiding the geometric distortion of forced square inputs.

    Illustrative code (not a real API):

    1. # Variable-length image packing (illustrative)
    2. def build_visual_tokens(images, patch=14, max_tokens=4096):
    3. sequences, seg_ids, grid_shapes = [], [], []
    4. for idx, img in enumerate(images):
    5. h, w = img.height, img.width
    6. # Resize proportionally under the token budget, preserving the original aspect ratio
    7. scale = min(1.0, math.sqrt(max_tokens * patch * patch / (h * w)))
    8. img = img.resize((max(patch, round(w * scale)),
    9. max(patch, round(h * scale))))
    10. gh, gw = img.height // patch, img.width // patch
    11. # (C, gh*patch, gw*patch) -> (C, gh, patch, gw, patch) -> (gh*gw, patch*patch*C)
    12. tokens = img.unfold(1, patch, patch).permute(0, 1, 3, 2, 4)\
    13. .flatten(0, 2).reshape(gh * gw, -1)
    14. sequences.append(tokens)
    15. seg_ids.extend([idx] * (gh * gw))
    16. grid_shapes.append((gh, gw))
    17. return torch.cat(sequences), seg_ids, grid_shapes
    18. # encoder(packed_tokens, seg_ids) builds a block-diagonal attention mask from seg_ids

    4.2 Full Weight Sharing Between Images and Video

    This design extends the multi-image packing idea to the time dimension, producing an encoding mechanism that models images and video together:

    • Every 4 consecutive video frames are treated as one spatiotemporal volume (a 4D patch).
    • The 2D patches of multiple frames are jointly flattened and packed, then handled by a fully shared vision encoder that performs joint attention over spatial and temporal dimensions.
    • No separate video branch or dedicated module is introduced, so the extra encoding parameters for video are zero and image understanding capability transfers directly to video.

    The difference from two common alternatives:

    Dimension Separate video encoder Per-frame encoding + temporal modeling Fully shared image/video weights
    Extra parameters Substantial Temporal module None
    Cross-frame attention Modeled within the framework Only inside the temporal module Joint spatiotemporal
    Knowledge transfer Image, text, and video learned separately Constrained by the framework structure Image knowledge transfers directly
    Engineering complexity High Medium Low

    Real-world results depend on two details:

    1. Frame organization: interleaving patches across frames lets each patch’s local window cover the same spatial position at adjacent time steps, which helps model motion, state changes, and UI operation trajectories; arranging frames contiguously is better for reading full pages of text.
    2. Frame sampling strategy: uniform sampling, keyframe extraction, or adaptive sampling can be used. Sampling too sparsely loses transient actions, while sampling too densely multiplies the video token count — tuning depends on the task type.

    5. The Execution Bottleneck: The “Latency Wall” of Serial Single-Agent Execution

    Multimodal understanding solves “can it read this”; completing real tasks end to end also requires “can it do this.” Take “organize desktop files, aggregate data, and generate a report” as an example. A single-agent flow might be:

    1. Screenshot the desktop and identify icons → open folders → read files one by one → call the spreadsheet tool to aggregate → write and save the report

    Any step can fail and trigger a retry. From a systems perspective, end-to-end latency is approximately:

    1. T_total ≈ Σ (T_infer_round_i + T_tool_call_i + T_retry_i)

    Latency comes from four kinds of serial waiting:

    • Serial inference rounds: in multi-round “reason–act” loops, each round must generate first and then call.
    • Tool waiting: browser navigation, script execution, and builds or tests take anywhere from seconds to minutes.
    • Context growth: history messages and tool outputs keep accumulating, raising prefill and decode costs every round.
    • Failure retries: timeouts, format mismatches, and tool errors all require returning to the reasoning stage to correct.

    From a directed acyclic graph (DAG) perspective, end-to-end latency depends on the length of the critical path. Task graphs often contain many mutually independent branches, for example:

    • Scraping information from multiple pages separately;
    • Implementing and testing several candidate approaches separately;
    • Analyzing the same piece of code from different angles.

    These branches could run concurrently, but serial execution simply adds up the time of the parallelizable parts. This has two consequences:

    • Latency grows linearly with task size: the more subtasks, the longer the wait.
    • A single point of failure blocks everything: when one step fails, the whole task rolls back and completed intermediate results are wasted.

    6. Agent Swarms: How Dynamic Parallel Orchestration Works

    6.1 Core Idea

    Agent-swarm-style dynamic parallel orchestration organizes multi-agent collaboration in a parallelism-first way, supported by a unified model:

    • A lead agent responsible for planning and aggregation dynamically splits complex tasks into subtasks that can run concurrently;
    • Multiple worker agents with different specializations execute in parallel;
    • Structured aggregation produces the final answer or sequence of operations.

    This differs from two common patterns:

    • Static parallelism in workflow engines: the parallel structure is fixed in advance, and the orchestration layer itself needs no “intelligence.” Here the model decides decomposition and parallelization dynamically, so open-ended tasks are supported.
    • Multiple samples of the same question with voting: repeated sampling parallelizes the same task to improve single-answer accuracy. Swarm parallelism runs different subtasks to shorten end-to-end latency.

    Public information indicates this approach can reduce execution latency for complex tasks by roughly 4.5x at best while also improving task performance. The reasons are:

    • Worker agents work with more focused contexts, reducing information interference in the lead agent’s context;
    • Multiple parallel paths provide exploration and cross-checking, lowering the probability of error on a single path.

    6.2 Key Mechanisms

    1. Task decomposition and dependency annotation: the lead agent annotates each subtask with required context, available tools, expected outputs, and dependencies, forming a DAG.
    2. Dynamic parallelism decisions: concurrency is adjusted based on the dependency graph, tool concurrency limits, and cost budget. Scheduling happens only when dependencies are satisfied; write conflicts or scarce shared resources trigger a deliberate downgrade to serial execution.
    3. Context isolation and structured handoff: worker agents receive only the minimum context needed for their subtask and return results in a uniform structure (files, conclusions, evidence links), preventing the lead agent’s context from being flooded by intermediate steps.
    4. Result aggregation and conflict resolution: structured results pass integrity checks before being merged or selected (by test pass rate, static-check results, and so on), with the lead agent doing final integration when necessary.
    5. Fault tolerance and fallback: timed-out or failed subtasks are retried or re-planned individually without affecting completed branches; when parallel execution fails, the system automatically falls back to serial execution to guarantee correctness.

    Illustrative pseudocode:

    1. # Agent swarm orchestration (illustrative, not a real API)
    2. def swarm_run(task, planner, worker_factory, verifier,
    3. max_parallel=8, max_rounds=6, budget=None):
    4. graph = planner.decompose(task) # Subtask DAG
    5. done, failed = {}, set()
    6. for _ in range(max_rounds):
    7. # Schedule only subtasks whose dependencies are met and whose resources/budget allow
    8. ready = [n for n in graph.ready(done)
    9. if all(d in done for d in n.deps) and n not in failed]
    10. ready = graph.prioritize(ready, budget)[:max_parallel]
    11. if not ready:
    12. break
    13. workers = [worker_factory(n) for n in ready] # Isolated context per subtask type
    14. for node, result in zip(ready, run_concurrently(workers, timeout=...)):
    15. if verifier.check(node, result):
    16. done[node] = result
    17. else:
    18. failed.add(node) # Retry or fall back for failed branches only
    19. return planner.merge(task, done, failed) # Aggregation + conflict resolution

    6.3 Costs and Governance

    Parallelism is not free, and it has to be actively constrained in engineering:

    • Compute amplification: each subtask maintains its own context, so total token consumption is usually higher than a single serial run.
    • Shared-state conflicts: when multiple agents write to the same file or branch simultaneously, rely on the default isolation of each execution environment, or assign each agent its own workspace and merge afterward.
    • Tool rate limits: concurrent calls to third-party APIs can trigger throttling, requiring queues and backoff retries.
    • Observability: logs must span session, task, and agent levels so failed paths can be replayed.

    6.4 Matching Scenarios

    Task shape Recommended orchestration Reason
    Single-step Q&A, single-file edits Serial single agent Low parallel gain, higher cost
    Multiple candidate implementations, bulk page scraping Swarm parallelism Branches are independent
    Build → test → deploy Serial pipeline Strong dependencies and shared state
    Long chains with few branches Serial + checkpointed retries Simple and controllable

    7. Building Agent Capability: From Data to Evaluation

    7.1 Post-Training Data Mix

    • Tool-calling trajectories: multi-round “reason–act–observe” data covering both success and failure paths.
    • Execution-feedback data: using compilation, test, and validation results as reward signals.
    • Visual grounding data: screen element coordinates, bounding boxes, and document layout annotations to support screenshot-based operation.
    • Long-horizon task data: multi-step task chains to train planning and self-correction.
    • Pure-text replay: code, math, and general dialogue data mixed in at a fixed ratio to curb capability degradation from specialized training.

    7.2 Evaluation Dimensions

    Capability area Typical tasks Key metrics
    Code generation Algorithm problems, repository-level completion and repair Pass rate, repair success rate
    Visual understanding Document, chart, receipt, and screenshot Q&A Field accuracy, numeric reading error
    Mathematical reasoning Competition problems, word problems Final answer accuracy
    Agent execution Web operation, software automation Task success rate, average steps
    Computer use GUI grounding and operation Click hit rate, task completion rate

    According to public materials, models of this kind have achieved leading results on several of these benchmarks, with some metrics surpassing mainstream closed-source flagship models. Benchmark scores, however, cannot replace validation on real business workloads, and two indicators deserve particular attention:

    • Task success rate: whether real tasks can be completed independently;
    • End-to-end latency: look at P50 and P99, not just the mean.

    Long-tail failure rates and per-task cost should also be part of the acceptance criteria.

    8. Deployment and Engineering Practice

    8.1 Inference-Side Optimizations

    • Visual token compression (merging adjacent tokens) and visual caching to reduce repeated encoding of screenshots and documents.
    • Cache reuse for multimodal inputs.
    • Quantization, continuous batching, chunked prefill.
    • Separate deployment of prefill and decode.
    • KV cache offloading and reuse for long contexts.
    • Smaller or distilled models for worker agents to lower parallel cost.

    8.2 Platform-Side Architecture

    • Sandboxed execution environments and controlled network egress.
    • Tool-call timeouts and retry caps.
    • Human-in-the-loop confirmation gates.

    On Baidu AI Cloud, deployment can combine general-purpose capabilities such as container platforms, object storage, message queues, logging services, and monitoring and alerting:

    • Containerize multimodal inference services and pair them with elastic scaling to handle traffic fluctuations;
    • Host model weights and multimodal assets in object storage;
    • Use message queues to distribute parallel subtasks asynchronously and support failure retries;
    • Track success rates and tail latency through logging and monitoring with alerting.

    8.3 Security and Cost Control

    • Least privilege: grant each worker agent only the permissions needed for its task to avoid unauthorized operations.
    • Untrusted content isolation: treat text and links inside screenshots, web pages, and documents as data rather than instructions, preventing prompt injection from propagating across agents.
    • Cost control: tier model usage by business value; assess parallel gains against task value; set budgets and timeouts per subtask.

    9. Conclusion and Selection Guidance

    Native multimodal agent capability depends on the coordination of architecture, data, and engineering:

    • Architecture: native vision-language fusion, native-resolution encoding, and shared image/video weights determine the quality of “seeing and reading.”
    • Training: the decoupled encoder pipeline and high-quality agent trajectory data determine the reliability of “planning and acting.”
    • Engineering: dynamic parallel orchestration and isolated sandboxes determine the balance between “fast and stable.”

    Three principles can guide selection:

    1. Establish a baseline before redesigning: first get a minimal single-agent pipeline working end to end, then use real business data to identify whether the bottleneck is visual understanding, task planning, or tool calling, instead of adopting a multi-agent framework blindly.
    2. Put parallelism on the critical path: parallelism pays off only when latency is dominated by branches that can actually run concurrently.
    3. Validate on success rate, tail latency, and per-task cost together: benchmark scores are only the starting point; stability and cost in production are the prerequisites for long-term viability.