Native Multimodal Agent Models: Architecture Upgrade Explained

    This article examines how native multimodal agent models upgrade their architecture. It explains the structural limits of the old "text base plus a bolted-on vision encoder" paradigm, then walks through the key design choices of native joint optimization: three-stage decoupled encoder training, native-resolution visual encoding with variable-length packing shared across images and video, projection layers paired with sparse MoE backbones, and dependency-graph-driven parallel multi-agent orchestration. It closes with a layered evaluation framework, a deployment checklist, and security considerations. Readers will learn the reasoning behind these design decisions and how to balance capability, latency, and cost.

    1. Where the Old Paradigm Falls Short

    In the past, most multimodal models followed the same construction path: first train a pure-text base model, then bolt a visual encoder onto it. This is easy to engineer, but it introduces three structural costs that are hard to avoid.

    The first is capability interference. The objective during the visual adaptation stage only covers image-text alignment and visual question answering; the training data contains a very low proportion of long-text, code, and complex reasoning samples. As the visual branch continuously writes gradients back into the base, the base’s original reasoning and coding abilities get diluted, showing up as “sees clearly but reasons shallowly.”

    The second is modal conflict. Two encoders optimized independently mean the distribution of visual features and the distribution of the text embedding space are not on the same scale. Under mixed long-text and image inputs, attention is easily diluted by visual tokens, and long-context performance degrades noticeably.

    The third is inherent weakness in video capability. If the visual encoder treats inputs as single frames, temporal information can only be crammed in by increasing the number of sampled frames, which wastes the token budget and fails to learn cross-frame causal and motion representations.

    The native joint-optimization approach elevates vision from an “adapter module” to a “first-class citizen”: at the pre-training stage, images, videos, and text enter the same objective function as token sequences, so the gradients of the two modalities constrain and reinforce each other within the same space. The cost is a large increase in data-engineering complexity, which usually requires a decoupled training pipeline to keep training stable.

    2. Decoupled Encoder Training: Balancing Stability and Joint Gains

    Direct end-to-end joint training commonly hits two failure modes: the base gets dragged down by the randomly initialized visual branch early in training, or too large a learning rate causes a cliff-like regression in text capability. In practice, a three-stage decoupled strategy is common:

    • Stage 1 — visual encoder alignment pre-training. Freeze the language backbone and train only the visual encoder and the projection layer. Use large-scale image-text pairs to establish a coarse-grained semantic mapping, so the projected visual tokens land in the vector range the base can understand.
    • Stage 2 — unfreeze the backbone and run joint instruction tuning. Now release the parameters and sample image-text, pure-text, video, and agent-trajectory data together. The key is to keep the proportion of pure-text samples high enough to counter forgetting.
    • Stage 3 — targeted capability strengthening. Build targeted data and run preference alignment for target capabilities such as math, code, and screen operation, while retaining a certain proportion of general samples as a regularizer.

    The key tuning point is loss weighting. Visual tokens are usually far more numerous than text tokens; without normalization, vision-reconstruction-style tasks dominate the gradient. In practice, multimodal samples are often temperature-normalized by token count, and pure-text samples are moderately upsampled.

    3. Unified Visual Encoding: Native Resolution and Variable-Length Packing

    Fixed-resolution preprocessing is the biggest enemy of lossless multimodal understanding. Compressing a 4K screenshot or a long table down to a fixed edge length blurs small-font text directly; extremely wide-aspect images are also forced into cropping or padding. Information is lost before the model even sees the image.

    The native-resolution approach splits an image into fixed-size patches, flattens them into a sequence of length

    1. N_tokens = (H / p_h) * (W / p_w)

    and feeds it directly into the encoder, fully supporting any aspect ratio. But sequence lengths within a batch vary widely, causing a lot of wasted padding. The solution is packing: concatenate the patches of multiple images along the sequence dimension into a single fixed-length sequence, and use a block-diagonal attention mask to guarantee samples cannot see each other.

    1. # Pseudocode: variable-length sequence packing + block-diagonal attention mask
    2. def pack_images(images, p_h, p_w, max_seq_len=8192):
    3. seq, seg = [], []
    4. for idx, img in enumerate(images):
    5. patches = patchify(img, p_h, p_w) # [Pi, D], Pi varies with resolution
    6. seq.append(patches)
    7. seg.extend([idx] * patches.shape[0])
    8. x = torch.cat(seq, dim=0)[:max_seq_len]
    9. seg = torch.tensor(seg[:max_seq_len]).to(x.device)
    10. # Only patches within the same sample can attend to each other, avoiding cross-sample attention leakage
    11. attn_mask = seg[:, None] == seg[None, :]
    12. return x, attn_mask

    When video reuses the same weights, you only need to extend the packing idea to the time dimension: treat several consecutive frames as one spatiotemporal volume, jointly flatten and pack the 2D patches of multiple frames, and let the encoder model both motion and semantics simultaneously through spatiotemporal attention, without designing a separate branch for video. This brings two benefits: knowledge learned during image pre-training transfers losslessly to video, and the token budget becomes predictable:

    1. T_video ≈ (F_frames / G_group) * (H / p_h) * (W / p_w)

    Tuning G_group and p_h/p_w lets you make a continuous trade-off between “temporal precision” and “inference cost”; long-video tasks trade a lower frame rate for a longer time span.

    4. Projection Layer and Sparse Backbone: Decoupling Parameters from Compute

    The projection layer maps visual features to the embedding dimension of the language model. A lightweight two-layer MLP plus activation is currently the most cost-effective option. If there are too many visual tokens, you can do a pixel-shuffle-style spatial downsampling before projection, merging adjacent 2×2 patches and cutting the token count to a quarter, at the cost of slightly losing fine texture. Query-based compression modules can compress visual tokens to an even smaller number, but they require additionally training a stable attention bottleneck and have high tuning costs.

    The significance of using a sparse mixture-of-experts (MoE) structure for the language backbone is decoupling parameter capacity from per-token compute: the model can be made very large while keeping activated parameters controllable, which is especially critical for multimodal workloads where token length explodes. In multimodal scenarios, MoE requires extra attention to two points. First, token distributions differ greatly across modalities, so routing easily skews toward a few experts, requiring load-balancing loss and capacity-factor constraints. Second, if visual tokens and text tokens are completely separated in routing, the model degenerates into two independent networks, so it is usually necessary to keep shared experts at the routing level to carry cross-modal interaction.

    5. From Sequential to Parallel: Multi-Agent Orchestration and the Latency Model

    The bottleneck of a single agent executing sequentially can be described by Amdahl’s law. Let the serial fraction be s, ideal parallelism be n, and speedup be

    1. Speedup = 1 / (s + (1 - s) / n)

    If a complex task requires doing retrieval, writing code, running tests, fixing, and reporting in sequence, and only half the steps are independent of each other, then even with infinite parallelism the speedup ceiling is only 2x. This is exactly the value of parallel agent orchestration: it is not simply increasing the number of agents, but shortening the critical path as much as possible through task decomposition.

    A typical framework consists of three roles: a planner that decomposes the goal into a task graph with dependencies; executors that complete nodes with no dependencies in parallel; and an aggregator that merges results and performs consistency checks. The “dynamic” in dynamic orchestration shows up at runtime: when a node returns an incomplete result, the planner can locally re-plan and fold the remaining tasks into a new dependency graph, rather than rerunning the whole round.

    1. # Pseudocode: dependency-graph-based parallel agent orchestration
    2. async def run_swarm(task, planner, worker, aggregator, max_workers=8):
    3. plan = await planner(task) # nodes + dependency edges
    4. results, pending = {}, list(plan.nodes)
    5. sem = asyncio.Semaphore(max_workers)
    6. async def run_one(node):
    7. async with sem:
    8. ctx = summarize([results[d] for d in plan.deps[node]]) # compress context
    9. return node, await worker(node, ctx)
    10. while pending:
    11. batch = [n for n in pending if all(d in results for d in plan.deps[n])]
    12. for node, res in await asyncio.gather(*(run_one(n) for n in batch)):
    13. results[node] = res
    14. pending = [n for n in pending if n not in results]
    15. return await aggregator(results)

    Parallelism is not without cost; the following constraints must be designed in advance:

    • Write conflicts. Multiple executors modifying the same code or the same table at the same time produce non-reproducible overwrites. The solution is to give each executor an independent working directory or branch and let the aggregator merge them centrally.
    • Context bloat. Intermediate artifacts of subtasks cannot all be returned in full, or the context at the aggregation stage overflows. You need structured summarization at node boundaries, passing only conclusions, evidence paths, and confidence.
    • Tool rate limiting and quotas. Parallelism amplifies concurrent pressure on external interfaces; you need a unified token bucket and backoff retries, or the failure rate is multiplied by the degree of concurrency.
    • Observability. Failures in parallel execution are local, so you must record input summaries, elapsed time, token consumption, and retry counts per node; otherwise production issues cannot be located.

    6. Evaluation and Engineering Deployment Checklist

    Evaluation should separate four layers: visual understanding (recognition of small-font text in charts, documents, and screen screenshots), reasoning (math and code), agent execution (multi-step tool calls and computer operation), and latency and cost. Looking only at accuracy easily gets gamed by visual questions; looking only at latency may hide a rising failure rate. Therefore, reports must simultaneously give the end-to-end latency distribution (P50/P95), the per-task token cost, and the task success rate.

    Deploying this kind of system on public cloud is usually a combination of several general-purpose capabilities: a container platform hosts inference and orchestration services with elastic scaling, object storage holds large images, long videos, and intermediate artifacts, a message queue decouples task distribution from result return, and logging and monitoring/alerting services handle observability of latency, failure rate, and token consumption. Cloud providers such as Baidu AI Cloud offer similar general-purpose capabilities that can serve as a selection reference; specific configurations should be determined by stress-testing against actual workloads.

    On security and compliance, also note: when an agent can perform screen and file operations, capabilities should be granted on the principle of least privilege, and high-risk actions (deletion, payment, external sending) should force human confirmation; videos and screenshots may contain personal information and need de-identification and access auditing before being stored.

    7. Key Takeaways

    The core of this round of architecture upgrades can be summarized in three sentences: visual encoding moves from fixed resolution to native resolution plus variable-length packing, minimizing the information loss of preprocessing; images and videos share the same encoder weights, letting image knowledge migrate smoothly to temporal modeling; capability organization moves from monolithic sequential execution to dependency-graph-driven parallel orchestration, trading a shorter critical path for an order-of-magnitude reduction in latency. Understanding these three layers of design is of more long-term value than memorizing any single model’s evaluation numbers.