Multimodal Large Model Architecture, Training Paradigms, and Reasoning Enhancement: A Deep Dive

    Vision-language models (VLMs) aim to understand images, video, and text within a single semantic space. This article dissects the technical stack of modern multimodal LLMs layer by layer: starting from the modular "visual encoder + projector + LLM" design and its inherent limitations, moving through native multimodal architectures built on spatiotemporal vision encoders and MoE language models, then covering the full training pipeline (pretraining, supervised fine-tuning, reinforcement learning, and reward design), and finally examining inference-time reasoning enhancement through the "thinking with image" paradigm. It is written for AI engineers and researchers who want to understand how architecture, training, and reasoning are co-designed rather than optimized as isolated points.

    1. Introduction: From Modular Assembly to Native Multimodal

    The goal of multimodal large models (Vision-Language Models, VLMs) is to let a model understand images, video, and text simultaneously, and to perform tasks such as question answering, captioning, and reasoning within a unified semantic space. Over the past few years, the mainstream approach in the industry has been to “stitch together” mature components—a visual encoder, a projection layer, and a large language model—into a modular architecture. This approach is easy to implement and fast to iterate, but it has increasingly exposed a ceiling on model capability. The more recent trend is toward native multimodal models, where visual and language signals are deeply fused early in training and the architecture is redesigned from the ground up to achieve stronger cross-modal understanding. This article breaks down the technical system layer by layer, from architecture, to training, to reward design, and finally to reasoning enhancement.

    2. Mainstream Modular Architectures and Their Pain Points

    Most VLMs today adopt a modular design, typically composed of three parts:

    • Visual Encoder: slices an image or video into patches and encodes them into a sequence of visual features.
    • Projector: usually a multi-layer perceptron (MLP) that maps visual features into the word-embedding space of the language model.
    • Large Language Model (LLM): receives a mixed sequence of “visual tokens + text tokens” and handles understanding and generation.

    The core pain point of this paradigm is the split in inductive bias. A visual encoder (such as a common self-supervised image model) is unaware of language during pretraining, and its feature space is optimized purely for visual tasks. The projector performs some alignment, but it is only a relatively shallow mapping. As a result, visual and language information are not fused thoroughly, and the model is limited on tasks that require fine-grained cross-modal reasoning. In addition, the modules are usually trained in separate stages and optimized independently, so end-to-end consistency is hard to guarantee. This leads to phenomena such as “can recognize but cannot reason” and “can describe but gets detail questions wrong.”

    3. Native Multimodal Architecture Design

    The core idea of native multimodal is to stop relying on a pretrained visual encoder for “ready-made features,” and instead let the visual branch and the language branch co-evolve under the same training objective. A representative architecture consists of three major components.

    1. Spatiotemporal Visual Encoder (3D Vision Encoder)

    For images, the encoder directly splits pixels into patches and applies positional encoding; for video, it introduces a temporal dimension. An effective approach is to treat every 4 consecutive video frames as a “spatiotemporal block” and jointly model spatial and temporal relations within the same encoding unit. This both compresses the number of video tokens—reducing memory and compute overhead for long videos—and captures motion and cross-frame semantics early on, rather than simply concatenating multiple independent images. This encoder can be initialized from weights obtained via image self-supervised pretraining and then further trained on multimodal data.

    2. Projector

    The projector still typically uses an MLP structure to align visual tokens to the language model’s embedding space. On the native multimodal route, its role leans more toward “interface alignment”; what truly determines cross-modal fusion capability is the subsequent end-to-end joint training.

    3. Mixture-of-Experts LLM (MoE LLM)

    The language side can adopt a Mixture of Experts (MoE) architecture. MoE uses a gating network to dynamically route tokens to different expert subnetworks, thereby expanding the total parameter count without a significant increase in per-token computation. In multimodal settings, visual tokens and text tokens may be routed to different experts, giving the model a natural degree of “modality specialization” that helps maintain inference efficiency while scaling up.

    The data flow of the three components can be abstracted as follows:

    1. # Native multimodal forward pass (illustrative)
    2. def forward(image_or_video, text):
    3. # Video: group every 4 frames into a spatiotemporal block; image: a single frame
    4. visual_tokens = vision_encoder_3d(image_or_video) # [N_v, d]
    5. visual_tokens = projector(visual_tokens) # Align to the LLM embedding space
    6. text_tokens = llm.embedding(text) # [N_t, d]
    7. fused = concat([visual_tokens, text_tokens]) # Unified sequence
    8. logits = moe_llm(fused) # End-to-end joint modeling
    9. return logits

    4. Training Pipeline: Pretraining, Post-Training, and the Reward System

    The goal of the pretraining stage is to establish the foundational ability for cross-modal alignment, typically using hundreds of millions or more image-text / video-text pairs. The key here is not the absolute volume of data, but rather data mixture and quality: the proportions of different tasks—image understanding, video temporal modeling, documents/OCR, charts, natural dialogue, and so on—directly affect the model’s capability distribution. Representative work shows that a few hundred million high-quality data points, combined with a carefully designed native architecture, can already achieve competitive results across multiple benchmarks, indicating that “data quality + architecture design” matters more than simply piling on data.

    The post-training stage is generally split into two steps:

    1. Supervised Fine-Tuning (SFT): uses high-quality instruction data to teach the model to follow dialogue formats and complete specific tasks.
    2. Reinforcement Learning (RL): further optimizes model outputs through reward signals so they better align with human preferences and task objectives.

    Reward System design is the core of the RL stage. Common approaches include:

    • Rule-based verifiable rewards: for tasks with clear right/wrong answers—math, code, format constraints—a program automatically judges correctness. The signal is stable and hard to exploit via “reward hacking.”
    • Preference rewards from a reward model: a separately trained reward model scores answer quality, suitable for more subjective, open-ended tasks.
    • Multi-dimensional reward fusion: combining multiple weighted signals such as correctness, relevance, format compliance, and safety.

    Improvements to the Proximal Policy Optimization (PPO) algorithm: PPO limits the magnitude of policy updates by clipping the importance sampling ratio, avoiding training collapse. Its objective can be simplified as follows:

    1. import torch
    2. def ppo_loss(old_log_prob, new_log_prob, advantage, clip_eps=0.2):
    3. ratio = torch.exp(new_log_prob - old_log_prob)
    4. clipped = torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps)
    5. # Take the smaller of the clipped and unclipped objectives to keep updates conservative
    6. return -torch.min(ratio * advantage, clipped * advantage).mean()

    Common improvement directions include: adding a KL penalty term to the loss to keep the new policy from drifting too far from the reference policy; normalizing the advantage to stabilize gradients; and introducing group-relative estimation methods that reduce dependence on an extra value network, thereby lowering memory overhead and improving training stability.

    5. Reasoning Enhancement: Using Images as a Thinking Medium

    A direction worth attention is the “think with image” reasoning paradigm. Traditional visual reasoning often stops at “look and describe,” yet in tasks such as document OCR, perception of damaged/low-quality images, and chart analysis, the model needs to “zoom in, rotate, crop, and enhance” images before making a judgment—much like a human would. Relying on a single tool such as zoom-in addresses only a limited set of problems, and multi-turn tool calling often ends after just two or three rounds.

    A more general approach is to use code as the intermediate medium: instead of directly outputting a final answer, the model first generates a piece of executable image-operation code (rotation, enhancement, region cropping, etc.), runs it in a restricted sandbox to obtain a new intermediate image, and then continues reasoning based on the result—forming a closed loop of “observe–operate–re-observe.” The flow can be abstracted as:

    1. def think_with_image(image, question, max_rounds=3):
    2. history = [image, question]
    3. for _ in range(max_rounds):
    4. # The model generates a piece of image-operation code
    5. code = model.generate_code(history)
    6. # Execute it in a restricted sandbox to obtain an intermediate result image
    7. result = sandbox.run(code, image) # Note isolation and resource limits
    8. history.append(result)
    9. # The model decides whether the information is sufficient to answer
    10. if model.is_ready(history):
    11. break
    12. return model.generate_answer(history)

    Compared with a fixed tool set, the code-as-medium approach is naturally extensible: adding a new image operation only requires extending the executable operator library, with no change to the model interface. The industry is also starting to see benchmarks specifically designed to evaluate image-operation capability, indicating that this direction is moving from scattered exploration toward a more systematic approach.

    6. Data Construction and Reward Design in Practice

    For the multi-turn “think with image” method, the difficulties in implementation concentrate on two areas—data construction and reward design—and the two must work together:

    • Data construction: samples need to be built with explicit intermediate operation sequences—for example, “for blurry text → first sharpen/enhance → then recognize”—to demonstrate a reasonable chain of operations to the model. It also requires constructing complex samples that need multiple rounds of operations, preventing the model from degenerating into single-round zoom-in.
    • Reward design: rewards should cover both the correctness of the final answer and the effectiveness of the operation process (whether meaningful image operations were performed, whether the process converged rather than looping indefinitely). For verifiable tasks, rule-based rewards can judge the final result; for the process, constraints can be set on “effective operation steps” and “avoiding redundant repetitions” to prevent score gaming.

    In terms of training strategy, compared with letting the model explore freely, introducing process supervision can significantly improve the stability of multi-turn reasoning—that is, judging not only whether the result is correct, but also whether each step of the operation is reasonable.

    7. Engineering Practice and Cloud Deployment

    Native multimodal models (especially those involving video and MoE structures) place high demands on compute, memory, and storage. For engineering deployment, the following general capabilities are worth considering (expressed here as generic cloud product categories):

    • Object storage: stores large-scale image-text/video training corpora and intermediate results, supporting high-concurrency reads.
    • Container platform: containerizes training and inference services to enable elastic scaling and GPU resource scheduling.
    • Message queue: decouples asynchronous tasks such as data preprocessing, sandbox execution, and post-processing.
    • Logging and monitoring/alerting: tracks training loss, reward curves, and inference latency to detect anomalies promptly.

    On the inference side, the large number of video tokens and the extra overhead from MoE routing mean that general optimization techniques such as KV Cache, token compression, and dynamic batching are advisable, combined with quantization to reduce memory usage. The sandbox that executes image-operation code must implement solid resource and permission isolation to prevent malicious code or infinite loops from bringing down the service.

    8. Summary and Outlook

    The evolution of multimodal large models follows a clear trajectory: from the modular “encoder + projector + LLM” assembly toward native multimodal, where vision and language are deeply fused at the architecture level; in training, from pure SFT toward “SFT + reinforcement learning + fine-grained reward design”; and in reasoning, from passive description toward active, multi-turn operation using images as a thinking medium. Looking ahead, as technologies such as spatiotemporal modeling, MoE sparse scaling, process rewards, and verifiable rewards mature, multimodal models will demonstrate greater practical value in complex scenarios such as document understanding, video analysis, and low-quality image perception. For developers, understanding the logic of this “architecture–training–reasoning” co-design is more useful for grasping the essence of the technology’s evolution than chasing any single metric.