How Multimodal Large Language Models Are Trained: Paradigms, Components, and Strategies
Multimodal large language models learn to understand text, images, video, and audio within a single framework, but training them is far harder than training text-only models. This article walks through why multimodal training is complex, the three generations of core training paradigms (dual-tower contrastive learning, vision-encoder-plus-language-model, and native unified architecture), and the key components behind them, including learnable queries, dynamic resolution and M-ROPE position encoding, streaming memory for video, and Mixture-of-Experts. It then covers the training strategies and engineering practices that determine success, such as data mixing ratios, preference optimization, test-time enhancement, phased training, and infrastructure support.
1. Why Multimodal Model Training Is Harder
The goal of a multimodal large model is to let a single model understand multiple forms of information at once, such as text, images, video, and audio, and to perform generalization tasks like cross-modal retrieval, visual question answering, image captioning, and video understanding within a unified framework. Compared with a pure language model, the difficulty of training one comes from three main sources.
First is semantic alignment. Text is a sequence of discrete symbols, while an image is a continuous, high-dimensional tensor, so the two live in representation spaces that are naturally different. One of the core training tasks is to construct a mapping in high-dimensional space that pulls “a picture of a cat” and the word “cat” close to each other.
Second is the information-density gap across modalities. A video that lasts only a few seconds can contain a massive amount of redundant pixels, while the key semantics it carries may boil down to a single sentence. Figuring out how to compress visual information into a length the language model can process efficiently is a fundamental design bottleneck.
Third is capability synergy and conflict. Ideally, multimodal pretraining should not degrade the model’s original language ability, and the two capabilities should reinforce each other. Early approaches often showed a pattern of “multimodal ability improves while language ability degrades,” which has been a major driver of architectural evolution in recent years.
Once you understand these three points, the logic behind the evolution of multimodal training paradigms becomes clear.
2. Three Generations of Core Training Paradigms
The mainstream multimodal training paradigms can be grouped into three generations. They are not simple replacements of one another; rather, each has its own applicable boundaries depending on the scenario.
2.1 First Generation: Dual-Tower Contrastive Learning
The earliest approach was relatively simple: use a separate vision encoder and text encoder to map images and text into the same vector space, then apply contrastive learning over massive image-text pairs. During training, the vectors of paired image-text samples are pulled closer while unpaired ones are pushed apart, jointly optimized through an image-text contrastive loss (ITC) and an image-text matching loss (ITM).
The strengths of this paradigm are its simple structure and high retrieval efficiency, which make it well suited to cross-modal retrieval tasks such as searching images by text or text by image. However, its fundamental limitation is the lack of generation ability: the two encoders operate independently, so the model can only judge whether things are “similar” and cannot describe image content or answer questions in natural language.
2.2 Second Generation: Vision Encoder + Language Model
To give the model both understanding and generation ability, the field proposed a design of “frozen vision encoder + frozen large language model + a learnable bridging module in between.” A representative example connects vision and language through a set of learnable query vectors (the Q-Former mechanism): the query vectors extract information from image features via cross-attention, then feed into the language model as “soft prompts.”
The key evolution in this generation is making the bridging module more efficient. This progressed from early query transformers, to perceptron resamplers, and later to dynamic projection layers. The trend is to complete alignment with fewer parameters and shorter sequences, reducing both training and inference overhead. At the same time, a unified autoregressive loss is widely adopted, allowing tasks such as retrieval, captioning, and question answering to be jointly trained within the same generation framework.
2.3 Third Generation: Native Multimodal Unified Architecture
The second generation still has an alignment bottleneck in the form of the “visual adapter”: the language model only ever sees a compressed and filtered version of the world. The third-generation paradigm argues that text and visual features should enter the model together during the pretraining stage, rather than relying on a bolted-on adapter.
Typical practices include mixing text and multimodal data at a fixed ratio in the pretraining corpus (for example, a text-to-image-text sample ratio close to 1:3) so the model learns both modalities simultaneously from scratch; introducing a Mixture-of-Experts (MoE) architecture so that different subsets of parameters carry language and multimodal capabilities, achieving “co-improvement of pure language ability and multimodal ability”; and combining mixed preference optimization with test-time enhancement to further push the performance ceiling. Experiments generally show that this unified modeling approach eases the capability-conflict problem of conventional designs.
3. A Closer Look at Key Components
3.1 The Core Role of Learnable Queries
In second-generation architectures and some third-generation ones, learnable queries serve as the hub of cross-modal alignment and carry two core responsibilities.
Modal bridging: The patch features output by a vision encoder (such as a ViT) are high-dimensional and long in sequence, so feeding them directly into the language model is both expensive and inefficient. A fixed number of learnable query vectors (for example, 256) aggregate the visual features through cross-attention in a “query-and-answer” fashion, compressing them into a compact sequence of fixed length that fits the language model’s input format.
Semantic alignment: These query vectors have no fixed semantics of their own; their meaning is entirely learned through pretraining objectives such as contrastive loss and matching loss. After training converges, they become a set of “semantically oriented probes,” where each vector tends to capture one stable type of semantic cue in the image.
The schematic code below shows the basic data flow of query bridging (simplified pseudocode, meant to illustrate the mechanism rather than a specific implementation):
class QueryBridge(nn.Module):def __init__(self, num_queries, dim, num_heads):super().__init__()# A set of learnable query vectors acting as "probes" for cross-modal alignmentself.queries = nn.Parameter(torch.randn(num_queries, dim))self.cross_attn = MultiHeadAttention(dim, num_heads)self.norm = LayerNorm(dim)def forward(self, visual_feats):# visual_feats: [B, N_patch, dim] — raw output from the vision encoderq = self.queries.unsqueeze(0).expand(visual_feats.size(0), -1, -1)# Query vectors serve as Q, visual features as K/V, aggregated into a fixed-length sequencefused = self.cross_attn(q, visual_feats, visual_feats)# Residual + normalization; output is a compact representation ready for the language modelreturn self.norm(fused + q)
3.2 Dynamic Resolution and Multimodal Position Encoding
Traditional vision encoders require inputs of a fixed size, which loses detail in high-resolution images and is unfavorable for long-video understanding. Newer architectures introduce a dynamic resolution mechanism that splits an image into a variable number of patches and processes them according to the original aspect ratio, preserving detail while keeping the token count under control.
For the two-dimensional and spatiotemporal structure of images and video, standard one-dimensional position encoding no longer suffices, so solutions such as multimodal Rotary Position Embedding (M-ROPE) are needed. M-ROPE decouples and encodes positional information along the time, height, and width dimensions, enabling the model to both understand spatial relationships in static images and model temporal evolution in video.
3.3 Streaming Memory for Video
The biggest challenge in video understanding is the sequence explosion caused by the number of frames. To address this, a streaming memory mechanism was introduced and combined with a hierarchical vision encoder for multi-scale feature fusion: instead of processing all frames equally, the model maintains a compact “memory” that is continuously updated as inference progresses, compressing and retaining historical information while encoding the current frame in fine detail. The direct benefits of this design are a significant drop in the number of interactions, a multiplicative speedup in inference, and support for processing at frame rates close to real time.
3.4 Mixture-of-Experts and Ultra-Long Context
The Mixture-of-Experts architecture routes different inputs to different subsets of parameters through a gating network, decoupling language and multimodal abilities at the parameter level while still sharing them, which makes it a key engineering enabler for landing the native unified architecture. Combined with strategies such as dynamic visual token sampling, the model can retain fine-grained abilities like zero-shot image segmentation even under ultra-long contexts (for example, tens of millions of tokens).
4. Training Strategies and Engineering Practices
Beyond the architecture, training strategies are just as decisive for success. The following points are especially important:
Data ratio and ratio-based sampling. The ratio of text to multimodal samples directly affects whether language ability degrades. Native unified architectures generally adopt controlled mixed sampling (such as a text-to-multimodal ratio close to 1:3) and apply weighting to samples such as high-resolution, multi-image, and long-video content.
Preference optimization. After supervised fine-tuning, mixed preference optimization (MPO) is introduced to rank and optimize model outputs using multiple preference signals, which effectively improves the factuality of answers and instruction-following ability.
Test-time enhancement (TTS). At inference time, applying multiple sampling rounds, voting, or reranking over candidate outputs to “trade compute for quality” is a common way to approach the performance of strong closed-source models.
Phased training. The field generally follows a “align first, then instruction tuning, then preference optimization” pipeline: first learn alignment on large-scale image-text pairs, then inject downstream task abilities, and finally polish interaction quality.
Infrastructure support. Large-scale training relies on the coordination of general cloud capabilities such as object storage (for storing massive image, text, and video data), message queues (for decoupling the data pipeline), container platforms (for distributed scheduling), and logging and monitoring alerts. These form the invisible foundation of training stability.
5. Summary and Outlook
At its core, training a multimodal large model is an engineering practice of answering “how to build an efficient, transferable mapping between different representation spaces.” From dual-tower contrastive learning, to query bridging, to native unified modeling, each leap in the paradigm answers the same question: how to let vision and language both collaborate and avoid conflict within a single model.
For developers and teams, what truly needs to be mastered is not a checklist of any particular model’s capabilities, but this transferable design logic: how the bridging module compresses and aligns information, how position encoding adapts to spatiotemporal structure, how MoE decouples capabilities, and how data ratios and preference optimization work together. Understanding these underlying mechanisms is what enables well-grounded decisions during selection, fine-tuning, and deployment. As unified architectures, streaming memory, and efficient sampling continue to evolve, multimodal models are moving toward lower training costs, stronger long-context ability, and more natural real-time interaction, opening up broad space for building next-generation intelligent applications.
