Multimodal LLM Training: Key Approaches, Architectures, and Design Trade-Offs
Explore the three major paradigms for training multimodal large language models: dual-encoder contrastive learning, visual encoders connected to language models, and native multimodal pretraining. This guide explains how to design modality bridges, handle image resolution and video sequences, balance training objectives and data, and evaluate models without sacrificing language capabilities.
Multimodal LLM training aims to let a model process images, video, and text within a shared set of parameters and a common decoding or matching space. This is more than attaching a vision module to a text generator. Data composition, modality connections, and loss functions determine whether the model can use cross-modal information reliably.
The field has broadly moved through three progressively more integrated approaches: dual-encoder contrastive alignment, a visual encoder connected to a language model, and native multimodal pretraining. Understanding their trade-offs is more useful for architecture selection and troubleshooting than memorizing individual model names.
1. Three Core Training Challenges
Representation mismatch. Images consist of local patch features and are often long and redundant; text consists of discrete symbols with high semantic density. Feeding visual features directly into a language model can disrupt attention because the two modalities differ in both sequence length and feature distribution.
Uneven capabilities. Models trained only for image–text matching can perform well at retrieval but poorly at generation. Adding a visual adapter to a frozen language model can also lead to a model that recognizes visual content but describes it imprecisely, or that loses some of its text-only capabilities.
Task generalization. Document understanding, multi-image comparison, long-video question answering, and referring segmentation have different requirements for resolution, temporal information, and output format. A fixed-length visual prefix may not serve all of them well.
Training is therefore often organized into four stages: modality encoding, cross-modal bridging, joint optimization, and preference alignment or other post-training. These stages can be iterated on independently, but evaluation should cover both multimodal and text-only tasks to avoid optimizing one at the expense of the other.
2. From Contrastive Learning to Joint Generative Modeling
Early vision–language pretraining largely relied on contrastive learning. A model uses separate image and text encoders and trains them on large collections of image–text pairs, bringing matched examples closer and pushing mismatched ones apart. This creates a shared semantic space for retrieval.
The approach has a clear objective and makes effective use of negative examples. It works well for zero-shot classification and cross-modal retrieval, but it lacks an autoregressive generation path and is not, by itself, well suited to open-ended question answering, description, or multi-turn dialogue.
A later, widely used approach combines a frozen or partially frozen visual encoder, a trainable bridge, and a language model. The visual encoder extracts hierarchical features, the language model handles reasoning and generation, and the bridge converts variable-length visual features into a prefix or an interleaved sequence the language model can consume.
Joint training may combine several objectives:
- Image–text contrastive learning
- Image–text matching
- Text generation conditioned on images
- Text-only next-token prediction
Contrastive and matching losses bring the modalities into alignment, while generation loss teaches the model to produce human-readable responses. If training uses only generation loss, fine-grained alignment may weaken. If the contrastive loss is weighted too heavily, it can constrain the diversity of open-ended generation. Track these objectives separately on validation data rather than relying on a single aggregate score.
Native multimodal pretraining takes integration further: the model consumes text and visual units through a unified autoregressive or sequence-modeling objective from early in pretraining. Instead of first training a text-only model and later attaching vision, the training process mixes text and multimodal samples at a fixed or curriculum-based ratio. Keeping the sampling balance stable can help language and cross-modal capabilities develop together.
This approach can reduce the alignment bottleneck created by an adapter: visual information does not have to pass through a narrow interface designed after the language model was trained. The trade-offs are greater demands on data engineering and compute scheduling, and a stronger dependence of language loss on the quality of visual tokenization.
3. What Learnable Queries Do in a Modality Bridge
In the encoder-plus-language-model approach, learnable queries are a common bridge between modalities. They are a fixed-length set of trainable vectors. After the visual encoder produces a large number of local features, the queries interact with those features through cross-attention: the queries act as receivers, and the visual features act as the memory being read.
After several layers, the bridge compresses a high-dimensional, variable-length visual sequence into a fixed-length vector sequence. This sequence can then be supplied to the language model as a prefix or as special tokens.
Learnable queries serve at least three purposes:
- Length adaptation: They prevent the full visual sequence from consuming the context window.
- Semantic selection: Cross-attention aggregates relevant regions and helps suppress background noise.
- Representation alignment: Together with contrastive and matching losses, they help place the resulting vectors near representations the language model can use.
The number of queries is an important hyperparameter. Too few may lose small objects or text; too many increase attention costs and consume space otherwise available for text. Documents, tables, and multi-image tasks may need more queries or a hierarchical query design, while ordinary image descriptions may work with a shorter prefix.
Later developments have generally changed how the query idea is implemented rather than abandoning it. Some use lighter resamplers to reduce the number of extra modules. Others vary projection with input resolution, assigning more tokens to high-resolution regions and merging less informative ones. Treat the bridge as a replaceable component and compare alternatives using the same downstream tasks—such as retrieval, fine-grained question answering, and multi-image reasoning—rather than assuming one design is always better.
4. Resolution, Video, and Positional Encoding
Resizing every image to the same fixed dimensions can irreversibly erase small text, seals, and dense table details. Dynamic resolution instead selects a grid based on the original aspect ratio within the available memory budget. The number of visual tokens can then vary with the amount of information rather than being forced to a single size.
Training must still impose a maximum token budget; otherwise, exceptionally large images can overwhelm batches. At inference time, the resizing strategy should match the training distribution. A mismatch between the image clarity seen during training and that of production inputs can undermine performance.
Video should not be treated simply as a stack of independent images. Neighboring frames are highly redundant, so encoding every frame in full makes compute grow linearly with the number of frames without necessarily modeling temporal relationships explicitly. Common techniques include:
- Sparse frame sampling, with denser sampling around key frames
- Reusing features across frames
- Including time in positional encoding
- Maintaining streaming memory that stores compressed representations of previously processed segments and updates incrementally
These techniques help a model answer questions such as when an object appeared and how it moved, rather than describing a single frame in isolation.
Methods such as multimodal rotary positional encoding assign height, width, and time to different rotary subspaces. This allows the same attention mechanism to distinguish spatial relationships as well as temporal order. Positional encoding addresses position conflicts, not semantics; semantics still depend on the data and training objectives.
Long videos also put pressure on the context budget. Dynamic visual-token sampling can merge or discard tokens from low-information segments, reserving the budget for scene changes and moments when text appears. Sampling needs to be reproducible; otherwise, repeated inference on the same video may produce inconsistent answers.
5. Losses, Data Mixtures, and Post-Training
During joint pretraining, autoregressive loss is typically the primary objective: given the preceding sequence, which may include visual tokens, the model predicts the next text token. Contrastive and matching losses are often applied to the bridge or dual-encoder components to stabilize alignment. They should not retain such a high weight in later training that they overwhelm the generation objective.
At a minimum, training data can be organized into three pools:
- High-quality text-only data
- Image–text pairs, including captions, question answering, and region-grounded descriptions
- Video–text pairs
A curriculum can first preserve stable language modeling, then increase the multimodal share, and finally introduce more expensive samples such as high-resolution documents and long videos. A sudden increase in multimodal data can weaken instruction following, increase hallucinations, and lower text-only benchmark performance.
Supervised fine-tuning uses instruction data to align capabilities with practical tasks: answering questions about images, comparing multiple images, summarizing video timelines, and identifying regions from descriptions. Data should include negative examples and cases where the model cannot answer. Without them, the model may learn to produce a complete-sounding answer regardless of whether the evidence supports one.
Preference optimization can then update a supervised model using pairs of responses, such as which answer is more accurate, less prone to hallucination, or more consistent with the requested format. Mixed preference optimization combines preference pairs with a portion of supervised signals, reducing the risk of language degradation from optimizing for preferences alone.
Test-time augmentation—such as generating multiple candidates, using self-consistency, or applying external checks—can improve complex reasoning performance. It adds inference-time cost, however, and should not be described as a capability already provided by training itself.
6. Engineering Considerations for Training and Evaluation
Because modalities and resolutions produce sequences of very different lengths, batch construction should bucket examples by visual-token count. This reduces wasted computation from padding.
Decide explicitly whether to freeze the visual encoder. Freezing saves memory and makes it easier to reuse general-purpose visual features. When domain images differ substantially from the training distribution—such as industrial inspection, remote sensing, or medical imaging—it may be necessary to unfreeze upper layers or fine-tune the entire encoder.
Use separate optimizer and learning-rate settings for the bridge, visual backbone, and language model. A bridge initialized from scratch will generally need a higher learning rate than a pretrained language-model backbone.
Evaluation should not rely on a single open benchmark. Report results separately for:
- Cross-modal retrieval or matching
- Fine-grained visual question answering
- Document and text understanding
- Multi-image reasoning and video temporal understanding
- Regression, if any, in text-only capabilities
For generation tasks, measure factual consistency and the ability to abstain when appropriate. If segmentation or localization is part of the model’s scope, specify whether outputs use text coordinates, special tokens, or a separate mask head. Classification accuracy is not a substitute for spatial metrics.
Experiment logs should record the data mixture, maximum visual-token count, query length, whether the visual tower is frozen, loss weights, and context length. These choices often affect results more than changing a similar model’s width.
For cloud training, sharded data in object storage, a container platform for multi-node jobs, and logging and monitoring for loss spikes and data-loading bottlenecks can support reproducible experiments without tying the setup to a proprietary interface. When using general-purpose compute and storage services such as Baidu AI Cloud, the priorities remain data versioning, checkpoint management, and frozen evaluation sets—not the platform itself.
7. Choosing a Training Paradigm
For retrieval, labeling, and zero-shot classification, dual-encoder contrastive learning is comparatively low-cost and straightforward to deploy; a generative LLM may not be necessary.
If a strong language model is already available and the main tasks involve description, question answering, or multi-turn dialogue, a visual encoder paired with learnable queries—or an equivalent resampling approach—is a practical and reliable incremental path. Pay close attention to query length and resolution handling.
If the requirements include preserving text-only performance, supporting multiple images and long videos, and reducing post hoc alignment, native unified pretraining may be a better fit. It comes with higher data and compute requirements, so use evaluation broken down by task and modality to ensure that one modality does not dominate.
Regardless of the approach, success should not be defined by a single score. Visual details must inform the model’s answers, language capabilities must remain stable, long sequences and high resolutions must stay within budget, and the model must be able to abstain rather than fabricate when uncertain. Treating losses, data, and evaluation as separable, iteratively testable components turns multimodal training from a simple connection between two models into a systematic engineering process.
