Multimodal Large Model Training: Methods and Paradigm Evolution
This article explains how multimodal large models map images, video, and text into a shared semantic space and how they are trained. It covers the shift from contrastive alignment to native multimodal pretraining, data and sampling strategies, connector architectures, loss functions, engineering practices, and common trade-offs.
Multimodal large models are designed to map heterogeneous signals such as images, video, and text into a unified semantic space, where they can perform understanding, retrieval, and generation. Training such models is not simply a matter of attaching a vision module to a language model. Instead, the process iterates around three main threads: representation alignment, information compression, and task generalization. The following sections describe how these models are typically trained across five areas: paradigm evolution, data and sampling, alignment structures, loss functions, and engineering practice.
How Training Paradigms Move from Alignment to Unification
Early vision-language pretraining relied mainly on contrastive learning. Large-scale image-text pairs were used to pull matching samples closer and push mismatched samples apart, producing a shared embedding space. This approach is effective for retrieval and zero-shot classification, but its generation capability is weak, and downstream tasks often require an additional generation head.
The second stage commonly adopted a combination of a frozen vision encoder, a learnable connector, and a language model. The connector could be query-based resampling, perceiver-style compression, or dynamic projection. It compressed variable-length visual features into a fixed or quasi-fixed token sequence, which was then fed into the language model for autoregressive training. This approach reuses a mature language foundation and keeps training costs relatively manageable. Its drawback is that vision and language are still optimized in stages, creating an alignment bottleneck.
The third stage emphasizes native multimodal pretraining. Text and multimodal samples are mixed at a certain ratio, often with text and image-text or video data sampled at roughly 1:3 or an adjustable ratio. The model learns under the same set of parameters and the same autoregressive objective. Position encoding also expands from text-only encoding to multimodal rotary position encoding or modality-specific position encoding, so that it covers both two-dimensional space and the time axis. This route reduces the information loss caused by adding an adapter after the fact, and language ability and multimodal ability can reinforce each other.
Data, Resolution, and Token Strategies
Training quality depends heavily on data ratios and cleaning. Image-text pairs, document screenshots, video captions, and instruction dialogues need layered filtering: deduplication, aesthetic and clarity screening, OCR quality checks, and image-text consistency scoring. Long-video understanding also requires temporal sampling strategies that avoid taking only the first and last frames and losing action information.
With respect to resolution, fixed resizing loses detail. A common industry practice is dynamic resolution: cropping or interpolating according to the original aspect ratio, then preserving spatial structure through two-dimensional position encoding. For ultra-high-resolution inputs, multi-scale feature fusion is used, going from coarse to fine. The number of visual tokens must match the context window. Too many tokens crowd out language tokens, while too few lose local information. Dynamic sampling or learnable compression is therefore often introduced so that the token budget adapts to the task.
Connector Structures and Training Objectives
The role of the connector is to compress while preserving semantics. Query-based structures use a set of learnable queries to extract information from visual features. Perceiver-style resampling uses cross-attention to map features of arbitrary length to a fixed length. Dynamic projection adjusts the projection layer according to resolution. In all cases, the goal is to make the visual tokens seen by the language model both compact and decodable.
For loss functions, contrastive loss is used in the alignment stage, and autoregressive cross-entropy is used in the generation stage. Unified autoregression places image tokens and text tokens in the same sequence and predicts the next token, simplifying multi-task heads. To improve instruction following and safety, mixed preference optimization can be introduced after alignment. It uses ranking data, rejection sampling, and test-time scaling together so that the model is more robust on complex question answering and document understanding. Video tasks also add temporal consistency constraints to reduce jitter between frames.
Key Engineering Practices
Large-scale training requires stable mixed precision, gradient checkpointing, and pipeline parallelism. The learning rates for the vision encoder and the language model are usually set separately: the encoder can be fine-tuned with a smaller learning rate or unfrozen in stages, while the connector and language model use relatively larger learning rates. In long-context scenarios, attention uses chunked or sparse strategies, together with a streaming memory mechanism that caches historical frame features. This supports real-time interaction without recomputing the full sequence.
Evaluation should cover retrieval, document question answering, video question answering, and open-ended generation, avoiding overfitting to a single metric. During training, zero-shot and few-shot spot checks should be performed regularly to monitor image-text alignment, hallucination rate, and instruction-following rate. Cloud training can use object storage to hold massive image-text data, container platforms for elastic scaling, and logging and monitoring services to track loss curves and GPU utilization. General compute and storage capabilities from cloud platforms such as Baidu AI Cloud can support the full pipeline from data preprocessing to multi-machine, multi-GPU training, but the specific scheduling still depends on how the task itself is partitioned for parallelism.
Common Challenges and Trade-offs
Insufficient alignment can cause image description to drift. Over-aggressive compression can lose text and fine-grained objects. A pure concatenation adapter may cause the model to degrade on text-only tasks. In practice, data ratios, unfreezing strategies, and token budgets must be repeatedly balanced. A native unified architecture reduces alignment friction, but it places higher demands on compute and data quality. Staged training makes it easier to reuse an existing language foundation, but it requires extra investment in connector design. There is no absolute optimum, only a combination that matches the business scenario.
In summary, the essence of multimodal large model training is to use contrastive or autoregressive objectives to bring heterogeneous modalities into the same prediction space, use learnable compression to control the scale of visual tokens, and use mixed data and preference optimization to improve generalization. Once this process is understood, the full loop from alignment and joint training to instruction fine-tuning can be reproduced on proprietary data without relying on a specific closed-source implementation.
