Multimodal Model Training Pipeline: From Alignment Pretraining to Preference Optimization

    This article explains the end-to-end training pipeline for multimodal large models, covering the shift from early alignment-based architectures to unified fusion models, the three core training stages of modality alignment pretraining, cross-modal instruction tuning, and preference alignment, as well as data engineering, dynamic resolution, multimodal positional encoding, training stability, and deployment considerations. Readers will gain a structured understanding of the key technical decisions and engineering trade-offs involved in building and optimizing multimodal models.

    1. The Evolution of Multimodal Training Paradigms

    The technical路线 for multimodal large models has undergone a paradigm shift from “stitched alignment” to “native fusion.” Early mainstream approaches equipped each modality with an independent encoder and then used a shallow alignment module to map visual features into the language model’s semantic space. In essence, this was still a “translation” approach. After 2023, the industry increasingly recognized that true multimodal intelligence requires deep intra-modal and cross-modal collaboration. As a result, training objectives began to move from “alignment” toward “fusion,” and model architectures evolved from external adapters to unified backbone networks.

    From a technical implementation perspective, this evolution is reflected in three key dimensions:

    1. Visual feature extraction: The approach moved from grid-based patching at fixed resolutions to dynamic resolution, multi-scale feature pyramids, and learnable-query-based compressed sampling. This allows models to retain denser visual information at lower computational cost.
    2. Depth of cross-modal interaction: Fusion evolved from shallow interaction only at the alignment layer to joint visual-language attention computation inside every Transformer layer, enabling a true “you see what I mean” effect.
    3. Consistency of training objectives: Training moved from separate staged optimization—alignment first, generation later—toward a unified autoregressive loss function, allowing the model to learn multimodal semantic mapping autonomously under a single objective.

    2. Technical Breakdown of the Three-Stage Training Pipeline

    Stage 1: Modality Alignment Pretraining

    This is the “foundation” stage of multimodal model training. Its core goal is to use large-scale image-text pair data to establish an initial mapping between visual features and language semantics. This stage typically combines contrastive learning losses with generative losses.

    Specifically, contrastive learning pulls matching image-text pairs closer in semantic space and pushes mismatched pairs apart, gradually aligning the visual encoder’s output with the text embedding space. At the same time, the generative objective requires the model to generate corresponding textual descriptions when given only image features. This process forces the model to extract key information from visual content that can be expressed in language, rather than simply “copying and pasting” surface signals from visual features.

    It is worth noting that data quality in the alignment stage directly determines the model’s later ceiling. In real engineering practice, strict data cleaning pipelines are required, including but not limited to removing noisy samples with irrelevant image-text relationships, filtering low-resolution or blurry images, eliminating potential social bias and harmful content, and removing large-scale duplicates through clustering methods. A widely cited practical guideline is that tens of thousands of high-quality manually annotated samples often achieve better alignment results than tens of millions of uncleaned automatically crawled samples.

    Stage 2: Cross-Modal Instruction Tuning

    Once the model has basic visual-language alignment capability, it enters the instruction tuning stage. The goal of this stage is to activate the model’s instruction-following ability and task generalization ability, enabling it to understand and execute diverse user requests.

    Instruction tuning data is typically structured in a conversational format such as:

    1. {
    2. "image": "path/to/image.jpg",
    3. "conversations": [
    4. {
    5. "role": "user",
    6. "content": "What is the emotion of the person in this image? Please make a judgment based on the background."
    7. },
    8. {
    9. "role": "assistant",
    10. "content": "The scene is a city street after rain. The person is smiling and walking lightly, and with the warm shop-window lighting, the overall emotion tends toward pleasant and relaxed."
    11. }
    12. ]
    13. }

    The core technical detail in this stage is balancing task diversity. Training data should cover multiple task types, such as image-text dialogue, visual question answering, image captioning, chart reasoning, document understanding, and multi-image comparison, to avoid overfitting to a single interaction pattern. In addition, the difficulty of instruction data should follow a gradient distribution—from simple factual answers to complex reasoning and multi-hop question answering—gradually increasing the model’s cognitive load.

    Stage 3: Preference Alignment and Safety Optimization

    Alignment pretraining and instruction tuning give the model strong capabilities, but they do not guarantee that the model always “acts according to human preferences.” The purpose of the preference alignment stage is to make model outputs better match human values and usage expectations while strengthening the ability to refuse unsafe inputs.

    The current mainstream technical approach is hybrid preference optimization. Its core idea is to introduce multiple sources of preference signals beyond traditional reinforcement learning from human feedback, such as RLHF. These sources include model self-evaluation, rule constraints, and GPT-4-assisted scoring, among other signals, to mitigate the reward hacking problem caused by a single reward model.

    Specifically, the preference alignment stage usually includes the following technical steps:

    1. Preference data construction: Generate multiple candidate responses for the same prompt, and have human annotators or advanced models rank and score them to form pairwise preference pairs.
    2. Reward model training: Train a reward model based on preference pairs to predict which response is better.
    3. Policy optimization: Use the reward model as a feedback signal to perform reinforcement learning or direct preference optimization (DPO) on the language model, gradually shifting its output distribution toward high-reward regions.

    It is important to emphasize that preference alignment is not a one-time task. After model deployment, user feedback still needs to be continuously collected and periodic iterative updates performed to address evolving usage scenarios and safety challenges.

    3. Data Engineering: The Hidden Champion of Multimodal Training

    If model architecture determines the upper bound of capability, data engineering determines whether the model can reach that upper bound. In multimodal model training, the core challenge of data engineering is unified representation and mixing strategy for heterogeneous data.

    Data Composition and Mixing Strategy

    A typical multimodal training dataset usually includes the following categories:

    Data category Description Typical ratio
    Image-text pairs Images plus textual descriptions 40%–50%
    Video-text Short video clips plus corresponding textual descriptions 10%–15%
    Pure text Used to prevent language capability degradation 20%–30%
    Interleaved image-text documents Web-page-level natural arrangements of images and text 15%–20%
    OCR/document images Scanned documents, tables, and screenshots 5%–10%

    Unlike pure text models, a subtle but important issue in multimodal model training is that too high an image-text ratio can cause language capability degradation, while too low a ratio makes it difficult to sufficiently learn visual-language mapping. A widely accepted industry rule of thumb is to keep the ratio between multimodal data and pure text data in the range of 3:1 to 1:1, with dynamic adjustments based on downstream task evaluation results during training.

    Dynamic Sampling and Curriculum Learning

    In addition to static data ratios, more training schemes are introducing dynamic data sampling strategies. The core idea is to use simple, high-signal-to-noise alignment data in the early stage of training to help the model quickly build basic capabilities. As training progresses, the sampling weight of difficult samples—such as long videos, multi-image reasoning, and complex charts—is gradually increased, creating a curriculum learning effect. This approach converges faster than uniform mixed training and also achieves better final convergence accuracy.

    4. Key Technical Details and Training Techniques

    Dynamic Resolution Processing

    Fixed-resolution image patching strategies struggle with the vastly different image sizes encountered in reality. The current mainstream approach uses a dynamic resolution mechanism: during training and inference, the model dynamically selects an appropriate grid partitioning method based on the input image’s aspect ratio, scales or pads when necessary, and finally merges visual features from different grids before feeding them uniformly into the language model. This method significantly reduces the computational overhead of high-resolution images while preserving detailed information.

    Modality Adaptation of Positional Encoding

    The standard positional encoding of language models is designed for text sequences, but visual tokens naturally have a two-dimensional spatial structure. Directly reusing one-dimensional positional encoding loses spatial relationship information. A commonly used improvement is multimodal rotary position embedding (M-ROPE), which decomposes positional information into horizontal, vertical, and temporal dimensions, computes rotation matrices separately, and adds them to attention computation. This design enables the model to perceive the up-down and left-right relationships of objects in images, as well as the temporal relationships between frames in videos, greatly improving spatial reasoning capability.

    Training Stability Assurance

    Multimodal model training is more unstable than pure text model training. A common problem is a mismatch in learning rates between the vision tower and the language model, causing one side’s parameters to update too quickly and破坏 the established alignment state. In practice, a layer-wise decoupled learning rate strategy is usually adopted: the visual encoder uses a lower learning rate, typically one-tenth to one-fifth of the language model learning rate, and the vision tower is frozen in the early stage of training. After the language model stabilizes, it is gradually unfrozen. In addition, gradient clipping and EMA (exponential moving average) are also common techniques for maintaining training stability.

    5. Engineering Considerations from Training to Deployment

    Completing model training is only the beginning of the entire pipeline. From an engineering perspective, performance optimization for inference services must also be considered. Multimodal models process both image and text inputs during inference, and image tokens usually occupy a large amount of KV cache space. One noteworthy optimization direction is dynamic visual token compression: adaptively reducing the number of visual tokens based on image content complexity—sampling fewer tokens for simple images and retaining more for complex scenes—to achieve a dynamic balance between inference speed and accuracy.

    In addition, deploying the visual encoder and language model separately is a common engineering practice. The visual encoder can run on GPU-accelerated nodes, complete feature extraction, and then pass compact visual feature vectors to language model nodes, thereby reducing cross-node bandwidth pressure.

    6. Summary and Future Outlook

    Multimodal model training has formed a complete methodology centered on “modality alignment pretraining → cross-modal instruction tuning → preference alignment optimization.” Each stage has its own unique technical challenges and engineering priorities, while data quality and mixing strategy run throughout and become key factors constraining the upper bound of model capability.

    Looking ahead, the boundaries of multimodal training continue to expand. Longer-context multimodal understanding, real-time video stream interaction, and unified multimodal generation and understanding architectures are all frontier directions worth exploring. For R&D teams, rather than chasing model scale expansion, it is more important to make steady progress on “underlying engineering” such as data quality, training stability, and alignment efficiency. After all, what determines a model’s real level is never just the number of parameters, but overall systems engineering capability.