Multimodal LLM Training: From Architecture Paradigms to Alignment in Practice

    This article explains the technical evolution of multimodal large language models (LLMs), from dual-encoder contrastive alignment to bridging architectures with learnable queries and native unified multimodal pretraining. It details the three-stage training pipeline—pretraining alignment, instruction fine-tuning, and preference alignment—and covers data engineering, evaluation, and practical engineering tips for building multimodal systems.

    Multimodal LLM Training: From Architecture Paradigms to Alignment in Practice

    1. Evolution of Technical Paradigms for Multimodal LLMs

    The core goal of multimodal large language models is to enable a single model to understand (and potentially generate) multiple modalities, such as images, video, and text. From a technical evolution perspective, the industry has roughly gone through three major paradigm shifts. Understanding the differences among these three approaches is a prerequisite for understanding multimodal training methods.

    1.1 Dual-Encoder Contrastive Alignment

    Early representative approaches used a two-tower structure with a text encoder and an image encoder, leveraging large-scale image-text pair contrastive learning to build a shared cross-modal semantic space. The training objective is to bring matched image-text pairs closer in the embedding space while pushing mismatched pairs apart. A typical loss function is InfoNCE:

    1. # Simplified contrastive loss
    2. logits = image_emb @ text_emb.T / temperature
    3. labels = torch.arange(batch_size)
    4. loss = (cross_entropy(logits, labels) +
    5. cross_entropy(logits.T, labels)) / 2

    This paradigm offers high training efficiency and strong retrieval capability, but it only achieves “understanding”-level alignment. It lacks text generation ability and cannot directly support generative tasks such as visual question answering or image captioning.

    1.2 Bridging Paradigm: Vision Encoder + Language Model

    The mainstream approach in the second phase is “frozen vision encoder + frozen (or lightly fine-tuned) LLM + trainable modality bridging module.” The vision encoder (usually a vision Transformer) extracts patch-level features from images; the language model handles generation and reasoning; and the bridging module in between is responsible for modality transformation and semantic alignment.

    Bridge module designs have evolved through several iterations:

    • Learnable queries + cross-attention: A fixed set of learnable vectors is introduced. These queries interact with visual features via cross-attention to “query” task-relevant information and compress it into a sequence length acceptable to the LLM.
    • Perceiver resampler: A simpler structure that replaces complex query mechanisms, reducing the parameter count and training difficulty of the alignment module.
    • Linear projection / MLP projection layer: Directly maps visual features into the LLM’s embedding space, combined with dynamic resolution mechanisms to adapt to images of various sizes and aspect ratios.

    1.3 Native Unified Multimodal Architecture

    The latest paradigm involves training the model on both text and visual tokens during the pretraining phase—i.e., “native multimodal pretraining.” This approach no longer relies on a two-stage process of “first train a language model, then align vision.” Instead, it mixes text and multimodal data (e.g., sampling text corpora, image-text, and video data in a certain ratio) under a unified autoregressive objective. The advantages include:

    • Avoiding the language capability degradation that can occur in post-alignment stages;
    • Text capability and multimodal capability improve synergistically;
    • More unified architecture that can be easily extended to arbitrary interleaved sequence modeling (e.g., mixed image-text sequences).

    2. Learnable Query Vectors: A Key Mechanism for Modality Alignment

    In the bridging paradigm, learnable query vectors are a classic entry point for understanding multimodal alignment. Their core functions are twofold.

    2.1 Modality Bridging: Adapting Sequence Length and Format

    The number of patch features output by the vision encoder varies with input resolution (e.g., high-resolution images may produce thousands of visual tokens), while the LLM’s input sequence length is limited. Learnable query vectors (typically ranging from a few dozen to a few hundred) actively “query” visual features via cross-attention, compressing them into a fixed-length sequence. This significantly reduces the token count sent to the LLM, balancing efficiency and information retention.

    2.2 Semantic Alignment: Driven by Loss Functions

    Query vectors are not used directly after random initialization; they are trained through multiple stages to align with the language semantic space:

    • Image-Text Contrastive loss (ITC): Aligns global semantics between visual and textual representations.
    • Image-Text Matching loss (ITM): Uses binary classification to determine whether an image and text match, learning fine-grained associations.
    • Image-Text Generation loss (ITG / LM Loss): Autoregressively generates text conditioned on visual input, directly linking to language model capabilities.
    1. # Simplified cross-attention: queries interact with visual features
    2. q = learnable_queries # [B, N_q, d]
    3. k, v = vision_proj(vit_feats) # [B, N_v, d]
    4. aligned = cross_attention(q, k, v) # [B, N_q, d]
    5. llm_inputs = llm_proj(aligned)

    From an engineering perspective, the design of query vectors is essentially a trade-off between “information compression rate” and “semantic preservation.” Too few queries lead to loss of detail (affecting OCR or fine-grained localization tasks); too many increase inference cost. This motivates later techniques such as dynamic resolution and dynamic visual token sampling, which adaptively decide the number and granularity of visual tokens based on input content.

    3. Complete Training Pipeline: A Three-Stage Process

    A typical multimodal LLM training pipeline can be broken down into three stages.

    3.1 Stage One: Pretraining Alignment

    This stage uses massive weakly aligned image-text pairs (e.g., alt-text data crawled from the web, video subtitle data). The main focus is training the bridging module; the vision encoder and LLM are usually frozen or only partially unfrozen. The goal is to establish basic “visual understanding” capability at the lowest cost. Data scale typically ranges from billions to hundreds of billions of multimodal tokens. Data cleaning—deduplication, filtering low-quality samples, and balancing data distribution—is extremely critical at this stage.

    3.2 Stage Two: Instruction Fine-Tuning (SFT)

    Using high-quality instruction data from tasks such as multi-turn visual QA, document understanding, chart reasoning, and video understanding, the model undergoes end-to-end fine-tuning (usually unfreezing parts of the vision encoder and LLM layers). This stage determines the model’s task generalization ability. Data quality matters far more than quantity. Common practices include building a high-quality instruction set covering multiple tasks, granularities, and languages, and running task ratio experiments.

    3.3 Stage Three: Preference Alignment

    In the post-training stage, human preference data is introduced. Preference optimization methods (e.g., contrastive training objectives on pairwise preference data) improve the usefulness, truthfulness, and safety of responses. In multimodal scenarios, special negative samples and preference data are designed to address the “hallucination” problem (e.g., describing objects that do not exist in the image). Some approaches also incorporate test-time augmentation strategies, where multiple sampling and aggregation during inference improve accuracy on complex reasoning tasks.

    4. Data Engineering and Evaluation Practice

    4.1 Data System

    • Pretraining data: Large-scale image-text pairs, interleaved image-text documents, video-subtitle pairs. Focus on coverage and scale.
    • Fine-tuning data: Task-oriented data such as OCR, chart understanding, document parsing, mathematical visual reasoning, multi-image comparison, and long-video QA. Focus on quality and diversity.
    • Preference data: Pairwise annotations for hallucinations, instruction following, and safety.

    4.2 Additional Challenges for Video

    Video understanding introduces a temporal dimension. Common approaches include: multimodal rotary position encoding (extending rotary position embeddings to three dimensions—time, height, and width) to model spatial-temporal relationships; streaming memory mechanisms to support incremental processing of long videos; and adaptive key-frame sampling to control token overhead.

    4.3 Evaluation Benchmarks

    Common capability dimensions include: general QA and reasoning (e.g., multidisciplinary multimodal understanding benchmarks), document and chart understanding, OCR recognition, video QA, image captioning, and retrieval. When evaluating, it is important to distinguish between “understanding-type” and “generation-type” tasks, and to supplement automatic metrics with human evaluation for hallucination rate and instruction-following quality.

    5. Engineering Recommendations for Deployment

    Based on common industry practices, consider the following when building a multimodal training system:

    1. Compute and memory planning: Multimodal training consumes significantly more memory than text-only training. Use activation checkpointing, ZeRO sharding, and mixed precision. Prioritize gradient unfreezing for vision encoders as needed.
    2. Build the data pipeline first: A data pipeline with deduplication, quality filtering, and modality ratio monitoring often yields greater gains than model architecture tweaks.
    3. Progressive unfreezing: First train the bridging module, then unfreeze the top layers of the LLM, and finally fine-tune all parameters. This significantly reduces training instability.
    4. If training on a cloud platform: Use object storage for managing multimodal corpora, container platforms for scheduling distributed training jobs, and logging and monitoring services to track loss curves and throughput metrics, reducing operational overhead.

    Overall, multimodal training fundamentally involves a continuous trade-off between “cross-modal semantic alignment” and “preservation of generative capability.” From contrastive learning to query-based bridging to native unified architectures and preference alignment, technical routes are converging toward simpler, more end-to-end solutions. Meanwhile, careful refinement of data quality, spatial-temporal modeling, and hallucination suppression is what currently differentiates state-of-the-art models.