Multimodal Model Training: A Complete Walkthrough from Paradigms to Evaluation

    Multimodal large models accept images, video, and text through a unified autoregressive interface, then emit answers, bounding boxes, or code as text. They are steadily evolving from task-specific models into general-purpose perception assistants. This article walks through the full training workflow: how to choose a modeling paradigm, the role of the Learnable Query bridge, a three-stage training recipe, data engineering, practical engineering concerns (memory, parallelism, and stability), and a closed evaluation loop. It is aimed at ML engineers who want a structured mental model for building and scaling multimodal training pipelines.

    Introduction: The Essence of Multimodal Training

    Multimodal large models typically receive images, videos, and text through a unified autoregressive interface, then output answers, localization boxes, or code as text. They are gradually evolving from single-task, purpose-built models into general-purpose perception assistants.

    The core of multimodal training is not “making a bigger model see more images.” It is solving three engineering problems:

    1. Semantic alignment: Mapping visual representations and language representations into the same semantic space, so that “cat,” a word, and a picture of a cat end up with similar representations.
    2. Task generalization: Enabling a single model to handle heterogeneous tasks at once, including image-text retrieval, document understanding, visual question answering, video temporal localization, and image segmentation.
    3. Scalable training: Matching data scale, visual token budget, and compute supply while keeping the training process stable and reproducible.

    This article covers, in order: paradigm selection, the modal-bridging component Learnable Query, a three-stage training recipe, data engineering, engineering implementation, and a closed evaluation loop.

    1. The Evolution of Mainstream Multimodal Paradigms

    Multimodal model paradigms are moving from “dual-encoder alignment” toward “end-to-end unified modeling.” Representative directions include:

    • Bridge-based multimodal models: Open-source models that support multi-image input and high-resolution processing. They replace the earlier query-based Transformer with a cross-attention resampling structure and adopt a unified autoregressive loss, improving Top-1 recall on image-text retrieval by roughly 5% over the previous generation and supporting training at the scale of tens of billions of multimodal tokens.
    • Dynamic-resolution video multimodal models: Designed for long-video understanding (20+ minutes) and multilingual capability. They replace “tile then force-resize” with a “naive dynamic resolution” approach and use Multimodal Rotary Position Embedding (M-RoPE) to unify 2D image positions and 1D temporal positions. They surpass some flagship closed-source models on document-understanding benchmarks, with video-QA accuracy improving about 27% over the previous generation.
    • Native multimodal pretraining models: Text and multimodal data are jointly pretrained with a roughly 1:3 sampling ratio. Combined with mixed preference optimization and test-time scaling (such as multi-sample ensembling and chain-of-thought guidance), they reach 72.2 on a composite multimodal benchmark, close to the closed-source models of the same period.
    • Streaming video segmentation models: They introduce a streaming memory mechanism and a hierarchical visual encoder, reducing the number of interactions by 67% and improving inference speed about 6x, enabling real-time processing of tens of frames per second.
    • Multimodal Mixture-of-Experts (MoE) models: They adopt an MoE architecture with dynamic visual token sampling, supporting multimodal generation and ultra-long context within a unified framework, and improving zero-shot image segmentation mean IoU (mIoU) by about 0.9 percentage points over the previous generation.

    The figures above come from public technical reports and will fluctuate with different evaluation setups. Their value lies in revealing a consistent pattern: structural improvements deliver steady gains.

    Three evolution threads emerge:

    1. Structure: Dual-tower contrastive learning → query bridging (Query Transformer, Perceiver-based resampling, dynamic projection) → native unification (unified token sequence, early fusion).
    2. Objective: A single contrastive loss → autoregressive language-modeling loss → a combination of autoregressive and preference optimization, plus unified generation over discrete visual tokens.
    3. Data: Small, high-quality image-text pairs → internet-scale weakly labeled data → rewritten high-quality captions → trillions of tokens of mixed corpus spanning documents, charts, GUIs, video, and audio.

    1.1 Paradigm Comparison

    Dimension Dual-tower contrastive Query-bridged Native unified
    Representative structure Independent vision tower + independent text tower sharing an embedding space Frozen vision encoder + learnable queries + language model Unified token sequence; text and visual data pretrained jointly
    Training objective Contrastive loss (InfoNCE) Autoregressive next-token prediction Unified autoregressive modeling, optionally with discrete visual generation
    Advantages Simple to train; efficient retrieval Reuses mature language models; low training cost Language and multimodal abilities reinforce each other; no alignment bottleneck
    Limitations No generation ability; weak fine-grained understanding Visual information bottleneck; depends on two-stage training High pretraining cost; heavy memory pressure for long sequences

    Dual-tower contrastive is the starting point for a cross-modal semantic space: positive pairs are pulled together and in-batch negative pairs are pushed apart. Its biggest advantage is that at inference time each modality’s embeddings can be precomputed and then matched at scale via vector retrieval. The limitation is equally clear: with no autoregressive generation ability, it can only perform classification or matching and struggles with long-form reasoning.

    Query-bridged is the current industrial mainstream: the vision encoder and language model each keep their mature form, with a learnable modal-bridging module (Learnable Query) inserted in between. It inherits the semantic-space advantage of the dual-tower approach while restoring generation ability through autoregressive prediction over “visual tokens + text tokens.”

    Native unified interleaves multimodal data into a single token sequence during pretraining, so text and visual features are learned together. It sidesteps the two-stage “align first, then fine-tune” bottleneck and lets pure-language and multimodal abilities improve in tandem, but it demands more pretraining compute, careful data mixing, and strong long-sequence parallelism.

    1.2 Visual Input Processing Strategies

    • Global scaling: Resize each image to a fixed size, then split into patches. Simple to implement, but small-text OCR and dense charts suffer heavy information loss.
    • Tile + global view: Split a high-resolution image into multiple sub-images, encode each independently, then concatenate with thumbnail features. Well suited to documents and GUI screenshots.
    • Dynamic resolution: Adaptively divide a grid according to the original aspect ratio, avoiding the geometric distortion caused by “force-resize + padding,” while setting upper and lower bounds for extreme aspect ratios.
    • Frame sampling and token compression: Sample video by keyframes or uniform intervals, then compress redundancy through techniques such as adjacent-frame differencing and temporal pooling.

    2. Learnable Query: Mechanism and Technical Evolution

    2.1 What Problem Does It Solve?

    Learnable Query is a set of trainable vectors that act as a “learnable intermediary” between vision and language, serving four roles.

    1. Modal bridging: The patch-feature sequence from a vision encoder grows with resolution (a 1024×1024 image can yield hundreds to over a thousand tokens), and feeding it directly into a language model quickly consumes the context budget. Learnable Query uses cross-attention to compress it into a fixed length (typically 16–256) of “soft visual tokens.”
    2. Information bottleneck and semantic selection: The query vectors extract semantically relevant content from the visual features via attention, suppressing background and noise and acting like “retrieval on demand.”
    3. Structural decoupling: The vision tower and language tower can be upgraded independently, with the bridging layer adapting to new models through parameter-efficient fine-tuning and avoiding full retraining.
    4. Interface standardization: Whether the input is image, video, or audio, it can be mapped into the same visual-token slots, so the downstream language model does not need to be aware of modality differences.

    2.2 Why “Learnable” Beats Fixed Pooling

    Fixed average pooling keeps only global statistics and loses spatial relationships; a fixed linear projection only changes dimensionality, not semantics. Learnable queries have data-driven attention weights that allocate attention across different semantic granularities: focusing on layout and text blocks for documents, and on the main subject region for natural images. This is effectively bringing “feature selection” into end-to-end optimization.

    1. import math
    2. import torch
    3. import torch.nn as nn
    4. import torch.nn.functional as F
    5. class LearnableQueryResampler(nn.Module):
    6. """Compress visual features of arbitrary length into a fixed number K of soft visual tokens.
    7. Core idea: K learnable queries q serve as the query,
    8. while image features serve as key/value. Cross-attention performs
    9. information aggregation. The output sequence length is always K,
    10. decoupled from the input resolution.
    11. """
    12. def __init__(self, hidden_size=1024, num_queries=64,
    13. num_heads=16, num_layers=2):
    14. super().__init__()
    15. self.num_heads = num_heads
    16. self.head_dim = hidden_size // num_heads
    17. # Learnable queries: shape [1, K, D], jointly optimized with the main model during training
    18. self.query = nn.Parameter(torch.randn(1, num_queries, hidden_size) * 0.02)
    19. self.kv_proj = nn.Linear(hidden_size, hidden_size)
    20. self.q_proj = nn.Linear(hidden_size, hidden_size)
    21. self.o_proj = nn.Linear(hidden_size, hidden_size)
    22. self.ffn = nn.Sequential(
    23. nn.Linear(hidden_size, 4 * hidden_size), nn.GELU(),
    24. nn.Linear(4 * hidden_size, hidden_size)
    25. )
    26. self.norm_q = nn.LayerNorm(hidden_size)
    27. self.norm_kv = nn.LayerNorm(hidden_size)
    28. self.norm_out = nn.LayerNorm(hidden_size)
    29. self.num_layers = num_layers
    30. def _attn_mask(self, pad_mask):
    31. # pad_mask: [B, N], True means this visual token is padding and must not be attended to
    32. if pad_mask is None:
    33. return None
    34. return (~pad_mask)[:, None, None, :] # Broadcast to [B, H, K, N]
    35. def _split_heads(self, x):
    36. B, L, D = x.shape
    37. return x.view(B, L, self.num_heads, self.head_dim).transpose(1, 2)
    38. def forward(self, vision_feats, pad_mask=None):
    39. # vision_feats: [B, N, D], where N varies dynamically with image resolution or number of video frames
    40. B = vision_feats.size(0)
    41. kv = self.norm_kv(self.kv_proj(vision_feats))
    42. k, v = self._split_heads(kv), self._split_heads(kv)
    43. q = self.norm_q(self.query.expand(B, -1, -1))
    44. q = self._split_heads(q)
    45. mask = self._attn_mask(pad_mask)
    46. for _ in range(self.num_layers):
    47. q = self.q_proj(
    48. F.scaled_dot_product_attention(q, k, v, attn_mask=mask)
    49. ).view(B, -1, q.size(-1) * self.num_heads)
    50. q = self.norm_out(q + self.ffn(q))
    51. return q # [B, K, D], K is fixed and independent of input resolution

    Engineering practices to keep in mind:

    • Number of queries: Too few loses detail; too many weakens the compression benefit. Common values are tuned together with the language model’s hidden dimension and the context budget.
    • Layers and parallelism: Multi-layer resampling offers stronger nonlinear expressiveness than a single layer but adds latency; when using many layers, tensor parallelism can be applied along the query dimension.
    • Working with positional encodings: Queries carry no positional prior, so the input side must retain 2D positional information (or switch to a rotation-based positional-encoding variant that models 2D positions), otherwise spatial relationships get “pooled away.”

    2.3 Technical Evolution: From Query Transformer to Zero Bridging

    • First generation: query-based Transformer. A set of learnable queries fuses frozen visual features through multiple cross-attention layers; a first-stage image-text contrastive objective handles alignment, after which a frozen language model is attached for second-stage generation-capability injection. The representative structure is the BLIP-2 series. Its drawback is a persistent “representation gap” between the queries and the language model.
    • Second generation: Perceiver-style resampling. A unified cross-attention layer replaces the self-attention in the Query Transformer, compressing an arbitrary-length sequence into a fixed length. The structure is simpler, uses fewer parameters, and naturally accommodates multi-image input. Representative structures are the BLIP-3 / XGen-MM series.
    • Third generation: dynamic projection and multimodal positional encoding. A lightweight MLP or 2D adaptive pooling projects patch features directly into the language model space, combined with “naive dynamic resolution” and “Multimodal Rotary Position Embedding” (M-RoPE) to unify 2D image positions, 3D video positions, and 1D text positions within a single sequence. The representative structure is the Qwen2-VL series.
    • Fourth generation: native unified architecture. Explicit bridging is removed; text and vision are interleaved into a unified token sequence from the pretraining stage, and “dynamic visual token sampling” can shrink the visual budget at inference time as the task requires. Representative directions include InternVL3, Emu3, and Llama 4.

    The evolution logic is clear: the bridging layer gets thinner, while positional encoding and resolution modeling get heavier. When pretraining data is abundant enough, the expensive explicit alignment modules are replaced by simpler projections plus strong positional encodings.

    3. Breaking Down the Three-Stage Training Recipe

    Taking current mainstream bridge-based and native-unified approaches as examples, training is typically divided into three stages.

    3.1 Stage 1: Cross-Modal Alignment Pretraining

    Goal: Under frozen or partially frozen backbones, teach the bridging module to “see.”

    • Data: Large-scale weakly labeled image-text pairs (web alt-text, stock-photo captions), on the order of hundreds of millions to billions of pairs; can be mixed with image-text retrieval and image captioning tasks.
    • Strategy: Fully freeze the vision encoder (or use a very low learning rate); freeze the language model or train only LoRA-style low-rank adapters; train only the projection and resampler. This stage has few parameters, so a larger batch and higher learning rate (1e-4 to 1e-3) can be used.
    • Optional objective: Add a contrastive loss on top of the autoregressive loss, using the contrastive objective to constrain the global consistency between visual tokens and text representations.
    1. # Contrastive loss (usable by both dual-tower and bridge-based designs)
    2. def contrastive_loss(img_emb, txt_emb, temperature=0.07):
    3. img_emb = F.normalize(img_emb, dim=-1)
    4. txt_emb = F.normalize(txt_emb, dim=-1)
    5. logits = img_emb @ txt_emb.t() / temperature # [B, B]
    6. labels = torch.arange(img_emb.size(0), device=img_emb.device)
    7. return 0.5 * (F.cross_entropy(logits, labels) +
    8. F.cross_entropy(logits.t(), labels))

    3.2 Stage 2: Visual Instruction Fine-Tuning

    Goal: Convert “alignment ability” into “task ability.”

    • Data: Visual instruction datasets covering image captioning, visual question answering, OCR and document understanding, chart understanding, GUI operation understanding, video QA and video temporal localization, and multi-image comparison; single-sample length ranges from a few hundred to tens of thousands of tokens.
    • Strategy: Unfreeze the language model (full parameters or efficient fine-tuning); the vision tower can be partially unfrozen slightly (e.g., only the last few layers, or grouped under a lower learning rate). Drop the learning rate to around 1e-5 to 2e-5, with warmup and cosine decay.
    • Key techniques:
      • Curriculum data ratios: Start with a high proportion of natural-image tasks, then gradually add long-tail tasks such as documents, charts, and video, to avoid being “pulled off course” by OCR tasks early on.
      • Tiered visual token budget: Limit the per-image token count early in training (e.g., ≤256) to allow a larger batch; later relax it to 1024 or more to improve fine-grained capability.
      • Mixed packing of single/multi-image/video: Allow multiple <image> placeholders within the same sequence; positional encodings must support multi-image offsets.

    3.3 Stage 3: Preference Optimization and Post-Training Enhancement

    Goal: Align with human preferences, suppress hallucination, and improve instruction following and reasoning-chain quality.

    • Data: Multiple candidate responses per prompt (good/bad pairs); preference labels come from human annotation, scoring by a stronger model, or rule-based verification (such as whether a math answer is correct or whether code passes unit tests).
    • Objective function: Add a preference-optimization term on top of supervised fine-tuning.
    1. # DPO-style preference loss (illustrative): πθ is the current policy, πref is the reference model
    2. def dpo_loss(policy_logp_win, policy_logp_lose,
    3. ref_logp_win, ref_logp_lose, beta=0.1):
    4. logits = (policy_logp_win - policy_logp_lose) \
    5. - (ref_logp_win - ref_logp_lose)
    6. return -F.logsigmoid(beta * logits).mean()
    • Mixed preference optimization: Combine the SFT loss, the preference loss, and a negative-example regularizer with weights, to mitigate the problem of “excessive preference optimization collapsing generation diversity.”
    1. # Mixed preference optimization (MPO-style): SFT + preference + negative-example penalty
    2. total_loss = w_sft * sft_loss \
    3. + w_pref * preference_loss \
    4. + w_neg * neg_logp_penalty
    • Test-time scaling: At inference, use techniques such as multi-sample voting, chain-of-thought guidance, and “visual re-look” (re-cropping and re-encoding key regions), trading inference cost for accuracy. A common approach is to combine “convergence at training time” with “scaling at inference time.”

    4. Data Engineering: The Key Factor That Sets the Ceiling

    4.1 The Data Processing Pipeline

    1. Collection: Image-text pairs, PDF/HTML documents, screen captures, video + subtitles, and audio transcripts.
    2. Cleaning: Deduplication (image perceptual hashing + text MinHash), filtering of blurry/low-resolution content, filtering of watermarked and ad-heavy layouts, and filtering of harmful content.
    3. Re-annotation: Use a stronger model to rewrite weakly labeled captions, adding structured information such as objects, attributes, spatial relationships, and text content, which significantly improves instruction fine-tuning quality.
    4. Ratios and curriculum: A text-to-multimodal ratio of about 1:3 is a common mix for native multimodal pretraining today; during the instruction stage, sample dynamically by task difficulty and long-tail distribution.
    5. Packing and splitting: Use sequence packing to reduce padding waste while keeping different samples isolated from each other with attention masks.

    4.2 A Sampler Example

    1. import random
    2. class MixedModalitySampler:
    3. """Mix different modality sources by preset ratios, preventing long-tail tasks from being drowned out by mainstream data."""
    4. def __init__(self, sources: dict, seed: int = 42):
    5. # sources: {"text": idx_list, "image": idx_list, "video": idx_list}
    6. self.sources = sources
    7. self.rng = random.Random(seed)
    8. def sample(self, ratios: dict, n: int):
    9. names, weights = zip(*ratios.items())
    10. out = []
    11. for _ in range(n):
    12. src = self.rng.choices(names, weights=weights, k=1)[0]
    13. pool = self.sources[src]
    14. out.append(self.rng.choice(pool)) # A real implementation should sample without replacement and shuffle per epoch
    15. return out

    5. Engineering Implementation: Memory, Parallelism, and Stability

    5.1 Memory and Parallelism Strategies

    • Memory budget breakdown: Parameters, gradients, optimizer states (Adam is about 2x the parameters), and activations. For long sequences, activations are often the real bottleneck.
    • Combining parallelism:
      • Data parallelism + optimizer-state sharding (ZeRO-1/2);
      • Tensor parallelism to split attention and FFN;
      • Pipeline parallelism to split language model layers;
      • Sequence parallelism to handle ultra-long multimodal sequences;
      • Context parallelism to support million-level tokens.
    • Activation recomputation: Apply selective recomputation to the vision and language towers, trading compute for memory.
    • Mixed precision: Use bf16 as the primary format, keep fp32 accumulation for sensitive operators (softmax, LayerNorm), and use low-precision attention kernels when possible.

    5.2 Training Stability

    • Gradient clipping: Occasional loss spikes in multimodal training usually come from anomalous samples; clip grad-norm to 1.0 and pair it with anomaly detection.
    • Learning rate grouping: Use different learning rates for the vision tower, bridging layer, and language model (e.g., the vision tower at 0.1x the language model’s rate).
    • Data-side filtering: Strong image-text relevance filtering (e.g., removing mismatched samples with CLIP-style scoring) can significantly reduce early-training oscillation.
    • Checkpoints and fault tolerance: Save checkpoints at fixed steps, support resuming from checkpoints and automatic rollback; make training jobs stateless so they can run on preemptible compute.

    5.3 Platforms and Operations (General Capability Categories)

    Large-scale multimodal training usually relies on a combination of cloud-platform capabilities:

    • Container platform: Hosts distributed training jobs alongside GPU resource pooling and queue scheduling;
    • Object storage: Stores massive image-text pairs, video shards, and checkpoints;
    • Message queues / job scheduling: Decouple data preprocessing from the training consumers and support retry on failure;
    • Logging service: Centrally collects training logs and loss curves;
    • Monitoring and alerting: Provides real-time alerts on GPU utilization, memory, throughput (tokens/s), and loss anomalies.

    These are general infrastructure capability categories common across the industry; specific choices should be evaluated against team size and compliance requirements.

    6. Evaluation and Closed-Loop Iteration

    1. Layered evaluation: General multimodal understanding (a composite multimodal benchmark), documents and OCR, chart understanding, video understanding, multi-image/long-video, and pure-language regression.
    2. Regression guard: Multimodal training can easily damage language ability, so run pure-text benchmarks (general knowledge, math, code) on every training run and set a degradation threshold.
    3. Hallucination-specific testing: Build a counter-questioning evaluation set for “objects that do not exist in the image” and track the misanswer rate.
    4. Efficiency metrics: tokens/s, per-GPU throughput, peak memory, and inference latency and visual token overhead.
    5. Closed loop: Online bad cases flow back → re-annotation → ratio adjustment → incremental training, forming a data flywheel.

    7. Common Problems and a Troubleshooting Checklist

    Symptom Likely cause Investigation direction
    Loss drops but generations miss the point The bridging layer is underfitting; visual information has not entered the language space Increase resampling layers/number of queries; fully complete Stage 1 alignment first
    Poor OCR and fine-grained ability Visual tokens are over-compressed or force-resized Increase resolution, relax the visual token budget, add a tile view
    Loss spikes mid-training Anomalous samples; learning rate too high Relevance filtering, gradient clipping, roll back to a checkpoint
    Language ability degrades Multimodal data crowds out the text distribution Adjust the text-to-multimodal ratio; add a text regression set
    Poor long-video understanding Sparse frame sampling; missing temporal positional encoding Add keyframe retrieval; use a positional encoding that models time

    Conclusion

    The methodology for multimodal model training is converging: subtract on structure (thinner bridges, stronger positional encodings), add on data (more precise ratios, deeper re-annotation), and multiply on process (three-stage recipe + preference optimization + test-time scaling).

    For engineering teams, the recommended priority is: first build a solid data pipeline and evaluation loop, then decide on the paradigm — because the gains from paradigm differences are usually smaller than the gains from data quality and ratio optimization. Bridge components like Learnable Query still hold value, especially when compute is limited and you need to quickly reuse mature language models; native unified architectures are the long-term direction and are worth planning ahead for when compute allows.