Multimodal LLM Training Pipeline: From Data to Alignment
This article breaks down the end-to-end training pipeline for multimodal large language models, covering architecture, visual input processing, modality connectors, data construction and ratios, staged training, preference optimization, troubleshooting, and compliance and resource planning. Readers will learn how visual information is converted into language-model tokens, where common failures occur, and what to check when training or debugging a multimodal model.
Multimodal LLM Training Pipeline: From Data to Alignment
Multimodal large model training is, in essence, teaching a language model that only handles text to understand images, videos, and documents, and to convert visual information into semantic representations that can be used for reasoning and dialogue. This work involves model architecture, data construction, training objectives, and compute resources at the same time. If any part fails, the final result will suffer.
This article explains the training pipeline step by step. The content is organized as follows:
- Why multimodal training is harder than text-only training
- Overall architecture and forward pass
- Visual input processing: resolution, frame count, and token count
- Modality connectors: projection layers and resamplers
- Data construction and mixing ratios
- Staged training
- Preference optimization
- Common problems and troubleshooting checklist
- Data compliance, privacy, and resource budgeting
1. Why Multimodal Training Is Harder Than Text-Only Training
A text-only model receives discrete tokens as input, and sequence length is roughly proportional to the amount of text. Images are different. After visual encoding, one image produces a large number of tokens, and video multiplies that number further. This creates three direct problems.
- Modality heterogeneity: Visual features are continuous dense vectors, while text consists of discrete symbols. Their distributions differ greatly. Direct concatenation usually performs poorly and requires a dedicated alignment mechanism.
- High compute cost: Visual tokens consume context length, and attention computation grows rapidly with sequence length.
- Data asymmetry: High-quality image-text paired data is far scarcer than pure text. Tasks such as reading tables, understanding UI screenshots, and analyzing charts require specially constructed data.
2. Overall Architecture and Forward Pass
Mainstream multimodal models usually consist of three parts.
- Vision encoder: Most use a Vision Transformer (ViT) structure, split an image into patches, and output a set of visual features.
- Modality connector (projector or resampler): Maps visual features into the language model’s embedding space and can compress the number of visual tokens.
- Language model backbone: Receives concatenated visual tokens and text tokens, and generates answers autoregressively.
Visual tokens refer to visual representations that enter the language model and share the same sequence as text tokens. The output of the connector is the visual token sequence.
The following code is illustrative. vision_encoder, connector, and llm are placeholders and should be replaced with implementations from the specific framework.
# Illustrative code: replace placeholders with implementations from the specific frameworkimage_feats = vision_encoder(pixel_values) # pixel_values: [N_img, 3, H, W] -> [N_img, N_patch, D_v]visual_tokens = connector(image_feats) # -> [N_img, N_vis, D_llm]; N_vis is usually smaller than N_patchtext_embeds = llm.embed(input_ids) # input_ids: [B, L] -> [B, L, D_llm]# Replace tokens at placeholder IMAGE_TOKEN_ID with visual tokens; sequence length changes from L to L'inputs_embeds, attn_mask, labels = insert_visual(text_embeds, visual_tokens, placeholder_id=IMAGE_TOKEN_ID, labels=labels)# Set labels at visual positions to -100 so they are excluded from the losslogits = llm(inputs_embeds=inputs_embeds, attention_mask=attn_mask).logits # [B, L', vocab]loss = cross_entropy(logits[:, :-1], labels[:, 1:], ignore_index=-100)
Several implementation points:
- The number of placeholders must match the number of visual tokens. If the counts do not match, images may be silently dropped or misaligned. This is one of the most common hidden errors.
- Multiple images and interleaved image-text content: Each image must have its own placeholder and position record to avoid mixing tokens from different images.
- Attention mask: The text portion usually uses a causal mask. Within image tokens, bidirectional attention can be used because image regions do not have a temporal order. Whether to use bidirectional attention depends on the specific model design; consult its implementation documentation.
3. Visual Input Processing: Resolution, Frame Count, and Token Count
The number of visual tokens is determined by input resolution, patch size, spatial merge ratio, and frame count. For a single image, it can be estimated as:
N_vis ≈ (H / p) × (W / p) / r²
where H and W are the input height and width, p is the patch side length, and r is the side length of adjacent patch merging (for example, r=2 for 2×2 merging).
For an image with patch size 14 and resolution 448×448:
- Patching produces 32×32 = 1024 patches.
- After 2×2 merging, the result is 256 visual tokens.
If the side length doubles, the area becomes 4 times larger. An 896×896 image produces 64×64 = 4096 patches, which become 1024 visual tokens after merging. Note that when the side length doubles, the token count increases by about 4 times. If only one dimension increases, the increase is smaller.
Video is more expensive. Suppose each video segment samples 8 frames with 256 visual tokens per frame, for a total of 2048 visual tokens. If 16 frames are sampled, the total increases to 4096.
The trade-offs of different processing strategies are as follows:
| Strategy | Token count | Training cost | Suitable scenarios |
|---|---|---|---|
| Low resolution, fixed size | Low | Low | Image captioning, coarse-grained question answering |
| High resolution, fixed size | High | High | Small-text recognition, detailed question answering |
| Dynamic resolution, split by aspect ratio | Varies by image | Medium to high | Documents, screenshots, tables |
| Uniform video frame sampling | Increases linearly with frame count | Medium to high | Action and event understanding |
| Video keyframes or inter-frame compression | Relatively low | Medium | Long-video overview |
In practice, first determine the minimum readable resolution required by the task. For documents and screenshots with very small text, if resolution is insufficient, the model will misread characters directly, and more training data may not compensate.
4. Modality Connectors: Projection Layers and Resamplers
The connector converts visual features into a form the language model can receive. There are two common designs.
| Type | Structure | Advantages | Disadvantages |
|---|---|---|---|
| MLP projection layer | Two linear layers plus an activation function | Simple to implement, stable training, preserves all visual information | Does not reduce token count; long-sequence overhead is high |
| Resampler (Q-Former style) | A set of learnable query vectors reads visual features through cross-attention | Fixed output token count, compresses visual information | More complex structure; compression may lose details |
If the task depends on fine-grained information, such as documents or tables, an MLP projection layer is usually more reliable. If the context budget is tight and the task focuses on image summarization, a resampler is more suitable. Whichever option is chosen, record the output token count and check during evaluation whether compression has affected detail-oriented questions.
5. Data Construction and Mixing Ratios
Multimodal training data generally falls into the following categories:
- Image-text paired data: Images with short descriptions, used to learn basic visual-language correspondence.
- Interleaved image-text documents: Articles or web pages with images and text interleaved, used to learn multi-image context.
- OCR and document data: Images containing text, used for text reading.
- Chart and table data: Used to read values and understand axes and row-column structure.
- Visual instruction data: Question answering, reasoning, and explanation around images, used for dialogue capability.
- Pure text data: Used to preserve the language model’s existing reasoning and writing abilities.
The ratios below are only a starting example, not the recipe of any published model:
| Data category | Example ratio | Purpose |
|---|---|---|
| Image-text pairs | 30% | Establish basic alignment |
| Interleaved image-text documents | 15% | Multi-image and context understanding |
| OCR and documents | 20% | Text reading |
| Charts and tables | 10% | Structured information understanding |
| Visual instructions | 15% | Dialogue and reasoning |
| Pure text | 10% | Preserve language capability |
The ratio needs to be adjusted based on evaluation results. If document-related questions perform poorly, increase the proportion of OCR and document data. If the model starts to lose writing or reasoning ability, increase pure text data appropriately. After each adjustment, use a fixed evaluation set and compare before and after results rather than changing things by feel.
Data deduplication is also important. Large numbers of repeated image-text pairs can cause the model to memorize specific samples and reduce generalization.
6. Staged Training
Multimodal models are usually trained in stages, with a different objective at each stage:
- Alignment stage: Freeze the vision encoder and language model backbone, train only the connector, and use image-text paired data. The goal is to make visual features understandable to the language model.
- Instruction fine-tuning stage: Unfreeze the language model backbone and unfreeze the connector as needed, using visual instruction data and pure text data. The goal is to teach the model to answer according to instructions.
- High-resolution or long-context stage: Increase input resolution or frame count and use documents, long videos, and other data to specifically improve detail recognition.
Learning rates should also be considered separately. The connector is a newly initialized module and can use a relatively higher learning rate. The vision encoder already has pretrained capability, so its learning rate is usually an order of magnitude lower; otherwise, it is easy to damage learned visual features. Specific values need to be determined experimentally based on model scale and data volume.
Before each stage starts, it is recommended to first run a small-scale pass to check whether the loss decreases and whether visual input actually takes effect. For example, input the correct image and a blank image for the same prompt, and compare the outputs. If the two are almost identical, the model is not using visual information.
7. Preference Optimization
After instruction fine-tuning, a common problem is that the model fabricates details about image content, which is hallucination. Preference optimization is used to reduce this problem.
One commonly used method is Direct Preference Optimization (DPO). For the same image and the same question, the model generates two answers: one preferred answer and one worse answer. The training objective is to increase the probability of the preferred answer relative to a reference model while decreasing the probability of the worse answer. The loss function can be written as:
L = −log σ( β · [ (log π(y_w|x) − log π_ref(y_w|x)) − (log π(y_l|x) − log π_ref(y_l|x)) ] )
where y_w is the preferred answer, y_l is the worse answer, π is the current model, π_ref is the reference model, β controls how far the model can deviate from the reference model, and σ is the sigmoid function.
In the multimodal setting, the key to preference data is constructing errors that have visual grounding. The worse answer should contain real visual errors, such as counting the wrong number of objects, misreading text, or inventing objects that do not exist in the image. Teaching the model only about stylistic differences in text has limited effect on hallucination.
8. Common Problems and Troubleshooting Checklist
| Symptom | Possible cause | Troubleshooting method |
|---|---|---|
| Model answers are unrelated to the image | Placeholders are not replaced; visual tokens do not enter the sequence | Check whether the number of placeholders matches the number of visual tokens; run a blank-image comparison test |
| Loss does not decrease | Learning rate is too high; vision encoder is updated too aggressively; labels for visual positions are not masked correctly | Check learning-rate groups and confirm that labels at visual positions are -100 |
| GPU memory overflow | Resolution or frame count is too high; batch size is too large | Count visual tokens per sample, reduce resolution or frame count, enable gradient checkpointing |
| Text is read incorrectly | Insufficient resolution; low proportion of OCR-style data | Compare outputs for the same image at different resolutions; add document data |
| Noticeable hallucination | Insufficient preference data; descriptions in instruction data are too loose | Measure the proportion of hallucination samples; add preference pairs with visual grounding |
| Pure-text ability drops after training | Proportion of pure text data is too low | Compare results before and after training on a fixed pure-text evaluation set; increase the pure-text share |
| Multi-image question answering is confused | Image position records are wrong; images are not distinguished from one another | Check the placeholder index for each image; test with two visually similar images |
During troubleshooting, change only one factor at a time and keep the configuration and evaluation results for each experiment. Otherwise, it is difficult to tell which change actually had an effect.
9. Data Compliance, Privacy, and Resource Budgeting
Data sources must be authorized first. Images and documents crawled from the web may be protected by copyright. Before use, verify the license agreement and record the data source. For content containing personal information such as faces, license plates, ID numbers, or addresses, perform de-identification before training, for example by blurring or removing the information. When user-uploaded content is involved, clarify the scope of data use and comply with relevant privacy regulations.
The resource budget should be estimated before starting. GPU memory is mainly consumed by model parameters, optimizer states, activations, and visual tokens. The number of visual tokens directly affects the size of activations, so resolution and frame-count settings must be considered together with the memory budget. Training time can be roughly estimated as “number of samples × tokens per sample × number of training epochs,” then corrected based on actual throughput.
Before scaling up, validate the data pipeline and compute budget with a small pilot run.
