How Vision-Language Models Learn to Reason: A Look Inside the Training Framework

    This article breaks down how modern vision-language models (VLMs) develop reasoning abilities rather than just perception. It walks through the full training pipeline—model architecture, data preparation, pre-training, supervised fine-tuning, and reinforcement learning—and explains how techniques like Chain-of-Thought, high-quality data, precise rewards, and curriculum sampling work together. Readers will gain a practical understanding of why reasoning is a system-level outcome and how to design a general-purpose multimodal training framework.

    1. Background: The Capability Gap Between Perception and Reasoning

    Over the past few years, vision-language models have evolved from “recognition” to “understanding” and now to “reasoning.” Early models focused primarily on perception—deciding whether a cat is present in an image. This later expanded to understanding—answering why the person in a picture looks happy. However, the vast majority of existing VLMs still stop at these first two levels: they excel at pattern-matching image-text alignment but lack the ability to break down problems and reason step by step in complex scenarios.

    The root of this gap lies in the training paradigm. Pure language models achieve strong reasoning largely thanks to two key techniques:

    • Chain-of-Thought (CoT): guiding the model to produce intermediate reasoning steps instead of jumping straight to an answer, breaking a complex problem into verifiable sub-steps.
    • Reinforcement Learning (RL): using task outcomes as reward signals to encourage the model to explore better reasoning paths rather than merely imitating labeled answers.

    The challenge is that porting these two techniques to the multimodal domain is not trivial. Visual input introduces additional cross-modal alignment problems, and high-quality “vision + reasoning chain” data is extremely scarce. This is precisely the core problem that a reasoning-centric training framework aims to solve.

    2. Model Architecture: A Simple, Scalable Backbone

    Most mainstream VLMs today adopt the classic three-part ViT-MLP-LLM structure. Its advantage lies in modular decoupling and scalability:

    1. Vision Encoder (ViT): splits the input image into patches and encodes them into a sequence of visual tokens, handling low-level visual feature extraction.
    2. Projection Layer (MLP): maps visual features into the language model’s embedding space to achieve cross-modal alignment. Compared with more complex cross-attention designs, a lightweight MLP often balances efficiency and effectiveness well.
    3. Language Model (LLM): takes the concatenated visual and text tokens and performs understanding, reasoning, and generation.

    The engineering value of this architecture is that the vision encoder and the language model can be selected and upgraded independently, while the projection layer acts as a lightweight “adapter” responsible for alignment. For video input, multiple frames are typically sampled and flattened into a token sequence, allowing the same architecture to be reused along the temporal dimension.

    3. Data Preparation: The Real Foundation of Reasoning

    The ceiling of a model’s reasoning ability is largely determined by its data. A mature training data pipeline usually covers the following dimensions:

    • Image caption data: provides the foundation for image-text semantic alignment and serves as the basic fuel for perception.
    • Interleaved image-text data: organizes images and text passages in reading order, helping the model learn to locate and reference visual information within long contexts.
    • Optical Character Recognition (OCR) data: covers documents, scene text, chart labels, and more, strengthening fine-grained text understanding and structured extraction.
    • Visual grounding data: binds text descriptions to specific image regions (such as bounding boxes), training the model’s ability to establish precise spatial correspondence.
    • Video data: through frame sampling and temporal-relationship modeling, gives the model an understanding of dynamic scenes and event sequences.
    • Instruction tuning data: organized as instruction-response pairs, this is key to enabling the model to follow complex instructions and transfer to downstream tasks.

    Data diversity determines the boundary of generalization. One practical principle is worth emphasizing: the difficulty distribution of the data should match the target capability—training for reasoning especially depends on high-quality samples containing explicit reasoning steps, not merely question-answer pairs.

    4. Pre-training: Building the Foundation of Vision-Language Alignment

    The goal of the pre-training stage is to establish a solid cross-modal representation. The typical approach is to:

    1. Run contrastive or generative pre-training on large-scale image-text pairs so that the vision encoder and the language model learn to “align”;
    2. Preserve the language model’s general capabilities (avoiding catastrophic forgetting);
    3. Gradually introduce higher-resolution, more complex image-text and interleaved data to improve fine-grained understanding.

    This stage does not directly pursue reasoning. Instead, it provides a strong base that can “see clearly and speak coherently” for the two training stages that follow.

    5. Supervised Fine-Tuning (SFT): Teaching the Model “How to Think”

    SFT is the key springboard for forming reasoning ability, and its core lies in high-quality CoT data and a standardized response format.

    A standardized response format makes the reasoning process parseable and verifiable. A typical structure looks like this:

    1. Question: <user input>
    2. [Reasoning] Step-by-step analysis: first identify the key elements in the image, then establish logical relationships...
    3. [Answer] Final answer and supporting rationale.

    The benefits of a unified format are twofold: on one hand, it makes it easy to automatically extract the answer during the RL stage for reward computation; on the other hand, it lets users review the model’s reasoning process, improving trustworthiness.

    Data quality management and iterative enhancement are another focus of SFT. The typical practical path is:

    1. First generate candidate reasoning chains using a strong model;
    2. Remove incorrect, redundant, or off-topic samples through rule-based filtering plus model scoring;
    3. For hard problems, perform multi-path sampling and majority voting, keeping samples whose reasoning paths are correct;
    4. Form a closed “generate-filter-retrain” loop that continuously increases the proportion of high-quality data.

    It is worth stressing that the SFT stage should retain a certain amount of general dialogue and harmlessness data, so the model does not lose its general capabilities while focusing on reasoning.

    6. Reinforcement Learning (RL): Unleashing Reasoning Potential

    SFT teaches the model to “imitate” reasoning; RL teaches the model to “explore” better reasoning. The two key components of this stage are the reward system and curriculum sampling.

    6.1 Reward System

    Reward design determines the direction the model optimizes toward. For tasks that can be automatically verified, rule-based rewards are typically used to obtain a stable, unbiased signal:

    • Answer-correctness reward: exact matching or numerical validation of the answer, giving a positive reward when correct;
    • Format reward: constraining the model’s output to conform to the predefined reasoning/answer structure;
    • Reasoning-process reward (optional): providing an auxiliary signal for the soundness of intermediate steps, mitigating the “right answer, wrong process” pseudo-correctness problem.

    A simplified example of a rule-based reward:

    1. def compute_reward(pred_answer, gt_answer, response):
    2. reward = 0.0
    3. # Answer correctness: exact match or numerical equivalence
    4. if is_correct(pred_answer, gt_answer):
    5. reward += 1.0
    6. # Format constraint: whether the reasoning/answer structure is complete
    7. if has_valid_reasoning_format(response):
    8. reward += 0.2
    9. return reward

    6.2 Reinforcement Learning with Curriculum Sampling (RLCS)

    This is the core innovation of the framework. Conventional RL samples from the same difficulty distribution, which easily falls into two extremes: if problems are too easy, rewards saturate and gradients vanish; if problems are too hard, positive rewards rarely arrive and training stalls.

    The idea of curriculum sampling is to let the sampling difficulty dynamically adapt to the model’s current level as training progresses. It can be abstracted as:

    1. def curriculum_sampling(model, problem_pool, pass_rate_window=0.6):
    2. samples = []
    3. for problem in sample_batch(problem_pool):
    4. # Estimate the model's pass rate on this problem from recent history
    5. p = estimate_pass_rate(model, problem)
    6. # Prefer problems that are "within reach with a little effort"
    7. if LOWER_BOUND < p < UPPER_BOUND:
    8. samples.append(problem)
    9. return samples

    That is, it preferentially samples problems whose “current pass rate falls in a medium range”—problems the model cannot solve effortlessly, yet cannot leave completely unsolved. This keeps the training signal consistently effective while gradually steering the model toward harder tasks.

    6.3 Optimization Details

    In practice, teams typically also: apply a lower learning rate or a freezing strategy to the vision encoder to stabilize features; use group-relative policy optimization–style algorithms to reduce dependence on a value network; suppress variance through experience replay and reward normalization; and combine distributed training to ensure throughput for large-scale sampling.

    7. Experimental Results: Capability Gains and Cross-Domain Benefits

    Judging from the overall trends reported publicly, this kind of reasoning-centric framework delivers gains on two levels:

    1. Leadership at comparable scale: among open-source models with similar parameter counts, they perform strongly on benchmarks for math, document understanding, visual grounding, GUI agents, and video understanding; on some highly difficult tasks, they can even surpass models an order of magnitude larger in parameter count, as well as some fairly capable general-purpose models.
    2. Cross-domain mutual reinforcement: because reasoning is a transferable general capability, the step-by-step reasoning developed on math problems transfers positively to seemingly unrelated tasks such as document analysis, chart interpretation, and interface operation. This “kill many birds with one stone” effect is exactly the unique value that distinguishes a general reasoning framework from task-specific models.

    8. Conclusion and Outlook

    The core contributions can be summarized in three points:

    • Proposing a general multimodal training framework centered on reasoning that systematically introduces CoT and RL into VLMs;
    • Designing Reinforcement Learning with Curriculum Sampling (RLCS) to effectively alleviate difficulty mismatch and signal sparsity in RL training;
    • Building a multi-dimensional data system covering image-text, OCR, grounding, video, and instructions, demonstrating that high-quality data + precise rewards + efficient RL strategies are the three pillars of a general reasoning model.

    Regarding limitations and outlook, several challenges remain: reward design under visual input is more complex than for pure text; the faithfulness of the reasoning process is hard to guarantee completely; and there are the compute costs of long videos and high-resolution documents. Future directions may focus on process-level rewards, finer-grained vision-reasoning alignment, and closed-loop reinforcement in real interactions (such as GUI agents).

    For engineering practitioners, the biggest takeaway from this framework is: reasoning ability is not an exclusive property of any single module, but a system-level result shaped jointly by data, training objectives, and optimization strategies.