How Vision-Language Models Learn to Reason: Multimodal Reinforcement Learning Training Explained

    This article breaks down how modern vision-language models (VLMs) move beyond image captioning toward verifiable, multi-step reasoning. It covers the full training pipeline—the ViT–MLP–LLM architecture and its trade-offs, the multimodal data foundation, pretraining, supervised fine-tuning with chain-of-thought, and reinforcement learning driven by verifiable rewards and curriculum sampling. It also summarizes publicly reported results, the current limitations, and practical guidance for deploying these models in production. Aimed at AI engineers and technical leaders, it explains why data quality, rule-based rewards, and adaptive sampling matter more than any single architectural choice.

    1. Problem Background: Where VLMs Fall Short

    Recent vision-language models have evolved far beyond “captioning an image in a single sentence” to handle question answering, document parsing, GUI operation, and many other tasks. Break that ability apart, and it roughly splits into three layers:

    1. Perception: Recognizing what is in an image, including objects, text, chart elements, and their spatial relationships.
    2. Comprehension: Mapping visual signals into semantic space and building a joint “image–text” aligned representation.
    3. Reasoning: Performing multi-step computation, constraint satisfaction, and evidence retrieval on top of comprehension to reach conclusions that can only be verified through a process.

    The first two layers are largely usable after large-scale image-text pretraining. The third has long been the weak link, for three main reasons:

    • Insufficient supervision signal: Traditional image-text alignment data provides only the “final answer.” The model lacks intermediate examples of how to reach that answer, so it can only learn shallow pattern matching.
    • Weak task generalization: Models tend to be task-specific; change the question type or phrasing and performance drops noticeably.
    • Lack of reliable verification: Visual tasks have failure modes such as hallucination, coordinate drift, and misread numbers. Without an automatically checkable correctness signal, the model cannot self-correct.

    The language model field has already proven a repeatable path: first teach the model to “write out its reasoning” (Chain-of-Thought, CoT), then use reinforcement learning (RL) to optimize the correctness of that reasoning. Vision-language models can reuse this path entirely—and that is the central thesis of the technical report this article is based on: a reasoning-centric end-to-end training framework.

    2. Model Architecture: Engineering Trade-offs in ViT–MLP–LLM

    The backbone of most mainstream VLMs is similar, usually written as a three-stage ViT–MLP–LLM:

    1. Image ──► Vision encoder (ViT) ──► Projection layer (MLP/2~3 layers) ──► LLM ──► Text output
    2. │ ▲
    3. └── Optional: high-res tiling / dynamic-resolution cropping ──┘

    Key engineering details:

    • Choice of vision encoder: Prioritize fine-grained capability (small text, chart ticks, UI controls), because this directly sets the ceiling for downstream OCR, grounding, and GUI tasks.
    • Dynamic resolution and tiling: Downscaling a whole image loses detail on small targets, so a common approach is to cut a high-resolution image into several sub-blocks, encode them in parallel, and reassemble them into the original image’s visual sequence. The cost is a larger number of visual tokens, which must be traded off against context length and memory overhead.
    • Role of the projection layer: It maps visual features into the input dimensionality of the language model—a lightweight “translation layer.” Keeping it simple avoids an information bottleneck and makes it easier to freeze or partially freeze the language model during large-scale pretraining.
    • Interleaving visual tokens and text tokens: For multi-image, long-document, and video-frame scenarios, how placeholders are inserted and how image boundaries are marked directly determine the model’s ability to refer to “which page, which line, which frame.”

    It is worth emphasizing that architecture itself is not the decisive variable for reasoning ability. With the same backbone, different data and different training objectives can produce final capability gaps greater than an order of magnitude. The real dividing line lies in the data and post-training.

    3. Data System: The Foundation of Multimodal Reasoning

    A reusable multimodal data system typically contains the following six categories, with deliberately maintained mixing ratios and difficulty gradients:

    1. Image-caption data: Provides coarse-grained image-text alignment, teaching the model “what this image is about overall.” This is the bedrock of visual semantics.
    2. Interleaved image-text data: Mirrors the natural “image–text–image–text” structure found in web pages, textbooks, and technical documents, training the model to cross-reference and reason across paragraphs among multiple images. This is the main source of capability for long-document and multi-image tasks.
    3. Optical Character Recognition (OCR) data: Covers scene text, document layout, tables, and formulas. The value of OCR goes beyond “reading the characters”—it builds a joint representation of “text–layout–semantics.”
    4. Grounding data: Labels target locations as bounding boxes or keypoints, enabling the model to produce spatially verifiable results. This is a key support point for making reasoning actionable—grounding correctness can be automatically judged by metrics such as IoU, making it naturally suited as a verifiable reward source for reinforcement learning.
    5. Video data: Includes annotations for temporal description, event ordering, and shot-level understanding, training the model to handle causal and sequential reasoning over the time dimension.
    6. Instruction-tuning data: Unifies the heterogeneous data above into a dialogue format of “instruction–input–expected output,” covering a broad range of tasks such as subject problem-solving, document QA, GUI operation agents, and code comprehension.

    Practical point: data quality matters far more than quantity. The three factors with the greatest impact on downstream reinforcement learning gains are annotation consistency, coordinate precision, and answer verifiability.

    4. Pretraining: Building “Clear Sight and Accurate Reading” First

    The goal of pretraining is not reasoning, but aligning the visual and language representation spaces to a usable degree. Common strategies include:

    • Progressive-resolution training: Start at a lower resolution to control early-stage cost, then increase resolution and introduce tiling in the mid-to-late stages, gradually approaching real-world usage scenarios.
    • Task-mix ratios: Sample image-text alignment, OCR, grounding, video captioning, and other tasks by weight to avoid overfitting a single task, which can degrade other capabilities.
    • Timing of unfreezing the vision encoder: Freeze the vision tower early to stabilize training, then unfreeze it in the mid stage and jointly optimize it with the projection layer and language model, which significantly reduces the difficulty of optimizing a large vision tower.

    An empirical indicator of whether pretraining is on track: whether the model is nearing saturation on “pure perception” tasks such as document OCR, chart reading, and fine-grained counting. If these foundational tasks still fail on a large scale, moving directly into reinforcement learning will only amplify noise.

    5. Supervised Fine-Tuning (SFT): Turning Implicit Ability into Explicit Reasoning Chains

    The SFT stage must solve two problems at once: teaching the model how to think, and teaching the model how to answer.

    5.1 Constructing High-Quality CoT Data

    Reasoning data usually comes from three sources, in increasing order of trustworthiness:

    • Distillation from a strong model: Use a more capable model to generate solution processes—low cost and broad coverage, but requires strict filtering.
    • Human writing and revision: Hand-write reasoning chains for high-value, high-difficulty tasks; few samples, but clean signal.
    • Programmatically verifiable generation: For problem types with standard answers, sample multiple reasoning paths, then validate them with rules or execution results, keeping only samples where “the path is correct and the answer is correct.”

    Filtering typically covers: answer correctness, consistency of intermediate steps, presence of unfounded visual claims (hallucination), and traces of “guessing the answer first and backfilling the process.”

    5.2 Standardized Response Format

    This is an easily underestimated step with very high payoff. After fixing an output template of “reasoning section + answer section + optional structured fields,” three benefits follow:

    1. Reasoning: First locate the decimal point in the image, then align digit by digit...
    2. Answer: 0.35
    • Reasoning content is explicitly separated, making it easy to localize whether an error occurred in “perception” or in “computation.”
    • Answer extraction becomes deterministic, making rule-based rewards possible to automate.
    • Downstream systems can parse the output directly, sharply reducing engineering integration cost.

    5.3 Iterative Enhancement of Data Quality

    The mature practice is a closed loop: run reasoning sampling with the current model on the training set → identify error patterns (misreading ticks, ignoring constraints, coordinate drift) → add or revise data accordingly → retrain. Each iteration concentrates the error distribution and makes data placement more precise.

    6. Reinforcement Learning (RL): Pushing the Model’s Ceiling with Verifiable Rewards

    SFT makes the model “do what it’s shown.” RL makes the model “do it more correctly.” Three key modules define the design.

    6.1 Reward System

    Reward design for multimodal tasks follows one principle: never hand to a model to score what can be rule-based. A typical composition:

    Reward term Judging method
    Answer correctness Exact match on numbers/options/text, or match after normalization
    Grounding correctness IoU between predicted box and annotated box; distance between center points
    Format compliance Whether the reasoning and answer sections follow the template
    Process validity No hallucinated claims, no redundant repetition, non-empty reasoning steps
    Length/efficiency Light penalty for outputs that are too long or terminate too early
    1. def compute_reward(pred, gt, task_type):
    2. ans = extract_answer(pred) # Extract via template
    3. reward = 0.0
    4. reward += format_reward(pred) # Format compliance
    5. if task_type == "grounding":
    6. reward += iou_reward(ans, gt.bbox) # Spatially verifiable
    7. else:
    8. reward += accuracy_reward(ans, gt.answer) # Answer verifiable
    9. reward -= hallucination_penalty(pred) # Visual-claim check
    10. return reward

    Key point: rewards must be verifiable, low-noise, and hard to game. If the format reward is weighted too heavily, the model degenerates into “writing the template without solving the problem”; if the answer reward is the only signal, the model easily learns to guess answers. In practice, format and correctness are often decoupled, with answer correctness weighted as the absolute dominant term.

    6.2 RLCS: Adaptive Policy with Curriculum Sampling

    The biggest stability risk in reinforcement learning is difficulty mismatch: if problems are too hard, the model’s samples are almost all wrong, the group advantage estimate collapses to zero, and the gradient carries no information; if problems are too easy, there is no gradient for the same reason. Reinforcement Learning with Curriculum Sampling (RLCS) dynamically maintains difficulty buckets and uses the model’s own real-time pass rate to choose the sampling distribution:

    • Problems with a pass rate near 1 or near 0 carry weak learning signal—down-weight them or sample them less.
    • Problems whose pass rate falls in the middle range (neither all-correct nor all-wrong) carry the richest gradient information—sample them preferentially.
    • Pass rates are refreshed dynamically as training progresses, so the curriculum “levels up” accordingly, achieving adaptive progression of difficulty.

    Pseudocode sketch:

    1. for step in range(total_steps):
    2. batch = sampler.sample(
    3. buckets=curriculum.buckets,
    4. strategy="favor_mid_pass_rate", # Prefer medium difficulty
    5. refresh_every=50 # Periodically refresh pass-rate stats
    6. )
    7. rewards = [compute_reward(o, g, t) for o, g, t in batch]
    8. advantages = group_relative_advantage(rewards) # In-group normalization, no extra critic
    9. update_policy(batch, advantages)
    10. curriculum.update(step, rewards)

    In essence, this design turns “the question-selection strategy of a human teacher” into a sampler: instead of pursuing monotonically increasing difficulty, it keeps the model at the learnable edge at all times.

    6.3 Optimization Details

    • Group-relative advantage optimization: Multiple samples of the same problem are compared against each other, eliminating a separate value network and significantly reducing memory and implementation complexity.
    • Clipping and entropy control: Constrain the magnitude of policy updates while maintaining adequate exploration, avoiding premature convergence onto a single solution.
    • Reasoning-length management: Overly long chains dilute the gradient, so length rewards or penalties are needed to strike a balance.
    • Asynchronous sampling and training: Decoupling rollouts from parameter updates significantly improves GPU utilization and training throughput.
    • Numerical stability: Reward normalization, gradient clipping, and learning-rate warmup are the baseline configuration for keeping long training runs from diverging.

    7. Observing the Results: Improvements Are “Omnidirectional”

    The publicly reported results of this line of work show three noteworthy patterns:

    1. Leadership at comparable scale: Across more than twenty public benchmarks covering subject knowledge, math visual reasoning, document understanding, video understanding, visual grounding, GUI operation agents, and code comprehension, a 9B-class model outperforms general VLMs that are an order of magnitude larger, and on some challenging tasks beats the closed-source general models available at the time.
    2. Cross-domain mutual reinforcement: RL gains trained on STEM reasoning transfer to video understanding and grounding; conversely, the “precisely locate—then judge” pattern reinforced through grounding reduces hallucination in long-document QA. This indicates that RL mainly learns a general capability of “evidence localization + multi-step verification” rather than the specific solution to any single task.
    3. Format and interpretability improve together: Explicit reasoning chains make errors attributable, which in enterprise scenarios is often more valuable than a raw score increase—to investigate a wrong answer, you only need to check whether it erred in perception or in computation.

    8. Limitations and Recommendations for Engineering Deployment

    Current main limitations:

    • Perception ceiling limits the reasoning ceiling: When the image itself lacks resolution or the text is blurry, even the strongest reasoning chain cannot compensate, and errors amplify along the chain.
    • Visual hallucination is not yet fully solved: The model can still “cite” elements that do not exist in the image, requiring an additional visual-claim verification mechanism.
    • Weak long-duration video understanding: Cross-frame causal reasoning over long videos remains an obvious shortcoming.
    • Higher reasoning cost: Explicit chains of thought bring a several-fold increase in token consumption, and latency and compute overhead require dedicated optimization.
    • RL training stability: Once a reward design has a loophole, the model quickly finds a “shortcut,” requiring continuous reward auditing and manual spot checks.

    Deployment recommendations:

    • Establish the evaluation system first: without a reliable task-level evaluation set, no training iteration can judge its gains.
    • Prioritize investing in automatically verifiable tasks (grounding, structured extraction, numerical computation), since they serve both as capability anchors and as reward sources.
    • On data governance, hold to the dual standard of “annotation consistency + answer verifiability”—better to have less that is accurate.
    • Deploy training and inference pipelines separately: training clusters handle rollouts and parameter updates, while the online-serving side focuses on quantization, caching, and concurrency scheduling. In cloud practice, data and checkpoints are typically kept in object storage, training jobs are scheduled by a container platform, and a message queue decouples sampling from updates, with logging and monitoring/alerting services observing the reward curve and training stability—the maturity of this infrastructure often affects the final iteration speed more than model selection does.

    Summary: The capability breakthrough of multimodal reasoning models does not come from any single new module, but from the combined force of three things: high-quality verifiable data + precise rule-based rewards + adaptive curriculum sampling. Do these three solidly, and a vision-language model can go fully from “recognizing images” to “performing verifiable multi-step reasoning grounded in images.”