From Pre-training to Reinforcement Learning: Building Reasoning-Enabled Vision-Language Models
Vision-language models (VLMs) aim to let machines not only describe what they see but also reason about it. While large language models (LLMs) have made major breakthroughs in text reasoning through chain-of-thought and reinforcement learning, most VLMs remain stuck at perception and understanding. This article explains how to transfer these proven LLM reasoning paradigms into the visual domain. It walks through the mainstream ViT-MLP-LLM architecture, the multimodal data pipeline, pre-training, supervised fine-tuning (SFT) with high-quality reasoning traces, and reinforcement learning with curriculum sampling and verifiable rewards. Readers will come away with a practical, end-to-end methodology for building general-purpose multimodal models centered on reasoning.
1. Introduction: Why Vision-Language Models Need “Reasoning”
The goal of a vision-language model (VLM) is to let machines “describe what they see” and even “think about what they see.” Its capabilities can be broken down into three progressive layers:
- Perception: recognizing “what is in” an image (objects, scenes, text);
- Understanding: linking visual information to language semantics to answer “what is it / how does it work”;
- Reasoning: performing multi-step logical inference from known information to answer “why / what to do next.”
For a long time, LLMs have made breakthroughs in pure text reasoning, relying mainly on two paths: Chain-of-Thought (CoT), which guides the model to explicitly generate intermediate reasoning steps instead of directly emitting an answer; and Reinforcement Learning (RL), which uses reward signals to teach the model to “think first, then answer” and to self-correct during exploration.
However, most existing VLMs still remain at the “perception + understanding” stage. Faced with problems that require multi-step logic—such as math word problems, chart analysis, and complex document layouts—they tend to jump straight to conclusions without intermediate derivation. This leads to high error rates, poor explainability, and weak generalization on out-of-distribution samples.
Therefore, transferring the reasoning paradigms that LLMs have already proven into the visual domain, and building a “reasoning-centric” general-purpose multimodal model, is a direction with both academic and engineering value. This article breaks down the complete methodology—architecture, data, training, and reinforcement learning—behind this topic.
2. Model Architecture: The Mainstream ViT-MLP-LLM Paradigm
Most current reasoning-enabled VLMs adopt a three-stage structure of “vision encoder + projection layer + large language model,” abbreviated as ViT-MLP-LLM:
- Vision encoder (ViT): Splits the input image into patches, encodes them into a sequence of visual tokens, and extracts low-level visual features. For high-resolution scenarios, dynamic resolution slicing is often used—dividing a large image into multiple sub-blocks to balance detail against compute cost.
- Projection layer (MLP/Adapter): Maps visual features into the LLM’s embedding space, aligning “visual tokens” and “text tokens” in a shared semantic space. The structure is typically lightweight (a multi-layer perceptron is enough), yet it is the key bridge for cross-modal alignment.
- Large language model (LLM): Serves as the core for reasoning and generation. It receives the concatenated sequence of “visual tokens + text tokens” and autoregressively generates the answer, and even the full reasoning process.
The advantage of this design is that it reuses the mature reasoning ability of LLMs, so the focus narrows to two problems: “how to inject high-quality visual features” and “how to elicit cross-modal reasoning.”
3. Data Preparation: The Foundation That Sets the Ceiling
The upper limit of a multimodal model’s capability is largely determined by its data. A complete data system typically covers the following categories:
- Image-caption data: Image-text pairs are the most basic alignment signal, used to help the model build a mapping between “visual regions and semantic concepts.” This data is usually large in scale and broad in source, making it the mainstay of pre-training.
- Interleaved image-text data: Natural corpora where images are embedded within text (similar to web pages and textbook layouts). This helps the model understand real reading scenarios of “images within text” and is especially critical for long-document understanding.
- Optical Character Recognition (OCR) data: Covering printed text, handwriting, scene text, and document formulas, this is the foundation for the model to “read text inside images” and the core source of capability for document and receipt tasks.
- Visual grounding data: Annotated with “object–bounding box” or “object–coordinate point” correspondences, this gives the model spatial grounding ability—hitting exactly what it points to—supporting tasks such as GUI agents and chart reading.
- Video data: Frames a video into a sequence to train the model to understand temporal changes and action semantics, extending capabilities from static images to dynamic scenes.
- Instruction tuning data: Organized in the form of “instruction–response” across diverse tasks, this is the key to moving the model from “being able to describe” to “being able to get things done.”
Data quality > data quantity. In practice, governance steps such as deduplication, cleaning, difficulty labeling, and answer verification often deliver more stable gains than simply piling on more data.
4. Pre-training: Building a General Visual Foundation
The goal of the pre-training stage is to build a well-rounded visual foundation model:
- Train on large-scale image-text and interleaved image-text data so that the ViT and the LLM complete initial alignment;
- Combine this with high-quality corpora to consolidate the model’s general language and commonsense ability;
- Use continued training to further improve the consistency and robustness of multimodal representations.
This stage does not aim to “solve hard problems.” Rather, it lays a solid foundation of perception and understanding, providing ample nourishment for injecting reasoning ability later.
5. Supervised Fine-Tuning (SFT): Teaching the Model “How to Think”
After pre-training, the model can already “see images,” but it may not yet “reason.” The core problem SFT must solve is: how to get the model to form a structured thought process before answering.
5.1 High-Quality CoT Data
The key to SFT is building high-quality reasoning-trace data. Ideal CoT data should have:
- Complete reasoning steps with coherent logic, rather than a leap straight to a conclusion;
- Intermediate steps that are verifiable and reproducible;
- A final answer that is correct (ensured via rule-based verifiers or manual spot checks).
For complex reasoning that is difficult to annotate manually, a “sample + verify” strategy can be used: have the model generate multiple reasoning paths for the same problem, and keep only the trajectories that reach the correct answer with a sound process as training samples.
5.2 Standardized Response Format
To make reward computation and result parsing easier, a unified output format is usually agreed upon—for example, separating the reasoning process from the final answer:
<Thinking>1. First identify the known conditions in the image...2. Establish the relationship...3. Solve and verify...</Thinking><Answer>Final result</Answer>
With a unified format in place, an automated reward system can reliably extract the answer from the output and judge its correctness.
5.3 Data Quality Management and Iterative Improvement
SFT is not one-off work but a continuous, iterative process: by analyzing the model’s error types on a validation set, you can add data for weak tasks in a targeted way, remove low-quality samples, and fix incorrect labels—forming a closed loop of “train–evaluate–supplement data.”
6. Reinforcement Learning (RL): Fully Unlocking Reasoning Potential
SFT taught the model to “imitate” the correct way of reasoning, while reinforcement learning lets it push its ceiling higher through self-exploration—especially learning to “try several approaches” on hard problems.
6.1 Reward System
Reward is the “baton” of RL. In reasoning-enabled VLMs, a common approach is to use rule-based rewards + verifiable objectives, which reduces the risk of the reward being “gamed”:
def reward_fn(sample, trajectory):reward = 0.0# Format reward: whether the output structure is well-formedif format_is_valid(trajectory):reward += 0.2# Correctness reward: whether the answer matches the ground truthif is_correct(extract_answer(trajectory), sample.ground_truth):reward += 1.0return reward
For tasks that can be automatically verified, such as math, OCR, and grounding, rule-based rewards are especially reliable; for open-ended generation, a preference model can be added to provide scoring.
6.2 RLCS: Reinforcement Learning with Curriculum Sampling
Running RL directly on the hardest tasks often lacks effective gradients because the initial success rate is too low. An effective strategy is curriculum learning: organizing training from easy to hard so the model first accumulates positive feedback on “reachable” tasks, then gradually takes on harder ones. This framework of Reinforcement Learning with Curriculum Sampling can significantly improve training stability and convergence quality.
def rl_train_loop(model, task_pool, reward_fn, steps):for step in range(steps):# 1. Sample tasks by current curriculum difficulty (easy to hard)difficulty = curriculum.schedule(step)batch = task_pool.sample(n=64, by_difficulty=difficulty)# 2. Generate multiple reasoning trajectories per sample to encourage explorationtrajectories = [model.generate(s) for s in batch]# 3. Score with rule-based/model rewardsrewards = [reward_fn(s, t) for s, t in zip(batch, trajectories)]# 4. Normalize relative advantages within the group and update the policyadvantages = group_normalize(rewards)model.update(trajectories, advantages)# 5. Dynamically adjust curriculum difficulty based on performancecurriculum.update(step, rewards)
6.3 Optimization Details
Engineering practice also requires attention to: balancing sampling temperature against the number of trajectories (balancing exploration and cost), normalizing reward scales (so that differences in reward magnitude across tasks do not interfere with training), and preventing “reward hacking” (e.g., the model learning only the format but not the reasoning). These details often determine whether the RL stage genuinely improves capability or leads to degradation.
7. Results and Cross-Domain Synergy
A reasoning-centric framework typically benefits multiple task types simultaneously: STEM problem solving, video understanding, content recognition, coding assistance, visual grounding, GUI agents, and long-document understanding reinforce one another—improvements in reasoning spill over into perception tasks, forming a positive feedback loop. At the same parameter scale, a well-designed reasoning model can often outperform a larger base model, showing that “training methodology” and “data quality” can sometimes matter as much as “piling on parameters.”
Note: The training and inference pipeline described above places high demands on compute, storage, and data processing. In practice, this can be carried on general-purpose cloud capabilities—for example, using a container platform to elastically schedule distributed training, object storage to manage massive image-text datasets, a message queue to decouple the data pipeline, and monitoring/alerting together with log services to track training metrics and anomalies. This kind of infrastructure can significantly lower the engineering barrier from experimentation to production.
8. Summary and Outlook
Building a “reasoning-enabled vision-language model” can be summarized in three points:
- Architecture: Adopt the ViT-MLP-LLM paradigm, reuse the mature reasoning ability of LLMs, and focus on high-quality injection of visual features;
- Data: Build a multi-source, high-quality dataset covering image-text, interleaved, OCR, grounding, video, and instructions, and stick to “quality first”;
- Training: Use SFT to teach structured thinking, use reinforcement learning with curriculum sampling to fully unlock potential, and rely on verifiable rewards to keep the direction correct.
Limitations and outlook: Current methods still have room for improvement in reward design for open-ended reasoning, ultra-long video understanding, and reliability verification of the reasoning process. Future directions include stronger process-level rewards, finer-grained cross-modal alignment, and evaluation of the explainability and safety of reasoning. For developers, combining “high-quality data + precise rewards + efficient reinforcement learning” is a methodology worth sticking to when building general-purpose multimodal reasoning models.
