Open-Source Vision-Language Model Reasoning: Architecture, Data, SFT, and RL
This article examines how vision-language models (VLMs) can move beyond perception and basic understanding toward reliable multi-step reasoning. It covers the capability boundary between seeing and reasoning, the vision encoder–projection–language model architecture, multi-source data design, pretraining, supervised fine-tuning, reward design and curriculum sampling for reinforcement learning, and evaluation for generalization. Readers will learn the key engineering considerations for building a reasoning-centric VLM training framework and the main limitations of current approaches.
1. From Seeing to Reasoning: The Capability Boundary of VLMs
The capabilities of vision-language models (VLMs) are often divided into three levels: perception, understanding, and reasoning. Perception answers “what is in the image,” understanding answers “what relationships exist between objects,” and reasoning goes further to answer “why, what will happen, and what should be done next.” Many models are mature on tasks such as image captioning and basic visual question answering, but once they enter scenarios such as multi-step reasoning, scientific problem solving, long-document understanding, and causal analysis of video events, they often show a problem: they can see, but they cannot reason clearly.
There are three main causes. First, training data is dominated by short question-answer pairs and descriptive samples, with little explicit supervision of the reasoning process, so models tend to give conclusions directly. Second, training objectives emphasize language likelihood fitting and lack joint constraints on whether the answer is correct, whether the reasoning process is self-consistent, and whether the visual evidence is sufficient. Third, after visual information enters the language model, it is easily compressed into a small number of semantic tokens; fine-grained localization, table structure, video temporal relations, and other information gradually decay in long-chain reasoning.
Therefore, a reasoning-centric training framework needs coordinated design across architecture, data, supervised fine-tuning, and reinforcement learning, rather than simply scaling parameters or increasing the number of image-text pairs.
2. Overall Architecture: Vision Encoder, Projection Layer, and Language Model
A common VLM architecture can be summarized as “vision encoder—projection layer—language model.” The vision encoder extracts visual features from images or video frames; the projection layer maps visual features into a representation space acceptable to the language model; the language model performs multimodal context fusion and text generation.
The following details deserve attention in engineering implementation:
- Resolution and tiling strategy: High-resolution images, document screenshots, and tables require finer visual token coverage, but an increasing number of tokens significantly raises computational cost. Common approaches combine dynamic resolution, image tiling, and global thumbnails.
- Visual token compression: Pooling, query-based compression, or attention selection can reduce redundant tokens and control context length while preserving information from key regions.
- Position encoding and spatiotemporal modeling: Images require two-dimensional positional relationships, and video also requires a temporal dimension. For long videos, keyframe selection or event segmentation should be added beyond uniform sampling.
- Decoupled training stages: During pretraining, some modules can be frozen to stabilize alignment; during fine-tuning, the vision encoder and language model can be gradually unfrozen to avoid early interference among multi-task signals.
- Long-context management: Long documents, multiple images, and video bring very long contexts. Chunked encoding, hierarchical summarization, and evidence citation mechanisms are needed so that the model can refer back to specific visual regions or time segments when generating answers.
The architecture itself is not mysterious. The difficulty lies in making visual representations, language representations, and the reasoning process work together stably under the same training objective.
3. Data System: Multi-Source Data Determines the Capability Ceiling
The data system for reasoning-oriented VLMs is usually not a single set of image-text pairs, but a mixture of multiple data types.
- Image-caption data: Used to establish basic visual-semantic alignment so the model can learn objects, scenes, and attributes.
- Interleaved image-text data: Web pages, documents, tutorials, and similar sources naturally contain alternating text and images, helping the model learn cross-modal context relationships.
- Optical character recognition data: Documents, receipts, tables, screenshots, and similar scenarios require text recognition and layout understanding, which is the foundation for long-document reasoning.
- Visual grounding data: Supervision such as region boxes, points, and masks enables the model to map objects in language to image regions and reduces the problem of “answering correctly but looking at the wrong place.”
- Video data: This needs to cover action recognition, temporal question answering, event causality, and long-video summarization so the model learns to understand changes between frames.
- Instruction fine-tuning data: This includes question answering, multi-turn dialogue, tool-style operation instructions, and formatted output, and is used to improve task-following ability.
Data quality is more important than scale alone. In engineering, teams should build checks for deduplication, noise filtering, difficulty stratification, answer verifiability, and visual evidence consistency. For reasoning data, it is also necessary to ensure that the thought process is consistent with the final answer and to avoid pseudo-reasoning samples whose process looks reasonable but whose conclusion is wrong.
4. Pretraining: Building a Stable Multimodal Foundation
The goal of pretraining is to give the model general vision-language alignment capability. In practice, multi-stage mixed training is commonly used: first establish basic alignment with large-scale image-text pairs, then introduce interleaved image-text, OCR, grounding, and video data to expand capability boundaries. A progressive resolution strategy can be used during training: first stabilize convergence at low resolution, then gradually increase resolution and context length.
The key to pretraining is not to mix all data at once, but to control data ratios and difficulty distribution. If descriptive data is too abundant, the model may focus too much on surface semantics; if OCR and grounding data is too abundant, open-scene understanding may be weakened. A reasonable approach is to set sampling weights by capability dimension and retain a certain proportion of general language data to prevent language capability degradation.
5. Supervised Fine-Tuning: Turning “Can See” into “Can Reason in the Required Format”
The core of supervised fine-tuning is high-quality reasoning data and standardized response formats. High-quality samples should contain clear reasoning steps, necessary references to visual evidence, and verifiable final answers. The response format can separate the “analysis process” from the “final conclusion,” making it easier for a later reward system to evaluate each part.
Data quality management can use an iterative enhancement process: first have a base model generate candidate answers, then filter samples through rule validation, answer consistency comparison, and manual spot checks; attribute errors in incorrect samples to visual recognition errors, broken reasoning chains, or format errors, and then supplement data in a targeted way. This is more effective at improving model capability than simply increasing sample size.
6. Reinforcement Learning: Reward Design and Curriculum Sampling
Reinforcement learning is used to further optimize reasoning behavior. An operational reward system usually consists of multiple parts:
total_reward =answer_reward # whether the final answer is correct+ format_reward # whether the output format meets requirements+ evidence_reward # whether reasonable visual evidence is cited+ process_reward # whether reasoning steps are self-consistent and free of obvious contradictions- length_penalty # penalty for overly long or redundant output
The answer reward can come from rule matching, numerical validation, or verifiable execution results; the format reward ensures that output can be parsed; the process reward needs careful design to prevent the model from learning template-like reasoning that only appears reasonable.
The idea of curriculum sampling reinforcement learning is to let training difficulty change dynamically: early stages focus on short answers, single images, and clear tasks, while later stages gradually add multiple images, long documents, video, and open-ended question answering. A scheduler can adjust the proportion of task types according to the model’s current pass rate, avoiding sparse rewards caused by overly difficult samples and ineffective training caused by overly easy samples. The pseudocode can be expressed as:
for batch in training:difficulty = scheduler.current_level(model_pass_rate)samples = sample_by_difficulty(dataset, difficulty)responses = policy.generate(samples)rewards = reward_model.score(samples, responses)policy.update(rewards, kl_constraint)scheduler.update(model_pass_rate)
For optimization details, it is necessary to control sampling temperature, output length, and policy deviation, and to set a KL constraint to prevent the model from damaging basic language capability in pursuit of rewards. At the same time, watch for reward hacking: the model may obtain high scores by stacking steps, repeating citations, or catering to the format, so the reward function should remain simple, verifiable, and adjusted in stages.
7. Evaluation and Generalization: Do Not Look Only at a Single Leaderboard Score
Evaluation of reasoning-oriented VLMs should cover tasks such as scientific questions, chart and document understanding, visual grounding, video understanding, interface operation instructions, and long-document question answering. A single leaderboard score reflects only partial capability. More important are cross-domain generalization and mutual reinforcement among capabilities: when OCR and grounding improve, document question answering and interface understanding also benefit; when video temporal understanding improves, event causality question answering also improves.
Evaluation should also focus on hallucination, evidence citation accuracy, long-context degradation, and format stability. For enterprise applications, it is advisable to add small-sample evaluation sets for business scenarios and record the distribution of error types, rather than comparing only average scores.
8. Conclusion and Outlook
The value of a reasoning-centric VLM training framework lies in connecting architecture, data, supervised fine-tuning, and reinforcement learning into a closed loop: use multi-source data to build a foundation for perception and understanding, use high-quality reasoning samples to teach the model how to organize its process, and then use a reward system and curriculum sampling to reinforce correct behavior. Its contribution is to provide an extensible engineering path for general multimodal reasoning; its limitations are that reward design remains complex, training costs for long video and long documents are high, and hallucination in open scenarios is difficult to eliminate completely. Future work can focus on verifiable rewards, visual evidence citation, adaptive data ratios, and finer-grained process evaluation.
