Vision-Language Model Reasoning Training Framework: Architecture, Data, SFT, and RL

    This article examines how to build a scalable training framework for reasoning-oriented vision-language models. It covers the model architecture that combines a vision encoder with a language model, a layered data system for multimodal reasoning supervision, pretraining and supervised fine-tuning strategies, reward design and curriculum sampling for reinforcement learning, and evaluation considerations for real-world generalization.

    Vision-Language Model Reasoning Training Framework: Architecture, Data, SFT, and RL

    1. Background: From “Seeing” to “Reasoning”

    The capabilities of vision-language models (VLMs) can usually be divided into three levels: perception, understanding, and reasoning. The perception level focuses on what is present in an image, such as object detection, text recognition, and scene description. The understanding level focuses on relationships between images and text, such as visual question answering, image-text matching, and document information extraction. The reasoning level requires the model to make multi-step judgments, calculations, comparisons, causal analyses, and plans based on visual information. In the past, most VLM progress concentrated on perception and understanding, while tasks requiring chain-of-thought reasoning, tool-based calculation, long-document inference, and video temporal reasoning often revealed a problem: the model could “see” but could not “reason deeply.”

    There are three main reasons for this phenomenon. First, training data mainly consists of image captions, image-text pairs, and short question-answer samples, lacking high-quality, multi-step, verifiable reasoning trajectories. Second, supervised fine-tuning usually encourages the model to output answers directly, without explicit constraints on intermediate reasoning processes. Third, reinforcement learning has been shown to improve complex reasoning in text-only large models, but in multimodal scenarios, reward design, curriculum sampling, and training stability are more complex. Therefore, a scalable multimodal reasoning training framework needs to address four questions at the same time: whether the architecture can carry multimodal inputs, whether the data can provide reasoning supervision, whether rewards can judge reasoning quality, and whether the training strategy can continuously unlock model potential.

    2. Model Architecture: Coordination Between Vision Encoder and Language Model

    From a general architectural perspective, vision-language models usually adopt a combination of “vision encoder—modality alignment layer—language model.” The vision encoder encodes images or video frames into visual tokens. The modality alignment layer maps visual features into the hidden space of the language model through projection or lightweight adaptation structures. The language model then fuses visual tokens and text tokens to perform understanding, generation, and reasoning.

    The key to this architecture lies not in module names but in detailed design:

    1. Resolution and image-tiling strategy: High-resolution images, document screenshots, and GUI interfaces contain many small texts and fine-grained controls. If simply downsampled, OCR and localization capabilities will degrade significantly. A common approach is to split large images into multiple local patches while retaining a global thumbnail, balancing overall layout and local details.
    2. Visual token compression: Too many visual tokens lengthen the context and increase computational overhead. It is necessary to balance preserving key information and reducing sequence length, for example by pooling local patches, cropping repeated regions, or using dynamic resolution.
    3. Position and temporal encoding: Video understanding requires not only single-frame content but also temporal order, action changes, and event boundaries. Temporal position encoding and frame sampling strategies directly affect video question answering and long-video summarization.
    4. Preserving language model capabilities: Multimodal training should not only pursue visual alignment but also avoid degrading the language model’s general reasoning, coding, and mathematical abilities. A common strategy is to mix text-only and multimodal data and set a reasonable ratio during training.

    The overall process can be abstracted into the following pseudocode:

    1. def vlm_forward(image, text, vision_encoder, projector, llm):
    2. visual_tokens = vision_encoder(image) # Visual features
    3. visual_tokens = projector(visual_tokens) # Align to language space
    4. text_tokens = llm.tokenize(text)
    5. inputs = concat(visual_tokens, text_tokens)
    6. return llm.generate(inputs)

    Architecture addresses whether information can enter the model, while whether reasoning capability is sufficient depends more on data and training strategy.

    3. Data System: Infrastructure for Reasoning Capability

    Data for multimodal reasoning models should not be a simple stack of image-text pairs but should form a layered and complementary data system. A relatively complete training data combination usually includes the following categories.

    1. Image-Caption Data

    Image-caption data is used to establish basic visual concept alignment, allowing the model to learn objects, scenes, attributes, and relationships. Such data is large in scale and broad in coverage, but descriptions are usually short and cannot directly provide complex reasoning supervision. It is better suited as foundational corpus for the pretraining stage to stabilize alignment between visual encoding and language space.

    2. Interleaved Image-Text Data

    Interleaved image-text data consists of multiple text segments and multiple images alternating, closer to the real structure of web pages, tutorials, manuals, and encyclopedia pages. It helps the model learn cross-image reference, contextual association, and multi-image comparison, and is especially important for long-document understanding, process instructions, and cross-page information integration.

    3. OCR and Document Data

    OCR data includes not only text recognition but also layout structure, table structure, reading order, and field relationships. For receipts, reports, contracts, exam papers, and interface screenshots, the model needs to understand both text content and spatial layout. Training should cover clear documents, low-quality scans, multi-column layouts, tables, and handwriting, and should include downstream tasks such as field extraction and table question answering.

    4. Visual Grounding Data

    Visual grounding requires the model to output target regions based on text descriptions or answer questions based on regions. It connects language reference and spatial coordinates and is foundational for tasks such as GUI agents, image editing, and robot perception. Data should include bounding boxes, points, masks, or region descriptions, and cover single targets, multiple targets, occlusion, and fine-grained parts.

    5. Video Data

    Video data requires modeling both spatial and temporal information. Training tasks can include action recognition, temporal localization, video question answering, event description, and multi-frame comparison. High-quality video data should have clear temporal annotations and avoid replacing video understanding with single-frame descriptions only.

    6. Instruction Fine-Tuning Data

    Instruction fine-tuning data is used to teach the model to organize responses according to user intent, including question answering, summarization, explanation, format control, and multi-turn dialogue. For reasoning models, it is also necessary to add chain-of-thought data with intermediate steps and ensure that final answers are verifiable and formats are stable.

    Data quality is more important than simply scaling up. Common governance processes include deduplication and near-duplicate detection, image-text relevance filtering, OCR confidence validation, answer consistency checking, sampling inspection of reasoning-chain verifiability, and stratified sampling by task difficulty. Through iterative enhancement, data can be supplemented for weak areas of the model instead of blindly increasing total volume.

    4. Pretraining: Building Foundational Multimodal Capabilities

    The goal of the pretraining stage is to give the model stable visual representations, cross-modal alignment, and basic language generation capabilities. Multi-stage training is usually adopted: first, large-scale image-text alignment enables visual features to enter a space usable by the language model; then interleaved image-text, OCR, grounding, and video data are introduced to expand task coverage; finally, text-only corpus is mixed in to maintain the general capabilities of the language model.

    Key engineering issues in pretraining include data ratios, curriculum order, and training stability. Early stages should focus on basic alignment and general descriptions, while middle and later stages gradually increase the proportion of document, grounding, video, and reasoning-related data. If high-difficulty reasoning data is introduced too early, the model may fail to learn effectively due to insufficient basic alignment; if training remains on simple image-text pairs for too long, complex task capabilities will be limited.

    5. Supervised Fine-Tuning: Making the Reasoning Process Learnable

    Supervised fine-tuning (SFT) is a key stage for turning a foundational multimodal model into a usable assistant. For reasoning-oriented VLMs, SFT should not only provide “question—answer” pairs but also high-quality “question—reasoning process—answer” samples.

    1. High-Quality Chain-of-Thought Data

    Chain-of-thought data needs to meet three requirements: steps are relevant to the question, intermediate conclusions are verifiable, and the final answer format is stable. For math and science problems, the reasoning chain should show key formulas, condition extraction, and calculation processes; for document question answering, it should show localization, field comparison, and conclusion formation; for GUI tasks, it should show interface element recognition, operation intent, and the next action.

    2. Standardized Response Format

    Multi-task models need a unified output protocol. For example, the answer part is wrapped with fixed markers, grounding results use coordinate structures, multiple-choice questions output option identifiers, and tool calls use structured fields. Standardization is not about restricting expression but about reducing parsing costs in training and evaluation and enabling the reward system to judge output quality stably.

    3. Data Quality Management and Iterative Enhancement

    SFT data should go through a combined process of model-assisted generation, manual sampling inspection, and rule validation. For reasoning chains, “result back-checking” can be used to check whether intermediate steps are self-consistent; for grounding data, whether coordinates fall within valid regions can be checked; for document question answering, whether the answer can be found in the original text can be verified. Typical errors exposed after model training should flow back as new hard-case data, forming an iterative enhancement loop.

    6. Reinforcement Learning: From “Can Answer” to “Answers Better”

    Reinforcement learning is used to further optimize model behavior on top of SFT. Its core is to design a computable reward system and improve model performance on complex tasks through sampling and policy optimization.

    1. Reward System Design

    Rewards for multimodal reasoning usually consist of multiple parts:

    • Outcome reward: judges whether the final answer is correct, such as numerical matching, option matching, or exact field matching.
    • Format reward: checks whether the output conforms to the structured protocol to avoid parsing failures.
    • Process reward: evaluates whether reasoning steps are reasonable, but should be used cautiously to avoid the reward model preferring fixed templates.
    • Grounding reward: evaluates visual grounding quality based on intersection-over-union, center-point distance, or region containment.
    • Consistency reward: checks whether the answer is consistent with image evidence and document source text, reducing hallucination.

    The pseudocode can be abstracted as:

    1. def reward(sample):
    2. score = 0.0
    3. score += 1.0 if match_answer(sample.pred, sample.gold) else 0.0
    4. score += 0.2 if valid_format(sample.pred) else -0.2
    5. score += 0.3 * grounding_iou(sample.pred_boxes, sample.gold_boxes)
    6. score -= 0.5 * hallucination_penalty(sample.pred, sample.evidence)
    7. return score

    2. Reinforcement Learning Strategy with Curriculum Sampling

    Multimodal tasks vary greatly in difficulty. If all samples are sampled uniformly, easy samples will occupy a large amount of training signal, while difficult samples may cause instability in early training. A reinforcement learning framework with curriculum sampling dynamically adjusts task difficulty and sampling ratios according to current model performance: first establish a stable policy with medium-difficulty and format-compliant tasks, then gradually increase complex tasks such as math, long documents, video, and GUI; reduce sampling weight for samples with excessively high pass rates, and increase weight for samples with learning value that are not completely failed.

    3. Optimization Details

    Reinforcement learning training needs attention to reward scale, sampling diversity, and stability. Rewards that are too strong may induce the model to pursue format while ignoring content, while rewards that are too weak lead to sparse signals. Group normalization, length penalty, repetition penalty, and temperature sampling can balance exploration and exploitation. At the same time, supervised fine-tuning data should be retained in the mixture to prevent the model from forgetting foundational capabilities during the reinforcement stage.

    7. Evaluation and Generalization: Focus on Real Tasks Rather Than a Single Leaderboard

    Evaluation of reasoning-oriented VLMs should cover multiple dimensions: STEM problem solving, document and long-text understanding, OCR, visual grounding, video understanding, GUI operations, code and chart understanding, and more. A single benchmark easily leads to overfitting, so public evaluations should be combined with internal tests on real scenarios.

    From general experience, multimodal reasoning training may bring cross-domain mutual reinforcement: mathematical reasoning improves structured analysis, document understanding improves evidence grounding, grounding tasks enhance spatial perception, and video tasks strengthen temporal modeling. However, capability conflicts may also occur, such as over-emphasizing long-chain reasoning leading to verbose answers, or over-optimizing format leading to degraded natural language expression. Therefore, multi-objective balance needs to be set during training.

    8. Summary and Outlook

    A training framework for general multimodal reasoning can be summarized as follows: use coordination between a vision encoder and a language model as the architectural foundation, provide supervision through a layered data system, build alignment capability through pretraining, teach reasoning formats through high-quality SFT, and strengthen complex task performance through a reward system and curriculum sampling. Its contribution lies not in a single module but in organizing data, supervision signals, and reinforcement learning strategies into a scalable training loop.

    Issues that still require attention include the authenticity and verifiability of reasoning chains, reward model bias, computational costs of long video and high-resolution documents, capability conflicts among multiple tasks, and hallucination control in real business scenarios. In the future, more fine-grained process supervision, more stable multimodal reward modeling, and more efficient curriculum sampling strategies will be important directions for improving the reasoning capabilities of vision-language models.