Reasoning Training for Open-Source Vision-Language Models: A Full Breakdown

    This article breaks down the full training pipeline behind open-source reasoning-oriented vision-language models (VLMs). It explains why the real bottleneck in multimodal capability is reasoning rather than perception, maps the standard three-stage architecture (vision encoder, projection layer, language backbone), and walks through the data system, pretraining, supervised fine-tuning, reinforcement learning, evaluation, and infrastructure in turn. Each section ties concrete engineering practices—visual token budgets, verifiable rewards, curriculum sampling, format enforcement, and monitoring—to the goal of making the reasoning "process" a supervisable, verifiable, and scoreable object. A practitioner checklist and an honest look at open limitations round out the article.

    I. Where the Problem Starts: Multimodal Models Lack Not “Seeing” but “Reasoning Clearly”

    1.1 Three Capability Layers Must Be Viewed Separately

    When people discuss vision-language models (VLMs), the industry often treats “strong capability” as a single whole. But from a technical attribution standpoint, it needs to be split into at least three layers:

    • Perception layer: object recognition, text recognition, layout structure, region localization. This layer can be tested with closed tasks and closed benchmarks, with clear metrics that saturate quickly.
    • Understanding layer: cross-modal coreference resolution between images and text, chart reading, intent judgment. It requires combining visual evidence with language priors, but usually completes in one to two steps.
    • Reasoning layer: multi-step derivation, hypothesis-and-verification, answers that depend on intermediate conclusions. Typical forms include competition-style math problems, cross-reference question answering over long documents, and software-interface action-sequence planning.

    A widely observed phenomenon across public benchmarks is that on perception tasks mainstream models have converged and are close to each other, and increasing parameter count yields little marginal benefit; but on tasks that require multi-step derivation, there remains a clear gradient between different models. This shows that the bottleneck in reasoning capability lies not in the resolution of visual encoding, but in whether the training objective explicitly supervises the “process”. Treating “increase input resolution” or “simply enlarge backbone parameters” as a universal solution is a form of attribution error.

    1.2 The Two Legs on the Text Side: Chain-of-Thought and Reinforcement Learning

    Over the past couple of years, the reasoning-capability leap in text LLMs has followed a repeatedly validated method path that can be summarized as two legs:

    • Chain-of-Thought (CoT) supervision: having the model externalize intermediate steps before producing the answer. The value is not merely “writing a few more words,” but decomposing a complex mapping that originally had to complete in a single forward pass into multiple autoregressive conditional generations—effectively increasing the depth of computation.
    • Reinforcement learning (RL): using answer correctness as a reward signal to let the model explore reasoning paths on its own. It does not need to demonstrate “how to write,” only to judge “whether it is right,” so it can break through the ceiling set by demonstration data.

    The relationship between the two is not replacement but a relay: supervised fine-tuning defines the output format, reasoning granularity, and capability floor; reinforcement learning raises the ceiling, compresses ineffective thinking, and corrects bad habits. This relay structure is also the core methodology of the open-source reasoning-oriented VLMs analyzed here.

    1.3 Five Additional Difficulties in the Multimodal Setting

    Directly porting the text-side recipe to VLMs fails, for the following reasons:

    1. Perception errors amplify along the reasoning chain. Misreading one character at the first step means every subsequent step is built on a false premise, and the final answer must be wrong. This link does not exist in the pure-text setting.
    2. Language priors can override visual evidence. Some models can “answer correctly without looking at the image” because they guessed the answer from language patterns in the question. Once such samples are selected by outcome rewards, they reinforce the bad habit of “ignoring visual input.”
    3. Rewards are hard to determine automatically. Free-form descriptions and question answering lack natural verifiers, yet reasoning-oriented training depends heavily on automatically judgeable rewards.
    4. The context budget is tight. A long CoT stacked on top of visual tokens can easily push sequence length to tens of thousands, sharply increasing memory and throughput pressure.
    5. Image-text data with process is scarce. High-quality “image + reasoning process + answer” triples can hardly be labeled at scale by humans, so they must rely on programmatic construction and filter-based distillation.

    The technical approaches that follow are, in essence, point-by-point responses to these five difficulties.


    II. Model Architecture: The Three-Stage Alignment Structure and Its Trade-offs

    2.1 Vision Encoder → Projection Layer → Language Backbone

    The architecture of mainstream open-source VLMs has converged into three stages:

    1. image / video frames
    2. ↓
    3. vision encoder (Vision Transformer, emits patch-level features)
    4. ↓
    5. projection layer (usually two to three MLP layers for dimension alignment and semantic bridging)
    6. ↓
    7. language backbone (decoder-only LLM, autoregressive over concatenated vision tokens)
    8. ↓
    9. text token sequence (including reasoning process and final answer)

    Three points are worth emphasizing:

    • The projection layer typically accounts for a tiny share of parameters (often around 1% of the total), but alignment quality is extremely sensitive to it. Its job is to “translate visual features into language space,” and the strategy of freezing the backbone in early pretraining exists precisely to ensure this layer can converge stably.
    • The language backbone determines the reasoning ceiling. A reasoning-oriented VLM and a perception-oriented VLM may be almost identical in architecture; the difference mainly comes from the backbone’s own reasoning ability and whether post-training genuinely did reasoning optimization.
    • Vision tokens and text tokens participate on equal footing in attention computation within the sequence, which is the prerequisite for end-to-end joint reasoning. If instead a two-stage pipeline is used—“describe into text first, then reason”—a large amount of spatial and layout information is lost during the description stage.

    2.2 Visual Token Budget: The Direct Knob for Precision

    Resolution and token count are two sides of the same coin. A common engineering practice is to allocate budget by task family:

    Task family Recommended bias Reason
    Documents, tables, formulas, small-text OCR High resolution, multi-crop Detail decides success or failure; one wrong character means everything is wrong
    Charts and scientific reasoning Medium-high resolution + local cropping Requires both reading precision and overall trend
    Video, long documents Low resolution, frame sampling Context budget is a hard constraint
    Interface understanding and localization Medium resolution, preserve coordinate mapping Requires precise selection and coordinate consistency

    One principle must be upheld: the resolution strategy at training and inference must be consistent. If the cropping scheme seen during pretraining, the preprocessing normalization parameters, or the coordinate-normalization interval are changed on the inference side, localization-type capabilities will exhibit systematic drift.

    2.3 Engineering Implementation Essentials

    • Variable-length sequence packing: image-text samples vary greatly in length, and padding to a batch wastes substantial compute. They should be sorted by length and packed into buckets, with attention masks constructed correctly to avoid cross-sample information leakage.
    • Long-sequence parallelism and memory optimization: sequence parallelism, gradient checkpointing, and activation recomputation are nearly mandatory in long-CoT scenarios.
    • Injecting temporal information for video and multi-image inputs: beyond uniform sampling, weight key frames and inject timestamps either as explicit text or as position encodings; otherwise the model struggles to distinguish the order of actions.

    III. The Data System: Capability Ceiling Is Determined by Data

    The data engineering for reasoning-oriented multimodal models typically covers six categories. They are not parallel but rather hierarchical and progressive.

    3.1 Image Caption Data

    The job is to establish the most basic image-text alignment. A common misconception is to pursue only “describing the subject clearly,” whereas reasoning requires higher information density: object attributes, counts, spatial relations, visible text, and the meaning of chart axes should all be covered. The granularity of captions directly determines how rich the evidence available for later reasoning is.

    3.2 Interleaved Image-Text Data

    In documents, web pages, and tutorial corpora, images and body text appear alternately. Its value lies in training the model to locate evidence within long contexts and understand the coreference between image and text—a direct prerequisite for long-document QA and chart reasoning. When constructing it, preserve the original relative positions of images and text, and do not degrade it into a “two-segment concatenation with images and text separated.”

    3.3 Optical Character Recognition (OCR) Data

    It covers scene text, documents, tables, formulas, receipts, and more. OCR is a prerequisite for many reasoning tasks: one digit read wrong, and all subsequent computation must be wrong. Training essentials include layout reconstruction (reading order), table structure (cell membership), and linearizing formula expressions. The format specification for OCR output must be strictly unified, otherwise it will pollute downstream parsing logic.

    3.4 Visual Grounding Data

    It expresses “find a certain thing” as a bounding box, point, or mask, and binds it to a language reference. It brings two benefits: first, it is automatically verifiable (compute IoU against the ground-truth box to score), so it naturally fits reinforcement learning; second, it forces the model to anchor its conclusions to specific visual regions during reasoning, which directly helps suppress visual hallucination.

    3.5 Video Data

    This includes temporal actions, event causality, shot transitions, and joint understanding of subtitles and audio tracks. The difficulty lies in the frame-sampling strategy: uniform sampling may miss key moments, so key-frame detection or scene segmentation is needed, along with an upper bound on the temporal span being sampled.

    3.6 Instruction-Tuning Data

    It organizes the above capabilities into task forms users can consume, and deliberately controls the difficulty distribution. If instruction data is entirely easy samples, the model will saturate quickly in the reinforcement-learning stage with no room left to improve.

    3.7 Five Gates of Data Governance

    1. Deduplication: beyond exact deduplication, also do near-duplicate and cross-modal near-duplicate removal (the same image at different compression levels).
    2. Quality grading: score samples on resolution, sharpness, image-text relevance, and annotation completeness; down-weight low-scoring samples rather than discarding them outright.
    3. Annotation consistency: within the same task family, output fields, units, coordinate systems, and language must be unified; a small validation set can be used to measure consistency.
    4. Difficulty calibration: run the current model over the data once and write the first-pass accuracy back into the metadata as a difficulty label. This step is the prerequisite for subsequent curriculum sampling—low cost, high return.
    5. Benchmark contamination screening: run near-duplicate comparisons against evaluation sets, paying special attention to covert contamination like “the same image from the same source + a rephrased question.”

    IV. Pretraining: Packing World Knowledge into the Multimodal Backbone

    The typical approach proceeds in three stages:

    1. Alignment pretraining: freeze the language backbone and train only the projection layer, using large-scale image-text pairs to pull visual features into language space. In this stage, the “regression test” cost for language capability is lowest.
    2. Joint pretraining: unfreeze all parameters, mix multiple sources—image-text pairs, interleaved image-text, OCR, grounding—and scale up data and task coverage. The most critical elements here are data mixing ratios and learning-rate scheduling: too high a learning rate will break the backbone’s general language ability, causing catastrophic forgetting.
    3. Long-context and long-video continued pretraining: at a lower learning rate, extend the processable sequence length and add video and long-document data.

    The practical recommendation is to keep a lightweight regression evaluation suite (subject knowledge, common sense, instruction following) throughout, and run it at every checkpoint. If gains in reasoning capability come at the cost of degraded basic language ability, it is usually not worth it in a real product.


    V. Supervised Fine-Tuning: Getting the Model to “Write Its Reasoning Out”

    5.1 Three Sources of High-Quality CoT Data

    • Human-authored: small in quantity and costly, its role is to define the format and reasoning depth; typically a few thousand examples suffice to produce a marked convergence in format.
    • Strong-model distillation + strict filtering: low cost and large scale, but filtering is mandatory. Note that a correct answer does not mean a correct process—the model may guess the answer right while using flawed reasoning. A viable approach is to check simultaneously: answer correctness, consistency between steps and cited evidence, and the absence of self-contradiction.
    • Programmatic generation: for tasks like math, OCR, grounding, charts, and interface actions where results are automatically verifiable, it is easiest to construct “standard answers with steps” at scale, and it should serve as the main data source for reinforcement learning.

    5.2 A Unified Response Format

    The value of a unified format is amplified in the reinforcement-learning stage: the reward function needs to parse model output stably, and any format drift will distort the reward signal. A schematic schema:

    1. <think>
    2. 1) Observe the image: list visual evidence relevant to the question (with region coordinates/reference positions)
    3. 2) Extract information: read values, recognize text or structure
    4. 3) Derive: compute/compare/eliminate step by step, stating the basis for each step
    5. 4) Self-check: verify units, criteria, and the conditions given in the problem
    6. </think>
    7. <answer>
    8. Final answer (strictly following the format required by the task, e.g. option letter/numeric value/JSON field)
    9. </answer>

    On the engineering side, pair this with a parsing validator that uniformly checks format legality before data enters the store and before training, keeping format-error samples out of the training set.

    5.3 Quality Governance and Iterative Enhancement

    • Rejection-sampling rewrites: sample the same question multiple times with the current SFT model, keep the one with a correct answer and the most self-consistent process, and use it to replace the original distilled data. This is a high-leverage, low-cost operation for improving data quality.
    • Difficulty-stratified sampling: combining the difficulty labels from Section 3.7, ensure each training batch contains both stable signal and traction signal.
    • Preserve the “uncertain” capability: explicitly construct samples of “insufficient information, cannot decide,” and allow the model to abstain or request more information. Reasoning models lacking this link show noticeably higher hallucination rates on open-domain questions.

    VI. Reinforcement Learning: From “Can Reason” to “Reasons Correctly”

    6.1 Reward System Design

    Reward design is the key to the success or failure of this kind of work. It is advisable to layer it by the “verifiable-first” principle:

    Task type Primary reward Verification method
    Math and science computation Answer correctness Numerical/expression equivalence checking (watch for equivalent transformations)
    Multiple choice and short answer Exact match String comparison after normalization
    OCR / document extraction Partial match Edit distance, field-level matching, F1
    Visual grounding Localization quality IoU threshold
    Interface action / agent Task completion Environment-state judgment or step-level process reward
    General description Auxiliary signal Format compliance, evidence-citation consistency

    Auxiliary rewards must be used cautiously: format rewards must be capped and decay over training, otherwise the model degrades into “writing the format first and filling in the answer afterward.” Also add anti-cheating constraints, such as penalizing high-overlap verbatim repetition of the question text and penalizing contradictions between the reasoning section and the answer section.

    6.2 Curriculum Sampling

    Training reinforcement learning with a fixed difficulty distribution wastes substantial compute: too-easy problems give no gradient signal (all correct or all wrong, so the within-group advantage is zero), while too-hard problems are too noisy. The core idea of curriculum sampling is to dynamically rebalance by real-time accuracy, keeping the model working in a “learnable” zone:

    1. # Illustrative pseudocode
    2. buckets = {} # difficulty_id -> deque of (prompt, group_acc)
    3. def sample_batch(buckets, model, batch_size, ratio=(0.3, 0.4, 0.3)):
    4. # ratio: (simple, medium, hard) target mix
    5. picked = []
    6. for bucket, quota in zip(("easy", "medium", "hard"), ratio):
    7. pool = [p for p in buckets if bucket_of(p, model) == bucket]
    8. picked += weighted_sample(pool, n=int(batch_size * quota))
    9. return picked
    10. def update_buckets(buckets, rollouts):
    11. for prompt, g in group_by_prompt(rollouts):
    12. acc = mean([r.reward > threshold for r in g])
    13. buckets[prompt] = classify(acc) # acc≈1 -> easy, 0.2~0.8 -> medium, ≈0 -> hard
    14. # down-weight hard samples with no long-term progress, to avoid gradients being dominated by noise

    Two practical points on implementation: first, difficulty must be estimated in real time against the current policy and refreshed periodically, because static difficulty labels expire quickly as training progresses; second, a certain proportion of easy samples must be retained, or the model will lose basic capabilities it had already mastered during training.

    6.3 Optimization Details

    • Group-relative advantage: sample multiple responses to the same prompt and compute the advantage using the group’s mean reward as the baseline, which eliminates the need for a separate critic model and significantly reduces memory and implementation complexity. The group size needs a trade-off: too small and the baseline is noisy, too large and per-step throughput drops.
    • KL constraint and entropy control: the KL constraint prevents the policy from drifting too far from the reference model, which would cause language degradation; entropy regularization prevents premature convergence to a single templated reasoning path.
    • Handling long CoT: set an upper limit on generation length, and give truncated samples a neutral or slightly negative reward, to avoid the model learning to “write long on purpose to dodge judgment.”
    • Numerical stability: log-probability computation should use a numerically stable implementation; in mixed-precision training, watch for probability underflow and clip ratio explosions.

    VII. Evaluation: How to Tell Whether Reasoning Capability Really Improved

    7.1 Layered Metric Reporting

    Reporting a single average score has almost no diagnostic value. It is recommended to report separately by capability layer:

    • Perception layer: text-recognition accuracy, localization IoU, description consistency;
    • Understanding layer: subject-knowledge and chart-understanding benchmarks;
    • Reasoning layer: multi-step math, long-document cross QA, interface-action tasks;
    • Process quality: format-compliance rate, average reasoning length, abstention rate.

    7.2 Cross-Task Mutual Promotion

    One phenomenon repeatedly observed in this kind of work deserves attention: improvements in grounding capability drive chart and document reasoning, math CoT training improves structured expression, and interface-action training improves action-sequence planning. A reasonable explanation is that these tasks share the same meta-strategy—“decompose the problem, gather evidence item by item, verify step by step, and consolidate the conclusion.” This also suggests that data mixing need not strictly correspond to the surface form of the target task; the transferability of basic reasoning strategies matters more than task labels.

    7.3 Common Evaluation Pitfalls

    1. Data contamination: near-duplicate detection between the training corpus and evaluation sets must be routine.
    2. Format sensitivity: whether a thinking section is allowed to be output, and differing output-length limits, can cause substantial score differences. The evaluation protocol must be fixed and public.
    3. Prompt variance: the fluctuation of the same model under different prompt templates can exceed the gap between models, so report results across multiple templates or confidence intervals.
    4. Judge-model bias: when using a model as the judge, check its preference for long answers and particular formats.
    5. Comparing only absolute scores: when comparing against models an order of magnitude larger in parameter count, “per-parameter performance” and the “reasoning-cost vs. benefit curve” are more informative.

    7.4 Conclusions Should Be Attributable

    If one can only say “it went up overall,” engineering cannot proceed further to optimize. The ideal state is to attribute gains to specific components: what a data-mixing adjustment brought, what a change in one reward-function weight brought, what curriculum sampling brought relative to uniform sampling. One of the important contributions of this kind of work is precisely the actionable experience it provides on the relative contributions of the trio “data quality, reward precision, and curriculum strategy.”


    VIII. Engineering Deployment: Infrastructure for Training and Inference

    • Storage and data supply: large-scale image-text-video corpora usually depend on both object storage and a high-performance parallel file system; shard formats, random reads, and prefetch strategies must be designed well so that data loading does not become the bottleneck.
    • Job scheduling and fault tolerance: long-cycle training must support checkpoint resume, faulty-node isolation, and automatic restart; it is recommended to rely on a container platform to manage heterogeneous training and inference resources uniformly.
    • Observability: logging services and monitoring alerts should cover the following core curves; missing any one of them can leave a training run “drifting without anyone noticing”:
    Monitoring item Meaning of the anomaly signal
    Mean policy reward A reward collapse usually points to the reward function being hacked
    Entropy Monotonic drop that is too fast → premature policy convergence, loss of diversity
    KL divergence Continuously rising → risk of language-ability and generality degradation
    Gradient norm Spikes → learning-rate or numerical-stability issues
    Ratio of all-correct/all-wrong within a group Too high → curriculum sampling has failed, insufficient effective gradients
    Format-compliance rate Declining → output structure starting to drift
    • Inference-side optimization: vision-token caching and prefix reuse, continuous batching, routing low-difficulty samples through a short reasoning path, and distilling a large model’s reasoning ability into smaller models are all effective means of controlling inference cost.
    • The data flywheel: online hard cases and user feedback flow back, and after annotation and filtering enter the next training round. The capability improvement of reasoning-oriented models depends heavily on how fast this flywheel turns.

    IX. Limitations and Outlook

    Objectively, this technical path still has clear open problems:

    1. Visual hallucination is not eradicated. Language priors can still dominate conclusions, requiring stronger consistency constraints (such as forced region citation and reasoning-evidence alignment checks).
    2. Context efficiency for long video and long documents. Current approaches are still limited by attention overhead under very long contexts; fusing sparse attention, memory compression, and retrieval augmentation is a viable direction.
    3. Open tasks lack verifiable rewards. Description, aesthetics, and creative tasks are hard to score automatically—this is the main blind spot of reinforcement learning.
    4. The cost of test-time compute scaling. Letting the model “think more” can raise scores, but latency and compute cost grow nonlinearly; how to do adaptive thinking length on the product side is a core engineering problem.
    5. The reasoning ceiling of small models. 9-billion-parameter-class models can already compete head-to-head with larger ones on some tasks, but extremely difficult multi-step derivation still has a ceiling, and the combination of distillation with inference-time search still has considerable room for exploration.

    X. A Practical Checklist for Practitioners

    • Architecture: confirm that the visual token budget is allocated by task family, and that preprocessing at training and inference is strictly consistent.
    • Data: first complete the six basic data categories, then do difficulty calibration and contamination screening; programmatically constructing verifiable tasks is the highest-value-for-cost investment.
    • SFT: human data defines the format, programmatic data scales it up, and rejection sampling improves quality—do not reverse this order.
    • RL: prioritize rewards for verifiable tasks, cap and decay format rewards, and keep a proportion of easy samples in curriculum sampling.
    • Evaluation: layered reporting + fixed protocol + process-quality metrics, to avoid being misled by a single average score.
    • Operations: turn reward, entropy, KL, and format-compliance rate into a real-time dashboard, with alerts on any anomaly.

    To summarize the methodology of this kind of work in one sentence: improvement in multimodal reasoning capability does not depend on how deep the vision encoder is, but on whether you have turned the “process” into an object that is supervisable, verifiable, and scoreable.