Building and Training Reasoning Capabilities in Vision-Language Models: A Practical Guide

    This guide walks through the end-to-end process of building a reasoning-centric vision-language model (VLM), covering data, architecture, training, and evaluation. It explains how VLMs progress from perception to reasoning, breaks down the common ViT-MLP-LLM architecture with practical parameter ranges, and details data engineering practices such as mixing ratios, deduplication, quality filtering, and compliance. It then covers pre-training, supervised fine-tuning with chain-of-thought (CoT) data, and reinforcement learning with curriculum sampling (RLCS), including reward design, sampling pseudocode, and training hyperparameters. Finally, it discusses evaluation methodology, cross-domain generalization, and a practical implementation checklist.

    1. Background: From Perception to Reasoning, How Do VLMs Learn to Reason?

    The development of vision-language models (VLMs) in recent years has gradually shifted from “describing images” to “reasoning over images.” If we break down capabilities by level, they can roughly be divided into three categories:

    • Perception: Recognizing what is in an image, such as object detection, text recognition, and region localization.
    • Understanding: Building semantic relationships on top of perception, such as judging scene relationships and answering attribute questions.
    • Reasoning: Performing multi-step logical deduction on top of understanding, such as solving math problems, root-cause analysis, and cross-modal evidence-chain inference.

    Many traditional VLMs perform well at the perception and understanding levels, but their performance drops noticeably once they encounter multi-step reasoning, cross-modal evidence integration, or long chains of logic. Therefore, the key to building a reasoning-centric multimodal model is not simply piling on parameters, but connecting “data—architecture—training—evaluation” into a closed loop.

    From an engineering perspective, common long-tail questions include: How do you train a VLM for reasoning? How should RL curriculum sampling select difficulty levels? How do you filter CoT data quality? These three questions correspond respectively to the training method, the reinforcement learning strategy, and data quality governance.

    2. Model Architecture: How Does ViT-MLP-LLM Connect Vision and Reasoning?

    Most mainstream general-purpose VLMs today adopt a ViT-MLP-LLM structure, whose design philosophy is to reuse strong existing components and only train the connecting and alignment parts.

    1. Image → Vision encoder ViT → Projection layer MLP → Large language model LLM → Text output
    2. Text ──────────────────────────────────────┘

    1. Responsibilities of Each Module

    • Vision encoder ViT: Splits the image into patches and encodes them into a sequence of visual tokens, responsible for extracting low-level visual features.
    • Projection layer MLP: Maps visual features into the word-embedding space of the language model to achieve modality alignment.
    • Large language model LLM: Receives the mixed sequence of visual tokens and text tokens and autoregressively generates responses.

    2. Engineering Boundaries and Parameter Recommendations

    To make the architecture description more actionable, the following boundaries need to be addressed:

    • Resolution: Low resolution suits general image understanding; high resolution suits OCR, document understanding, and fine-grained localization. Common engineering ranges to start testing from include 448, 672, and 1024.
    • Number of visual tokens: More visual tokens preserve more detail but increase computational cost. For a single image, it is recommended to make trade-offs between 256 and 4096 tokens; if dynamic resolution is used, avoid letting a single image’s token count exceed what GPU memory can handle.
    • Projection layer dimension: The projection layer output dimension usually needs to align with the LLM hidden dimension, for example 4096, 5120, or 8192. The projection layer should not be too deep, otherwise it increases training cost.
    • Training stages: Generally, freeze ViT first and train the MLP or only a lightweight alignment layer; then unfreeze the LLM, or use parameter-efficient fine-tuning methods such as LoRA.

    3. Alternatives

    Beyond the classic MLP projection, you can also consider structures such as Q-Former and Resampler. Their advantage is that they can compress a large number of visual tokens into fewer, more controllable tokens, but they are more complex to implement and need to be evaluated against data scale and task type.

    3. Data Engineering: How Do You Build VLM Reasoning Data?

    Data quality often determines the final results more than model structure does. A general-purpose multimodal reasoning model usually requires the collaboration of multiple data sources.

    1. Example Data Mix

    A practical initial mix could be:

    Data type Suggested share Purpose
    Image-caption data 30% Establish basic image-text alignment
    Interleaved image-text data 20% Cultivate long-context narrative understanding
    OCR data 15% Strengthen text recognition and document understanding
    Visual grounding data 15% Support coordinates and region localization
    Video data 10% Introduce temporal and causal understanding
    Instruction and CoT reasoning data 10% Align reasoning format and task types

    This ratio is not a fixed standard and should be adjusted according to the target scenario. For example, when document QA makes up a large share, OCR and layout-understanding data should be increased; when math or chart reasoning makes up a large share, more chart, formula, and table CoT data should be added.

    2. Deduplication and Quality Filtering

    • Text deduplication: Use MinHash or SimHash; if the text Jaccard similarity is greater than 0.9, it is considered highly duplicated.
    • Image deduplication: Use perceptual hashing; if the pHash distance is less than 5, it is considered a near-duplicate.
    • OCR quality filtering: Samples with too-low OCR confidence, severe text corruption, or unrecognizable key fields should be filtered out or down-weighted.
    • Answer verifiability: Math problems should be computable, grounding problems should have ground-truth boxes, and OCR problems should have ground-truth strings; unverifiable samples should not be fed directly into RL.
    • Teacher model consistency: Use a strong model to generate multiple candidate answers; if the consistency rate is below 0.8, they can go to human review or be discarded.
    • Difficulty stratification: Samples can be stratified by pass rate, number of reasoning steps, image complexity, and text length, for example by pass rate 0, 0–0.25, 0.25–0.75, and 0.75–1.

    3. Compliance Checklist

    Multimodal data involves images and text, OCR, video, and user-uploaded materials, so a compliance process must be added:

    • Data authorization: Confirm that training data has legal provenance and usage rights.
    • Copyright filtering: Identify and filter images, documents, and film/video clips that are clearly copyrighted.
    • Privacy anonymization: Anonymize or mask sensitive information such as ID cards, phone numbers, addresses, and faces.
    • Sensitive content review: Filter out violence, pornography, illegal content, infringement, and misleading content.
    • Retention policy: Define data retention periods, deletion mechanisms, access permissions, and audit logs.
    • Cloud security: If using object storage and container platforms, configure access control, least privilege, transmission encryption, and log security.

    4. Pre-training: Building the Visual Foundation and Downstream Interfaces

    The goal of the pre-training stage is to establish a solid visual perception foundation. This stage usually does not pursue complex reasoning, but instead ensures that the model can “see clearly and recognize accurately.”

    Pre-training should focus on:

    • Large-scale image-text alignment;
    • Basic tasks such as OCR, localization, and video understanding;
    • Basic mapping between visual tokens and language space;
    • Providing a stable interface for subsequent SFT and RL.

    The output interfaces of pre-training should include: vision encoder weights, projection layer weights, image-text alignment checkpoints, tokenizer, prompt templates, and the visual token upper-limit configuration. Only with clear interfaces can subsequent SFT and RL reuse the same data and evaluation protocols.

    5. Supervised Fine-tuning: How Do You Filter CoT Data Quality?

    The core of SFT is not to make the model memorize answers, but to teach it a proper reasoning process.

    1. Minimal SFT Data Example

    1. Reasoning: <First identify the key information in the image, then analyze the problem conditions, and finally give the reasoning path>
    2. Final answer: <A concise, unique, verifiable conclusion>

    Example:

    1. Reasoning: First identify the values required by the question, then read the axis scales in the image, and finally compare the two values.
    2. Final answer: A
    3. Verification rule: The final answer must match the ground truth; the reasoning must contain at least two steps; it must not reference non-existent image regions.

    2. CoT Data Quality Filtering Rules

    • Reasoning steps must be logically consistent with the final answer;
    • Steps should be verifiable step by step by rules or by humans;
    • Cover diverse reasoning types such as math, causality, retrieval/localization, document QA, and root-cause analysis;
    • Avoid pseudo-CoT where “the answer is correct but the process is fabricated”;
    • Avoid excessively long, ineffective reasoning.

    6. Reinforcement Learning: How Do You Select Samples for Curriculum-Sampling RL?

    SFT teaches the model a problem-solving paradigm, while RL makes the model stronger through self-exploration. Here it needs to be made clear: RLCS stands for Reinforcement Learning with Curriculum Sampling, a training strategy that combines RL with curriculum sampling, rather than one fixed algorithm. It is usually combined with RL optimizers such as GRPO and PPO.

    1. Reward System Design

    Multimodal tasks are better suited to verifiable rewards than fully subjective scoring.

    Reward type Computation Applicable tasks Common risks
    Answer-rule reward Exact match, numeric match Math, multiple choice May encourage answer guessing
    Format reward Check the structure of reasoning and final answer All SFT/RL tasks May learn format only
    Process-completeness reward Check step count and answer consistency Math, causal reasoning Process rules must be designed carefully
    OCR string reward Edit distance or exact match Document understanding Sensitive to fonts and layout
    Localization reward IoU or center-point distance Grounding Threshold settings affect the distribution

    2. RLCS Curriculum Sampling Pseudocode

    The following code is pseudocode for illustrating the flow; it is not a directly runnable library implementation.

    1. # Pseudocode for RLCS curriculum sampling
    2. # Example input template:
    3. # {
    4. # "image": image,
    5. # "text": "The image has been provided as visual tokens. Question: {question}
    6. # Please answer in the following format: Reasoning: <step-by-step reasoning> Final answer: <conclusion>"
    7. # }
    8. # Stop condition: generate until </Final answer> or reach max_new_tokens.
    9. def build_input(item):
    10. return {
    11. "image": item["image"],
    12. "text": item["prompt_template"].format(question=item["question"])
    13. }
    14. def generate(item, temperature=0.8, max_new_tokens=1024):
    15. text = model.generate(
    16. input=build_input(item),
    17. temperature=temperature,
    18. max_new_tokens=max_new_tokens,
    19. stop="</Final answer>"
    20. )
    21. return text
    22. def parse_response(text):
    23. think = extract_between(text, "<Reasoning>", "</Reasoning>")
    24. answer = extract_between(text, "<Final answer>", "</Final answer>")
    25. return {
    26. "think": think,
    27. "answer": answer,
    28. "ok": think is not None and answer is not None
    29. }
    30. def compute_pass_rate(responses, golden, task_type):
    31. scores = []
    32. for r in responses:
    33. parsed = parse_response(r)
    34. if not parsed["ok"]:
    35. scores.append(0.0)
    36. continue
    37. if task_type == "math":
    38. scores.append(1.0 if exact_match(parsed["answer"], golden) else 0.0)
    39. elif task_type == "grounding":
    40. scores.append(1.0 if iou(parsed["answer"], golden) >= 0.5 else 0.0)
    41. elif task_type == "ocr":
    42. scores.append(1.0 if parsed["answer"] == golden else 0.0)
    43. else:
    44. scores.append(1.0 if parsed["answer"] == golden else 0.0)
    45. return sum(scores) / len(scores)
    46. def curriculum_sampling(model, data_pool, k=8):
    47. samples = []
    48. for item in data_pool:
    49. responses = [generate(item) for _ in range(k)]
    50. pass_rate = compute_pass_rate(responses, item["golden"], item["task_type"])
    51. # Prioritize keeping samples that are sometimes right and sometimes wrong
    52. if 0.0 < pass_rate < 1.0:
    53. weight = 4.0 * pass_rate * (1.0 - pass_rate)
    54. samples.append({
    55. "item": item,
    56. "pass_rate": pass_rate,
    57. "weight": weight
    58. })
    59. return balance_by_difficulty(samples)
    60. def balance_by_difficulty(samples, bucket_edges=(0.25, 0.5, 0.75)):
    61. buckets = {
    62. "(0,0.25]": [],
    63. "(0.25,0.5]": [],
    64. "(0.5,0.75]": [],
    65. "(0.75,1)": []
    66. }
    67. for s in samples:
    68. p = s["pass_rate"]
    69. if p <= bucket_edges[0]:
    70. buckets["(0,0.25]"].append(s)
    71. elif p <= bucket_edges[1]:
    72. buckets["(0.25,0.5]"].append(s)
    73. elif p <= bucket_edges[2]:
    74. buckets["(0.5,0.75]"].append(s)
    75. else:
    76. buckets["(0.75,1)"].append(s)
    77. # Samples can be further drawn by weight, with per-bucket share limited
    78. # to avoid overloading any single difficulty bucket
    79. return buckets

    The core formula can be written as:

    1. pass_rate_i = 1/k * sum(1[verify(y_ij, y_i*) >= threshold])
    2. sample_weight_i = 4 * pass_rate_i * (1 - pass_rate_i)

    The closer pass_rate_i is to 0.5, the more information the sample usually carries; samples that are always right or always wrong contribute limited gradient.

    3. Key Training Parameters and Resource Boundaries

    The following parameters are empirical ranges and need to be adjusted according to model scale, GPU memory, and task type:

    • Visual token upper limit: 256–4096 tokens per image is recommended; high-resolution tasks can increase this locally but need to control the total sequence length.
    • Dynamic resolution: Start testing from three levels: 448, 672, and 1024.
    • Batch size: Each batch can contain 64–256 prompts; if GPU memory is insufficient, first reduce the number of prompts, then increase gradient accumulation.
    • Sampling count per prompt: For the RL stage, k=8–16 is recommended; too small a k makes the pass-rate estimate unstable.
    • Learning rate: SFT can reference 1e-5 to 2e-5; RL is usually lower, referencing 5e-7 to 2e-6.
    • Sampling temperature: 0.6–1.0 is common; too high a temperature causes formatting chaos, while too low a temperature reduces exploration.
    • Resource scale: 7B–9B models can use 8 or more 80GB-class GPUs; for larger models, add more nodes or use distributed training.

    4. Optimization Details

    • Group-relative advantage: Perform relative comparison across multiple sampling results for the same question, reducing dependence on absolute reward calibration.
    • Stability control: Control learning rate, batch size, and sampling temperature, and observe reward variance.
    • Phased progression: First use medium-difficulty samples to establish a stable format, then gradually introduce harder samples.
    • Common mistakes: Using uniform sampling directly, ignoring format rewards, continuing to focus training on always-correct samples, and not setting a parsing stop condition.

    7. Evaluation and Cross-Domain Generalization: Metrics, Benchmarks, and Results Comparison

    When evaluating a multimodal reasoning model, you should not look at a single score alone, but should focus on comparisons within the same scale, comparisons across scales, and task-transfer effects.

    1. Evaluation Criteria

    If discussing that “a 9B-scale model performs notably well,” the evaluation criteria must be specified, otherwise it cannot be verified. A more reliable way to phrase this is:

    • Model scope: Same-scale models such as 2B, 3B, 7B, 8B, 9B, and 13B; for cross-scale comparisons, larger models such as 30B and 70B can be added as references.
    • Benchmark examples: MMMU, MathVista, DocVQA, TextVQA, RefCOCO, MMBench, and so on.
    • Metric examples: accuracy, exact match, ANLS, IoU, pass@1, pass@8, format-compliance rate, and process-correctness rate.
    • Result statements: If there are no public results, write “empirical observations on an internal validation set” rather than directly claiming leadership.

    2. Cross-Domain Generalization

    Reasoning training not only improves single-point scores but can also benefit basic perception tasks. For example, after training on math or chart reasoning data, tasks such as OCR, localization, and video understanding may gain improvements. This shows that reasoning ability helps the model form a more general multimodal understanding framework.

    8. Summary: Implementation Checklist

    When building a general-purpose multimodal reasoning model, the following six-item checklist can be used for a quick self-check:

    Checklist item Key question
    Data Is there high-quality, verifiable, compliant reasoning data?
    Architecture Are the resolution, token count, and projection dimension of ViT, MLP, and LLM clearly defined?
    SFT Is the CoT format unified, and is its logical authenticity verifiable?
    RL Are verifiable rewards used, and is curriculum sampling adopted?
    Evaluation Are benchmarks, metrics, comparison models, and parameter ranges provided?
    Compliance Have authorization, copyright, privacy, sensitive content, and retention management been completed?

    In engineering practice, large-scale training usually relies on container platforms, object storage, distributed training frameworks, and monitoring/alerting. Cloud environments should also pay special attention to access control, log security, data encryption, and compliant retention. For teams that want to validate quickly, they can also prioritize incremental innovation in data and training strategies on top of mature open-source models rather than building a foundation from scratch.