Training Multimodal Reasoning Models: From Pretraining to Curriculum Reinforcement Learning

    This article outlines a practical training path for vision-language models, covering large-scale multimodal pretraining, dynamic resolution handling, curriculum sampling for reinforcement learning, difficulty scheduling, hybrid reward design, trustworthy evaluation, and engineering risk controls. It provides starting parameters and a developer checklist that teams can adapt to their hardware, data scale, and target tasks.

    Vision-language models are moving from image-text alignment toward reasoning-centered general multimodal capabilities. Training such models usually follows two stages: first, build a solid visual foundation through large-scale pretraining; then, unlock reasoning ability with reinforcement learning that uses curriculum sampling. This article follows the training workflow and gives actionable starting parameters in each section. The values here are starting points and should be adjusted for your hardware, data scale, and tasks.

    1. How Should the Pretraining Stage Build the Visual Foundation?

    The goal of pretraining is to help the model capture both pixel-level detail and high-level semantics while aligning visual features with the language space. Data usually comes from three sources: image-text pairs, video clips, and document scans. A reference starting mix is about 50% image-text pairs, 25% document scans, 15% video frames, and 10% synthetic data and other sources. The mix should be written into the configuration file, and the random seed should be recorded; otherwise, later experiments cannot be compared reliably.

    Common training objectives include contrastive learning and masked reconstruction. Contrastive learning pulls matching image-text features closer and pushes mismatched features apart. Masked reconstruction randomly masks some image patches and asks the model to recover them from context. The two can be combined: the former is more effective for semantic alignment, while the latter helps preserve detail.

    Resolution handling is one of the most error-prone parts of this stage. Fixed-size cropping can lose information from document tables and small fonts, so most approaches use dynamic resolution. A common method is to preserve the original aspect ratio, split the image into fixed-size patches such as 14×14 or 16×16 pixels, and cap the maximum number of pixels per image. A typical setting limits the longest side of a single image to between 1024 and 2048 pixels while also setting an upper bound on the number of visual tokens. Very large images can be split into local tiles plus a thumbnail of the full image, all fed into the model together. Each doubling of resolution increases the number of visual tokens by roughly four times, and memory usage and inference latency rise accordingly, so the resolution cap should be chosen together with the memory budget.

    Dependency versions need to be pinned. It is advisable to lock the versions of the deep learning framework, tokenizer, and visual encoder, and document them in experiment records. Otherwise, the same code can produce very different results across environments.

    After pretraining, the model can already recognize objects, describe scenes, and answer simple questions. Its performance on tasks requiring multi-step logic is often still limited. Therefore, a post-training stage is almost always necessary. If you only need a small number of reasoning tasks in a narrow domain, you can first try supervised fine-tuning with a small amount of chain-of-thought data, then decide whether reinforcement learning is needed.

    2. How Should Curriculum Sampling Reinforcement Learning Arrange Difficulty?

    Curriculum sampling can be divided into three parts: difficulty scheduling, hybrid rewards, and variance control. Difficulty scheduling decides which problems are used at each training step, hybrid rewards decide what feedback the model receives, and variance control keeps updates stable.

    Difficulty labeling is the first step. Sample each problem eight times with the base model and count the proportion of correct answers. Problems with a 100% pass rate are labeled easy, those with 1% to 99% are labeled medium, and those with 0% are labeled hard. This method is less labor-intensive than manual labeling and reflects the model’s own capability boundary.

    During training, adjust the proportions of the three difficulty groups by stage. The following is a starting three-stage table:

    Stage Training-step share Easy Medium Hard
    Stage 1 0% to 20% 70% 25% 5%
    Stage 2 20% to 60% 40% 40% 20%
    Stage 3 60% to 100% 20% 40% 40%

    There are two conditions for stage transitions. One is switching by step count, for example, moving to the next stage every 2,000 steps. The other is switching by performance, for example, entering the next stage only when the recent average reward exceeds 0.7. The two can also be combined, with a minimum number of steps in each stage to avoid moving to hard problems too early. If you want the proportions to change smoothly, use linear interpolation instead of segmented switching.

    The following simplified pseudocode for the training loop shows how the parts connect:

    1. for step in range(T):
    2. ratio = schedule(step) # Problem proportions for the current stage
    3. batch = sample_pool(pool, ratio) # Sample problems by difficulty bucket
    4. rollouts = policy.generate(batch, n=8) # Generate 8 responses per problem
    5. rewards = hybrid_reward(rollouts) # Mix rule-based rewards and model scores
    6. adv = group_normalize(rewards) # Relative advantage within each group
    7. loss = policy_loss(adv) + beta * kl(policy, ref_policy)
    8. optimizer.step(loss)

    The key to group-relative policy optimization is comparison within a group. If all eight responses to the same problem receive exactly the same reward, all advantages are zero, and the group contributes nothing to the update. Therefore, if the proportion of hard problems is too high, many samples will be entirely wrong; if the proportion of easy problems is too high, many samples will be entirely correct. Both cases waste compute. During training, monitor the proportion of effective samples in each batch. When it falls below half, adjust the difficulty distribution.

    The KL constraint coefficient is another parameter that needs tuning. A starting value can be between 0.01 and 0.04. If the coefficient is too small, the policy may drift too far from the reference model; if it is too large, the policy barely updates.

    This method applies to tasks whose answers can be verified or have clear scoring criteria, such as math problems, code execution, and localization tasks with annotated boxes. For open-ended questions without standard answers, difficulty labeling for curriculum sampling requires manual work or assistance from a stronger model, which is significantly more costly.

    3. Which Reward Design Approach Should Be Used?

    The reward signal should reflect both the correctness of the final answer and the soundness of intermediate reasoning. Three common approaches have different strengths and weaknesses, as shown below:

    Approach Typical method Advantages Disadvantages Applicable conditions
    Rule-based reward Numeric matching, code execution results, format checks Verifiable and less prone to bias Covers only tasks with standard answers; the model may find loopholes in the rules Math, code, localization
    Model scoring Use a language model to score reasoning process or answer quality Covers open-ended questions and reasoning processes Biased, can be manipulated, and inference cost is high Video description, document QA
    Hybrid reward Weighted combination of rule-based rewards and model scores Balances verifiability and coverage Weights require repeated tuning Most reasoning tasks

    One way to write the hybrid reward is R = 0.7 × R_answer + 0.1 × R_format + 0.2 × R_score. These weights are only a starting point. Format checks should act as a gate: if the output format is invalid, assign a score of 0 directly rather than deducting only a small amount. Otherwise, the model will learn to use fixed templates to collect partial credit.

    A process reward model can score each reasoning step, providing denser signals than final outcomes and helping alleviate sparse rewards. However, its labeling cost is high, and the policy may learn to cater to the process scorer’s preferences, producing superficially complete steps while the actual reasoning is incorrect. Therefore, process rewards should be used together with final-answer rewards and periodically checked by human sampling.

    4. How Can Evaluation Be Made Trustworthy?

    Evaluation of multimodal reasoning should cover several independent dimensions. Science and engineering questions focus on formula derivation and chart interpretation. Video understanding focuses on temporal causality and event localization. Content recognition focuses on fine-grained objects and text extraction. Programming tasks require generating runnable code from screenshots or flowcharts. Visual grounding requires precise bounding boxes or pixel-level masks. GUI agents require understanding screen elements and planning operation sequences. Long-document understanding examines cross-page information integration and citation accuracy.

    Every evaluation result should include a complete evaluation protocol, covering at least the following: benchmark name and version, metric definition, sample size, whether tool use is allowed, decoding parameters such as temperature and maximum generation length, and number of repetitions. For the same test set, run three random seeds and report the mean and standard deviation. Reporting only a single run makes it difficult to determine whether the gap between two models reflects real capability.

    In addition to accuracy, report reasoning-chain completeness, average generation length, and end-to-end latency. Reasoning-chain completeness is usually judged by humans or models, and the judging criteria must be stated. If generation length is not recorded, models may inflate scores by producing longer text, especially in later training stages.

    Comparative experiments need to distinguish two cases. When comparing models of the same parameter scale, focus on stability on medium-difficulty benchmarks. When comparing a small model with a large model, specify which challenging subsets the small model surpasses the large model on after reinforcement learning, and confirm that the large model used the same prompt template and decoding settings. Otherwise, the conclusion does not hold.

    Data isolation is a prerequisite for evaluation. Training and test sets need deduplication checks: text can be checked with n-gram overlap, and images can be checked for near-duplicates with perceptual hashing. The test set should ideally include adversarial samples, such as images with occlusion, blur, noise, or text interference, to test whether the model truly understands content rather than relying on surface features.

    5. Engineering Deployment and Risk Control

    The reinforcement learning stage has high memory and communication bandwidth requirements. Common cost-reduction methods include gradient checkpointing, bf16 mixed-precision training, and placing rollout generation in a separate inference engine for asynchronous execution. This allows training and sampling to run in parallel, improving GPU utilization.

    For inference deployment, quantization and speculative decoding are two common acceleration methods. INT8 or INT4 quantization can reduce memory usage, but long chain-of-thought and numerical computation are sensitive to precision; after quantization, calculation errors or broken reasoning chains may appear. Before release, compare results before and after quantization on your own evaluation set. Speculative decoding relies on a small draft model to predict tokens, which are then verified by the target model. If the draft model’s output distribution differs greatly from the target model’s, the acceptance rate will be low, making it slower than ordinary decoding. Therefore, measure the acceptance rate first, then decide whether to enable it.

    Three types of risk need close monitoring during training. The first is reward hacking: the model may learn to output fixed formats or specific keywords to game the reward, shown by continuously rising reward while validation accuracy falls. The second is policy entropy collapse: outputs gradually become uniform and exploration ability is lost. The third is data compliance risk: image-text data and document scans need authorization confirmation, and data containing faces, identity documents, or personal information needs anonymization.

    Monitoring thresholds can be set as starting points. Trigger an alert when policy entropy falls below 20% of its initial value. Pause training and inspect the reward function when reward rises for 500 consecutive steps but validation accuracy drops by more than 2 percentage points. Check whether the model is gaming length when average output length grows by more than 50% within a week. These thresholds should be adjusted according to the fluctuation range of the task.

    6. Developer Checklist

    1. Write the data mix into the configuration file, and record random seeds and data versions.
    2. Make the curriculum schedule and stage-transition conditions reproducible in configuration, and record the actual number of samples used in each stage.
    3. Complete deduplication checks between training and test sets, and record the methods and results.
    4. Include at least one set of adversarial samples in the test set, such as occluded, noisy, or blurred images.
    5. Log each reward component separately so problems can be traced to their source.
    6. Validate the curriculum sampling strategy on a small-scale model first, then scale up after confirming effectiveness.
    7. Confirm the source and usage authorization of each type of training data, and complete privacy review.

    Data quality, curriculum arrangement, and reward design often determine training results more directly than simply increasing parameter count. By implementing the steps above item by item, a medium-scale model can also achieve competitive results on multiple public benchmarks and provide a reliable foundation for later deployment in vertical scenarios.