How to Build a General-Purpose Multimodal Reasoning Model: Architecture, Data, and Reinforcement Learning in Practice

    A practical guide to building a general-purpose multimodal reasoning model, focused on the three factors that actually limit it—data compliance, reward precision, and curriculum design. It covers the ViT-MLP-LLM architecture and its alternatives, six-category data preparation with ratios, cleaning thresholds, and compliance gating, the pretraining and SFT loops, and reinforcement learning built on precise reward systems and difficulty-based curriculum sampling. It includes acceptance tables, launch checklists, and rollout/rollback strategies.

    Bottom Line Up Front

    The bottleneck for a general-purpose multimodal reasoning model is usually not parameter count, but three things: whether the data system is compliant and usable, whether the reward signal is precise, and whether the training curriculum matches the model’s current level.

    • Architecture: ViT-MLP-LLM is the engineering default starting point, but not the only option. The choice depends on GPU memory, latency, and whether the LLM natively supports visual tokens.
    • Data: Six categories of data must each carry a ratio, cleaning thresholds, and sampling-based acceptance, and must all pass a compliance gate first.
    • Reinforcement learning: Reward precision determines the direction of reasoning-quality improvement; curriculum sampling determines output efficiency per unit of compute.

    1. Capability Tiers: Define the Goal First, Then the Evaluation Criteria

    Capability Tier Typical Tasks Acceptance Metric Common Failure Mode
    Perception Counting, color/attribute recognition, short-text OCR Counting accuracy, character accuracy Missing small objects, missing small text, digit repetition
    Comprehension Scene/intent judgment, chart reading Multiple-choice accuracy, chart-QA EM Describing only surface elements, getting distracted by irrelevant details
    Reasoning STEM word problems, GUI planning, long-document cross-paragraph reasoning, video causality pass@1, per-step trajectory correctness, manual spot-check Drifting off track mid-way, inconsistent steps, correct answer but wrong process

    Most traditional VLMs stop at the perception and comprehension tiers; the third tier is what is genuinely hard. Define the capability tier before defining the evaluation, or you will get the illusion of “OCR improved but multi-step reasoning didn’t move.”

    Breakthroughs in reasoning ability come mainly from two techniques: first, chain-of-thought (CoT), which makes the model output intermediate steps first and decompose complex problems into verifiable sub-problems; second, reinforcement learning (RL), which uses reward signals to let the model autonomously explore reasoning paths and learn “when to think, how to verify, and when to stop.” The three major shortcomings of existing approaches also correspond exactly to the improvement entry points: insufficient reasoning depth, coarse data utilization, and uneven sub-task capability.

    2. Architecture Selection: How to Choose ViT-MLP-LLM, and What Alternatives Exist

    2.1 Preconditions (Agree Before Jumping to Code)

    • Image input [B, 3, H, W]; when ViT patch=14, N_v = (H/14)*(W/14); under dynamic resolution N_v varies with the image, requiring in-batch padding or grouping by length.
    • ViT output [B, N_v, D_vit] (e.g., D_vit=1024); MLP output [B, N_v, D_llm], where D_llm must match the LLM hidden dimension (e.g., 4096). Dimension mismatches often do not raise errors—they silently degrade.
    • Concatenation is not simply joining head to tail, but replacing visual tokens into the image slot position according to the conversation template, and inserting <image_begin>/<image_end> separators; separators must enter the vocabulary and participate in the attention mask.
    • The loss is computed only on text and answer segments; visual token positions are masked out, otherwise the loss is diluted by the large number of non-generatable visual positions.
    • The following is only data-flow pseudocode, not directly trainable code.
    1. # See preconditions above; this only illustrates the data flow
    2. def forward(image, input_ids, labels):
    3. vision_tokens = vit(image) # [B, N_v, D_vit]
    4. vision_embeds = mlp_projector(vision_tokens) # [B, N_v, D_llm]
    5. text_embeds = llm.embed(input_ids) # [B, L, D_llm]
    6. # Insert visual tokens into image slots per template, not simple concatenation
    7. inputs = replace_image_slots(text_embeds, vision_embeds, sep=IMAGE_SEP)
    8. return llm(inputs, labels=labels) # labels are -100 at visual positions

    2.2 More Than One Path: How to Choose Among Mainstream Alternatives

    Approach Method Suitable Scenario Main Cost
    ViT-MLP-LLM (default here) Project to align, then concatenate into a token sequence General VLM bootstrapping, fast iteration Many visual tokens, context bloat at high resolution
    Cross-attention A small number of learnable queries extract visual information Need to compress visual tokens, latency-sensitive Complex structure and training, fine-grained OCR/localization easily hurt
    Adapter / lightweight side-path Insert adapters within a frozen LLM Tight compute, training only a few parameters Limited capacity, capability ceiling below full joint training
    Native multimodal early-fusion Unify visual and text patches from pretraining Pursuing the strongest unified representation, ample compute High training cost, hard for small teams to reproduce

    Two hard constraints:

    1. The MLP projection layer must train stably: near-identity or scaled initialization, separate warmup, freeze the LLM backbone early on. When loss does not decrease, suspect the alignment layer first, rather than tuning the learning rate first.
    2. The LLM side must support visual tokens: the context window must cover N_v + text length; the attention implementation must support masking of non-text tokens; new separators require vocabulary expansion, otherwise they will be split into multiple tokens and break format compliance.

    2.3 Three High-Frequency Errors

    Using fixed cropping instead of dynamic resolution (small objects and dense text are lost directly); not compressing visual tokens (high-resolution images fill up the context, long-document tasks collapse); only end-to-end training without managing resolution and token budget (short descriptions improve, OCR does not).

    3. Data Preparation: Ratios, Cleaning, and Compliance for Six Data Categories

    3.1 Six Data Categories and Starting Ratios

    Data Category Which Stage It’s For Starting Ratio Sampling Note
    Image-caption Pretraining 40–55% Deduplicate image-text pairs, mix long and short descriptions
    Interleaved image-text Pretraining 20–30% Preserve document order, truncate per document
    OCR Pretraining + SFT 10–15% Balance across languages, layouts, fonts
    Visual grounding SFT 5–10% Bin by bbox area, avoid learning only large objects
    Video Late pretraining + SFT 5–10% Frame count consistent with temporal annotation
    Instruction tuning SFT + RL Mainly SFT Dynamically adjust quota per evaluation weaknesses

    The ratios are starting values and need adjustment based on your business evaluation set weaknesses; it is recommended to do a per-category ablation before and after each training round to confirm “where it improved and where it dropped.”

    3.2 Cleaning Rules and Deduplication Thresholds (Ready-to-Use Starting Parameters)

    • Image-level dedup: pHash Hamming distance ≤ 6 counts as duplicate; text MinHash Jaccard ≥ 0.9 counts as duplicate.
    • Low-quality filtering: drop short side < 100px, JPEG quality factor < 60; drop non-OCR samples with text < 5 words.
    • OCR-specific: drop recognition confidence < 0.7; downweight when watermark/subtitle/sticker covers > 30% of the frame; single-source layout share no more than 15%.
    • Safety filtering: NSFW, violence, and hate content filtered with a two-layer classifier + keyword approach.
    • Source constraint: single-source share capped at 20%, preventing one site’s style from dominating the entire mix.

    3.3 Sampling Acceptance Table

    Check Item Sampling Volume Pass Line Action on Failure
    Image-text consistency (manual) 1000 pairs/batch ≥ 95% Roll back that source
    OCR character accuracy 2000 lines ≥ 98% Relabel or downweight
    Grounding IoU 1000 boxes ≥ 0.80 Notify annotator to retrain
    Video temporal label consistency 500 segments ≥ 90% Reduce frames and relabel
    Privacy/compliance spot-check Full automatic + 1% manual 0 violations Take the batch offline immediately

    3.4 Data Compliance and Privacy (A Step That Can’t Be Skipped)

    • Authorization and licensing: Use only commercially licensed, CC-permitted, or self-collected data; scraping must comply with platform terms and robots, and retain source URLs and batch numbers.
    • De-identification: Blur or replace faces, license plates, ID cards, medical records, and home addresses; use regex + model dual de-identification for real names, phone numbers, emails, and order numbers in OCR.
    • GUI data red lines: Prohibit collecting accounts, passwords, cookies, session tokens, payment info, and real user identity information; recording must use test accounts and synthetic environments.
    • Copyright filtering: Continuous scans of entire books/papers, film/TV posters and screenshots, and watermarked stock images need filtering; long-text OCR is compared against publishing corpora via n-gram.
    • Video and audio: Speech transcription content also goes through privacy screening, and on-screen content in frames must be checked frame by frame.
    • Storage and audit: Encrypted storage, traceable by batch, access logged, and a retention period and deletion process defined.

    It is recommended to make compliance checks a hard gate at ingestion: batches that fail checks do not enter the training pool, rather than doing spot-checks after the fact.

    4. Pretraining: Building the Visual Foundation

    The goal is to complete modality alignment and basic representation learning. The data order is recommended as image-caption → interleaved image-text → OCR → video/localization, gradually increasing granularity. On the freezing strategy, stage one freezes the LLM backbone and trains only ViT and MLP; once alignment stabilizes, unfreeze for joint training. On scheduling, use multi-stage mixing with dynamic ratio adjustment, rather than fixing it once and for all. Each stage evaluates per-category metrics on a fixed hold-out set (no overlap with the training set); if metrics regress, roll back to the previous stage’s weights and re-adjust the ratios.

    5. SFT and the Data Quality Loop

    SFT aligns the “unpolished capability” after pretraining to the human-expected answer format and reasoning habits, especially the organization of chain-of-thought answers.

    5.2 Difficulty Tiering and Iterative Enhancement (Executable Parameters)

    • Difficulty bucketing: Bucket into 5 tiers by the current model’s pass@1 (0–0.2 / 0.2–0.4 / … / 0.8–1.0), refreshed each round or every N steps.
    • Failure reason tags: Format errors, visual misreads, unit/dimension errors, multi-step reasoning breaks, out-of-bounds citations, and over-refusal.
    • Stage switching condition: Only advance tiers when both the hold-out pass rate and format compliance rate are met; stay in the current tier if two consecutive checkpoints fall short.
    • Rollback mechanism: Keep the most recent K checkpoints and corresponding data versions; when a tier’s metrics decline continuously, roll back the data recipe rather than adding more data.
    • Enhancement priority: For format issues, add format examples; for visual misreads, add high-resolution and dense-text samples; for reasoning breaks, add CoT samples with process annotation.

    6. Reinforcement Learning: Precise Rewards and Curriculum Sampling

    SFT teaches the model to answer in a standardized way, and RL teaches the model to think better—this is the key leap from “usable” to “excellent.”

    6.1 Reward System: First Distinguish Closed Tasks from Open Tasks

    1. def compute_reward(prediction, ground_truth, task_type):
    2. if task_type == "closed": # math result, OCR field, bbox, GUI element ID
    3. accuracy = exact_match(prediction, ground_truth) # use IoU for localization
    4. format_ok = check_format(prediction)
    5. return 1.0 * accuracy + 0.1 * format_ok
    6. # Open tasks: description, explanation, report summarization; need three-layer fallback
    7. rule_ok = rule_check(prediction) # format/length/forbidden words/traceable citation
    8. pref = reward_model(prediction, ground_truth)
    9. return 0.3 * rule_ok + 0.7 * pref # high-value samples still go to manual review queue

    Key points:

    • exact_match only applies to tasks with closed, automatically judgeable answers; open tasks must add rule checks, preference models, and manual review.
    • Reward hacking defense: Focus on monitoring two types of divergence—“format score rises but accuracy does not” and “answer is right but process is wrong”—and spot-check the reasoning process.
    • Overfitting monitoring: Only when the training reward and hold-out pass@1 rise together does it count as real improvement; reward rising without hold-out rising counts as overfitting the metric.
    • Fallback strategy: When hold-out declines for N consecutive steps or format compliance falls below a threshold, lower the RL weight and roll back to the SFT checkpoint, and investigate whether the reward is being gamed.

    6.2 Curriculum Sampling: Where Difficulty Comes From, How Often to Update, How to Roll Back

    1. CURRICULUM = {"stage1": (0.0, 0.3), "stage2": (0.3, 0.6), "stage3": (0.6, 0.9)}
    2. ACC = {b: model.pass_rate(b) for b in BUCKETS} # refreshed every REFRESH_EVERY steps
    3. def sample_batch(pool, stage, acc, step):
    4. if step % REFRESH_EVERY == 0:
    5. acc.update(refresh_pass_rate(model, pool))
    6. lo, hi = CURRICULUM[stage]
    7. batch = [s for s in pool if lo <= acc[s.acc_bucket] <= hi]
    8. if len(batch) < MIN_BATCH: # expand to adjacent tiers first when samples are insufficient
    9. batch = nearest_difficulty(pool, hi, need=MIN_BATCH)
    10. return batch
    • Difficulty source: Primarily bucket by the model’s own historical accuracy, with annotation difficulty as cross-validation and manual spot-checks for calibration, avoiding purely subjective labeling.
    • Update frequency: Refresh once every 200–500 training steps or every data epoch; refreshing too slowly disconnects the sampling distribution from capability.
    • Sampling failure fallback: When samples in the target interval are insufficient, first expand to adjacent tiers, then lower the difficulty ceiling; do not use repeated sampling to fill.
    • Stage acceptance: Entering the next stage requires hold-out pass@1 improvement to reach a threshold, format compliance rate ≥ 99%, and no obvious regression in manual spot-checks.
    • Preventing single-task overfitting: Apply stratified quotas per batch across STEM / OCR / localization / video / coding / GUI / long document.

    7. Evaluation and Launch Checklist

    Dimension Metric Suggested Threshold
    STEM pass@1, per-step trajectory correctness Set per version baseline
    OCR Character accuracy, field F1 ≥ 98%
    Localization Ratio with IoU ≥ 0.5 Viewed separately per area tier
    Video Temporal event accuracy Tiered by frame count
    GUI Per-step task success rate Per app per version
    Long document Cross-paragraph retrieval QA accuracy Tiered by document length
    Format CoT format compliance rate ≥ 99%
    Manual spot-check ≥ 300 per version, two-person cross No systematic regression
    Canary 5% → 20% → 50% Auto-rollback if hold-out drops > 2%

    8. Three Practice Principles (With Preconditions)

    1. Data is king: The precondition is that compliance is passed and ratios are validated through ablation; data cleaning and the iteration loop matter more than piling on parameters.
    2. Rewards are fundamental: The precondition is that verifiable rewards cover core tasks and open tasks have three-layer fallback of rules + preference + human review.
    3. Curriculum is the instrument: The precondition is that difficulty bucketing comes from measured accuracy and has stage acceptance; otherwise curriculum sampling degenerates into random sampling.

    The statement about parameter count needs qualifiers: under the premise that the data system, reward design, and evaluation criteria are consistent, and the task distribution matches the target evaluation set, a reasoning model with equal parameter count may outperform a larger-parameter approach on multiple challenging tasks. This conclusion cannot be extrapolated across tasks or evaluation sets—switching to uncovered scenarios (such as OCR of niche languages or industry-specific GUIs) commonly shows obvious degradation. For teams looking to deploy in their own business, rather than blindly piling on parameters, it is better to focus energy on these three high-leverage areas—data system, reward mechanism, and training strategy—while retaining complete evaluation and rollback capabilities.