How to Build a General-Purpose Multimodal Reasoning Model: Architecture, Data, and Reinforcement Learning in Practice
A practical guide to building a general-purpose multimodal reasoning model, focused on the three factors that actually limit it—data compliance, reward precision, and curriculum design. It covers the ViT-MLP-LLM architecture and its alternatives, six-category data preparation with ratios, cleaning thresholds, and compliance gating, the pretraining and SFT loops, and reinforcement learning built on precise reward systems and difficulty-based curriculum sampling. It includes acceptance tables, launch checklists, and rollout/rollback strategies.
Bottom Line Up Front
The bottleneck for a general-purpose multimodal reasoning model is usually not parameter count, but three things: whether the data system is compliant and usable, whether the reward signal is precise, and whether the training curriculum matches the model’s current level.
- Architecture: ViT-MLP-LLM is the engineering default starting point, but not the only option. The choice depends on GPU memory, latency, and whether the LLM natively supports visual tokens.
- Data: Six categories of data must each carry a ratio, cleaning thresholds, and sampling-based acceptance, and must all pass a compliance gate first.
- Reinforcement learning: Reward precision determines the direction of reasoning-quality improvement; curriculum sampling determines output efficiency per unit of compute.
1. Capability Tiers: Define the Goal First, Then the Evaluation Criteria
| Capability Tier | Typical Tasks | Acceptance Metric | Common Failure Mode |
|---|---|---|---|
| Perception | Counting, color/attribute recognition, short-text OCR | Counting accuracy, character accuracy | Missing small objects, missing small text, digit repetition |
| Comprehension | Scene/intent judgment, chart reading | Multiple-choice accuracy, chart-QA EM | Describing only surface elements, getting distracted by irrelevant details |
| Reasoning | STEM word problems, GUI planning, long-document cross-paragraph reasoning, video causality | pass@1, per-step trajectory correctness, manual spot-check | Drifting off track mid-way, inconsistent steps, correct answer but wrong process |
Most traditional VLMs stop at the perception and comprehension tiers; the third tier is what is genuinely hard. Define the capability tier before defining the evaluation, or you will get the illusion of “OCR improved but multi-step reasoning didn’t move.”
Breakthroughs in reasoning ability come mainly from two techniques: first, chain-of-thought (CoT), which makes the model output intermediate steps first and decompose complex problems into verifiable sub-problems; second, reinforcement learning (RL), which uses reward signals to let the model autonomously explore reasoning paths and learn “when to think, how to verify, and when to stop.” The three major shortcomings of existing approaches also correspond exactly to the improvement entry points: insufficient reasoning depth, coarse data utilization, and uneven sub-task capability.
2. Architecture Selection: How to Choose ViT-MLP-LLM, and What Alternatives Exist
2.1 Preconditions (Agree Before Jumping to Code)
- Image input
[B, 3, H, W]; when ViT patch=14,N_v = (H/14)*(W/14); under dynamic resolutionN_vvaries with the image, requiring in-batch padding or grouping by length. - ViT output
[B, N_v, D_vit](e.g., D_vit=1024); MLP output[B, N_v, D_llm], whereD_llmmust match the LLM hidden dimension (e.g., 4096). Dimension mismatches often do not raise errors—they silently degrade. - Concatenation is not simply joining head to tail, but replacing visual tokens into the image slot position according to the conversation template, and inserting
<image_begin>/<image_end>separators; separators must enter the vocabulary and participate in the attention mask. - The loss is computed only on text and answer segments; visual token positions are masked out, otherwise the loss is diluted by the large number of non-generatable visual positions.
- The following is only data-flow pseudocode, not directly trainable code.
# See preconditions above; this only illustrates the data flowdef forward(image, input_ids, labels):vision_tokens = vit(image) # [B, N_v, D_vit]vision_embeds = mlp_projector(vision_tokens) # [B, N_v, D_llm]text_embeds = llm.embed(input_ids) # [B, L, D_llm]# Insert visual tokens into image slots per template, not simple concatenationinputs = replace_image_slots(text_embeds, vision_embeds, sep=IMAGE_SEP)return llm(inputs, labels=labels) # labels are -100 at visual positions
2.2 More Than One Path: How to Choose Among Mainstream Alternatives
| Approach | Method | Suitable Scenario | Main Cost |
|---|---|---|---|
| ViT-MLP-LLM (default here) | Project to align, then concatenate into a token sequence | General VLM bootstrapping, fast iteration | Many visual tokens, context bloat at high resolution |
| Cross-attention | A small number of learnable queries extract visual information | Need to compress visual tokens, latency-sensitive | Complex structure and training, fine-grained OCR/localization easily hurt |
| Adapter / lightweight side-path | Insert adapters within a frozen LLM | Tight compute, training only a few parameters | Limited capacity, capability ceiling below full joint training |
| Native multimodal early-fusion | Unify visual and text patches from pretraining | Pursuing the strongest unified representation, ample compute | High training cost, hard for small teams to reproduce |
Two hard constraints:
- The MLP projection layer must train stably: near-identity or scaled initialization, separate warmup, freeze the LLM backbone early on. When loss does not decrease, suspect the alignment layer first, rather than tuning the learning rate first.
- The LLM side must support visual tokens: the context window must cover
N_v + text length; the attention implementation must support masking of non-text tokens; new separators require vocabulary expansion, otherwise they will be split into multiple tokens and break format compliance.
2.3 Three High-Frequency Errors
Using fixed cropping instead of dynamic resolution (small objects and dense text are lost directly); not compressing visual tokens (high-resolution images fill up the context, long-document tasks collapse); only end-to-end training without managing resolution and token budget (short descriptions improve, OCR does not).
3. Data Preparation: Ratios, Cleaning, and Compliance for Six Data Categories
3.1 Six Data Categories and Starting Ratios
| Data Category | Which Stage It’s For | Starting Ratio | Sampling Note |
|---|---|---|---|
| Image-caption | Pretraining | 40–55% | Deduplicate image-text pairs, mix long and short descriptions |
| Interleaved image-text | Pretraining | 20–30% | Preserve document order, truncate per document |
| OCR | Pretraining + SFT | 10–15% | Balance across languages, layouts, fonts |
| Visual grounding | SFT | 5–10% | Bin by bbox area, avoid learning only large objects |
| Video | Late pretraining + SFT | 5–10% | Frame count consistent with temporal annotation |
| Instruction tuning | SFT + RL | Mainly SFT | Dynamically adjust quota per evaluation weaknesses |
The ratios are starting values and need adjustment based on your business evaluation set weaknesses; it is recommended to do a per-category ablation before and after each training round to confirm “where it improved and where it dropped.”
3.2 Cleaning Rules and Deduplication Thresholds (Ready-to-Use Starting Parameters)
- Image-level dedup: pHash Hamming distance ≤ 6 counts as duplicate; text MinHash Jaccard ≥ 0.9 counts as duplicate.
- Low-quality filtering: drop short side < 100px, JPEG quality factor < 60; drop non-OCR samples with text < 5 words.
- OCR-specific: drop recognition confidence < 0.7; downweight when watermark/subtitle/sticker covers > 30% of the frame; single-source layout share no more than 15%.
- Safety filtering: NSFW, violence, and hate content filtered with a two-layer classifier + keyword approach.
- Source constraint: single-source share capped at 20%, preventing one site’s style from dominating the entire mix.
3.3 Sampling Acceptance Table
| Check Item | Sampling Volume | Pass Line | Action on Failure |
|---|---|---|---|
| Image-text consistency (manual) | 1000 pairs/batch | ≥ 95% | Roll back that source |
| OCR character accuracy | 2000 lines | ≥ 98% | Relabel or downweight |
| Grounding IoU | 1000 boxes | ≥ 0.80 | Notify annotator to retrain |
| Video temporal label consistency | 500 segments | ≥ 90% | Reduce frames and relabel |
| Privacy/compliance spot-check | Full automatic + 1% manual | 0 violations | Take the batch offline immediately |
3.4 Data Compliance and Privacy (A Step That Can’t Be Skipped)
- Authorization and licensing: Use only commercially licensed, CC-permitted, or self-collected data; scraping must comply with platform terms and robots, and retain source URLs and batch numbers.
- De-identification: Blur or replace faces, license plates, ID cards, medical records, and home addresses; use regex + model dual de-identification for real names, phone numbers, emails, and order numbers in OCR.
- GUI data red lines: Prohibit collecting accounts, passwords, cookies, session tokens, payment info, and real user identity information; recording must use test accounts and synthetic environments.
- Copyright filtering: Continuous scans of entire books/papers, film/TV posters and screenshots, and watermarked stock images need filtering; long-text OCR is compared against publishing corpora via n-gram.
- Video and audio: Speech transcription content also goes through privacy screening, and on-screen content in frames must be checked frame by frame.
- Storage and audit: Encrypted storage, traceable by batch, access logged, and a retention period and deletion process defined.
It is recommended to make compliance checks a hard gate at ingestion: batches that fail checks do not enter the training pool, rather than doing spot-checks after the fact.
4. Pretraining: Building the Visual Foundation
The goal is to complete modality alignment and basic representation learning. The data order is recommended as image-caption → interleaved image-text → OCR → video/localization, gradually increasing granularity. On the freezing strategy, stage one freezes the LLM backbone and trains only ViT and MLP; once alignment stabilizes, unfreeze for joint training. On scheduling, use multi-stage mixing with dynamic ratio adjustment, rather than fixing it once and for all. Each stage evaluates per-category metrics on a fixed hold-out set (no overlap with the training set); if metrics regress, roll back to the previous stage’s weights and re-adjust the ratios.
5. SFT and the Data Quality Loop
SFT aligns the “unpolished capability” after pretraining to the human-expected answer format and reasoning habits, especially the organization of chain-of-thought answers.
5.2 Difficulty Tiering and Iterative Enhancement (Executable Parameters)
- Difficulty bucketing: Bucket into 5 tiers by the current model’s pass@1 (0–0.2 / 0.2–0.4 / … / 0.8–1.0), refreshed each round or every N steps.
- Failure reason tags: Format errors, visual misreads, unit/dimension errors, multi-step reasoning breaks, out-of-bounds citations, and over-refusal.
- Stage switching condition: Only advance tiers when both the hold-out pass rate and format compliance rate are met; stay in the current tier if two consecutive checkpoints fall short.
- Rollback mechanism: Keep the most recent K checkpoints and corresponding data versions; when a tier’s metrics decline continuously, roll back the data recipe rather than adding more data.
- Enhancement priority: For format issues, add format examples; for visual misreads, add high-resolution and dense-text samples; for reasoning breaks, add CoT samples with process annotation.
6. Reinforcement Learning: Precise Rewards and Curriculum Sampling
SFT teaches the model to answer in a standardized way, and RL teaches the model to think better—this is the key leap from “usable” to “excellent.”
6.1 Reward System: First Distinguish Closed Tasks from Open Tasks
def compute_reward(prediction, ground_truth, task_type):if task_type == "closed": # math result, OCR field, bbox, GUI element IDaccuracy = exact_match(prediction, ground_truth) # use IoU for localizationformat_ok = check_format(prediction)return 1.0 * accuracy + 0.1 * format_ok# Open tasks: description, explanation, report summarization; need three-layer fallbackrule_ok = rule_check(prediction) # format/length/forbidden words/traceable citationpref = reward_model(prediction, ground_truth)return 0.3 * rule_ok + 0.7 * pref # high-value samples still go to manual review queue
Key points:
exact_matchonly applies to tasks with closed, automatically judgeable answers; open tasks must add rule checks, preference models, and manual review.- Reward hacking defense: Focus on monitoring two types of divergence—“format score rises but accuracy does not” and “answer is right but process is wrong”—and spot-check the reasoning process.
- Overfitting monitoring: Only when the training reward and hold-out pass@1 rise together does it count as real improvement; reward rising without hold-out rising counts as overfitting the metric.
- Fallback strategy: When hold-out declines for N consecutive steps or format compliance falls below a threshold, lower the RL weight and roll back to the SFT checkpoint, and investigate whether the reward is being gamed.
6.2 Curriculum Sampling: Where Difficulty Comes From, How Often to Update, How to Roll Back
CURRICULUM = {"stage1": (0.0, 0.3), "stage2": (0.3, 0.6), "stage3": (0.6, 0.9)}ACC = {b: model.pass_rate(b) for b in BUCKETS} # refreshed every REFRESH_EVERY stepsdef sample_batch(pool, stage, acc, step):if step % REFRESH_EVERY == 0:acc.update(refresh_pass_rate(model, pool))lo, hi = CURRICULUM[stage]batch = [s for s in pool if lo <= acc[s.acc_bucket] <= hi]if len(batch) < MIN_BATCH: # expand to adjacent tiers first when samples are insufficientbatch = nearest_difficulty(pool, hi, need=MIN_BATCH)return batch
- Difficulty source: Primarily bucket by the model’s own historical accuracy, with annotation difficulty as cross-validation and manual spot-checks for calibration, avoiding purely subjective labeling.
- Update frequency: Refresh once every 200–500 training steps or every data epoch; refreshing too slowly disconnects the sampling distribution from capability.
- Sampling failure fallback: When samples in the target interval are insufficient, first expand to adjacent tiers, then lower the difficulty ceiling; do not use repeated sampling to fill.
- Stage acceptance: Entering the next stage requires hold-out pass@1 improvement to reach a threshold, format compliance rate ≥ 99%, and no obvious regression in manual spot-checks.
- Preventing single-task overfitting: Apply stratified quotas per batch across STEM / OCR / localization / video / coding / GUI / long document.
7. Evaluation and Launch Checklist
| Dimension | Metric | Suggested Threshold |
|---|---|---|
| STEM | pass@1, per-step trajectory correctness | Set per version baseline |
| OCR | Character accuracy, field F1 | ≥ 98% |
| Localization | Ratio with IoU ≥ 0.5 | Viewed separately per area tier |
| Video | Temporal event accuracy | Tiered by frame count |
| GUI | Per-step task success rate | Per app per version |
| Long document | Cross-paragraph retrieval QA accuracy | Tiered by document length |
| Format | CoT format compliance rate | ≥ 99% |
| Manual spot-check | ≥ 300 per version, two-person cross | No systematic regression |
| Canary | 5% → 20% → 50% | Auto-rollback if hold-out drops > 2% |
8. Three Practice Principles (With Preconditions)
- Data is king: The precondition is that compliance is passed and ratios are validated through ablation; data cleaning and the iteration loop matter more than piling on parameters.
- Rewards are fundamental: The precondition is that verifiable rewards cover core tasks and open tasks have three-layer fallback of rules + preference + human review.
- Curriculum is the instrument: The precondition is that difficulty bucketing comes from measured accuracy and has stage acceptance; otherwise curriculum sampling degenerates into random sampling.
The statement about parameter count needs qualifiers: under the premise that the data system, reward design, and evaluation criteria are consistent, and the task distribution matches the target evaluation set, a reasoning model with equal parameter count may outperform a larger-parameter approach on multiple challenging tasks. This conclusion cannot be extrapolated across tasks or evaluation sets—switching to uncovered scenarios (such as OCR of niche languages or industry-specific GUIs) commonly shows obvious degradation. For teams looking to deploy in their own business, rather than blindly piling on parameters, it is better to focus energy on these three high-leverage areas—data system, reward mechanism, and training strategy—while retaining complete evaluation and rollback capabilities.
