Open-Source Small-Parameter Visual Reasoning Models: A System-Level Evaluation

    This article examines open-source small- and medium-parameter visual language reasoning models as complete systems rather than isolated checkpoints. It reviews their technical positioning, architecture, multimodal data strategy, staged pretraining, supervised fine-tuning, reinforcement learning with curriculum sampling, evaluation protocols, open-source value, and engineering deployment. The goal is to help readers assess whether reasoning behavior is reproducible and verifiable, whether gains generalize to real tasks, and how far open-source releases support secondary development and production use.

    Open-Source Small-Parameter Visual Reasoning Models: A System-Level Evaluation

    Evaluating a class of small- and medium-parameter visual language reasoning models should not rely only on parameter count, a single leaderboard score, or demo performance. A more reasonable perspective is to treat such a model as a system composed of model architecture, data recipe, training pipeline, reward mechanism, evaluation protocol, and engineering interfaces. Three questions are especially worth asking: First, does reasoning capability come from a reproducible and verifiable training loop? Second, can capability gains generalize across tasks to real business scenarios? Third, to what extent is the so-called open source release actually open, and is it sufficient to support secondary research and production deployment?

    The recent trend in the open-source community of introducing chain-of-thought reasoning and reinforcement learning into visual language models provides a useful sample for observation. Without an extreme parameter scale, a model can still make clear progress on tasks such as document understanding, chart question answering, video understanding, visual grounding, UI operation, and scientific calculation through high-quality multimodal data, structured supervised fine-tuning, and reinforcement learning with curriculum sampling. The significance of this route is not merely that “small models catch up with large models,” but that it pushes visual language models from “describing what is in an image” toward “analyzing an image, reasoning step by step, calling tools, and providing verifiable answers.”

    1. Technical Positioning: From Perception to Understanding to Reasoning

    The capabilities of visual language models can generally be divided into three layers.

    1. Perception: identifying objects, text, tables, layouts, icons, colors, and spatial relationships in images. OCR, detection-style grounding, and image classification mainly fall into this layer.
    2. Understanding: associating visual elements with language semantics and understanding scenes, actions, causality, document structure, and the temporal order of video events.
    3. Reasoning: decomposing tasks, locating evidence, comparing conditions, performing calculations or rule-based judgments, and explaining uncertainty in conclusions when faced with complex questions.

    The main tension for small-parameter models is that limited parameter count gives them less world knowledge and long-chain reasoning capacity than very large models, but they also offer advantages in cost, latency, deployability, and customization. Therefore, a reasonable strategy is not to make a medium or small model memorize everything, but to use data quality and post-training mechanisms to teach it “how to look, how to find evidence, how to solve problems step by step, and when to admit it does not know.” The value of reinforcement learning here is to convert signals such as final answer correctness, format compliance, evidence grounding accuracy, and verifiable calculation into optimization objectives, so that the model does not merely imitate samples but improves its behavior through attempts and feedback.

    2. Architecture Design: Encoding, Alignment, and Language Generation

    Models of this type typically adopt a three-stage structure: visual encoder, feature projection layer, and language model. The visual encoder encodes images or video frames into visual features; the projection or adapter layer maps visual features into the representation space of the language model; the language model fuses visual tokens with text tokens and performs multi-step generation.

    Architecture evaluation should focus on the following details:

    • Resolution and image-tiling strategy: Documents, tables, small text, and UI elements depend heavily on resolution. Dynamic resolution, image tiling, and local magnification can reduce OCR and grounding errors, but they significantly increase the number of visual tokens and must be balanced against context budget and inference cost.
    • Visual token compression: Higher resolution brings more tokens. Compression strategy, pooling strategy, and positional encoding design affect whether fine-grained information is preserved.
    • Multiple images and interleaved image-text inputs: Real tasks often involve multi-page documents, multiple screenshots, or mixed image-text layouts. Models need to understand cross-image references, page order, and contextual relationships, not just single-image descriptions.
    • Video temporal modeling: Video tasks require handling frame sampling, temporal positions, event boundaries, and long-video context. Fixed sparse sampling is low-cost but may miss key moments; adaptive sampling is more accurate but more complex to schedule.
    • Language model interface: Whether the model supports structured output, tool calling, long context, multi-turn dialogue, and safety refusal determines whether it can enter business workflows.

    It is important to emphasize that architecture alone does not necessarily produce reasoning capability. With the same encoder and language model, insufficient data coverage, inconsistent supervision formats, or poorly designed rewards can still cause the model to degrade into a “visual keyword matcher.” Architecture is only the foundation that carries capability; the training loop is the source of differentiation.

    3. Data System: The Foundation of Multimodal Capability

    A high-quality data mixture is a core asset of visual reasoning models. It can be divided into six categories.

    1. Image-text pairs: Used for basic visual-semantic alignment. Evaluation should consider description granularity, noise ratio, language diversity, and whether the data covers charts, documents, natural scenes, and industrial images. Coarse captions can only teach a model “what is in the image,” while fine-grained region descriptions and relationship descriptions are needed to support more complex question answering.
    2. Interleaved image-text data: Used for multi-image, multi-page, mixed image-text, and long-context understanding. This type of data helps models build cross-image references and page-level structure awareness, but it is difficult to clean and prone to repeated templates and low-information-density samples.
    3. OCR and document data: Covers scene text, scanned documents, tables, formulas, receipts, handwriting, and multilingual layouts. Beyond the text itself, layout coordinates, reading order, and cell relationships also matter. If a model is trained only on text recognition and not layout understanding, it is likely to fail on complex tables and cross-page questions.
    4. Visual grounding data: Includes boxes, points, region descriptions, and spatial relationships. It serves both “where is this” question answering and helps reduce hallucination. Annotation consistency, coordinate normalization, and negative sample design directly affect grounding quality.
    5. Video data: Should cover action recognition, event order, temporal grounding, subtitle understanding, and long-video summarization. It is necessary to guard against subtitles or titles leaking answers, so that models do not bypass visual information and rely directly on textual cues.
    6. Instruction fine-tuning data: Includes question answering, math and science problems, chart parsing, code and UI operation, document extraction, refusal, and safety samples. The closer the task distribution of instruction data is to real business scenarios, the easier it is to observe gains from post-training.

    Data management should not pursue scale alone. It is advisable to establish unified mechanisms for deduplication, difficulty stratification, source weighting, quality scoring, and contamination checks; training, validation, and test sets should be split by source or template to avoid leakage from same-source samples. For OCR, grounding, and video tasks, hard samples and negative samples should be retained to prevent the model from memorizing only common layouts. For synthetic data, distribution bias must be controlled and validated against real samples.

    4. Pretraining: Building the Vision-Language Foundation in Stages

    A relatively mature training pipeline is usually not a single mixed training run, but a staged process:

    • The first stage performs vision-language alignment so that the language model learns to read visual features.
    • The second stage expands interleaved image-text, OCR, grounding, video, and question-answering data to build general multimodal capability.
    • The third stage strengthens high-resolution, long-context, multi-image, and multi-frame processing.
    • Finally, instruction warm-up is added to prepare formats and interaction habits for subsequent supervised fine-tuning and reinforcement learning.

    From an engineering perspective, attention should be paid to balancing visual token and text token budgets, loss weights across tasks, resolution bucketing, sequence packing, and long-sequence efficiency. During training, fixed evaluation sets and small real-business sample sets should be used for process evaluation rather than looking only at training loss. Common pretraining risks include excessive caption-style descriptions leading to poor answer generalization, mismatch between synthetic OCR and real scanned documents, video subtitle leakage, inconsistent grounding annotations, and long-context noise caused by low-quality interleaved data.

    5. Supervised Fine-Tuning: Turning “Can See” into “Can Answer and Reason Step by Step”

    The key to supervised fine-tuning is not simply increasing the number of question-answer pairs, but shaping a stable task-solving pattern.

    First, selectively construct high-quality reasoning samples. For complex tasks such as math, charts, document comparison, UI operation, and visual grounding, samples can be constructed as “question understanding, evidence grounding, step decomposition, conclusion, and uncertainty explanation”; for simple recognition questions, answers should remain concise, avoiding lengthy reasoning for every task. Overly long and templated reasoning increases latency and can teach the model to pile up wording.

    Second, standardize response formats. It is advisable to organize final answers, brief rationale, cited regions or image IDs, and confidence explanations into separate fields so that answers are easy to parse and automatically evaluate. When internal reasoning is involved, the full chain of thought should not be exposed directly to end users; production systems are better suited to output brief reasoning summaries and verifiable evidence while retaining structured intermediate results for debugging and review.

    Third, establish a data quality loop. Samples can be filtered through a combination of strong-model generation, manual spot checks, rule validation, and execution-based verification. Math problems can be checked for numerical or symbolic equivalence; document questions can be checked for whether the answer can be located in the source text; grounding questions can be checked for region consistency; UI operation questions can be checked for whether actions are reachable. Error samples should be labeled by error type, such as missed observation, misreading, calculation error, coordinate error, format error, and inappropriate refusal, and then data should be supplemented by type.

    Fourth, prevent template overfitting. If training data consists entirely of fixed sentence patterns, the model may learn “correct format but empty content.” Different question phrasings, multi-turn follow-ups, distractor options, unanswerable questions, and cross-domain samples should be added to improve robustness.

    6. Reinforcement Learning: Using Verifiable Feedback to Improve Reasoning Behavior

    Reinforcement learning is suitable for optimizing objectives that are difficult to stabilize through supervised learning alone, such as whether the final result is correct, whether evidence is grounded, and whether the format is parseable. A reward system usually consists of multiple types of signals:

    • Answer correctness reward: numerical error, symbolic equivalence, unit consistency, and multiple-choice matching;
    • Structured reward: whether the final answer fields are complete and whether the format is parseable;
    • Evidence reward: whether OCR regions, image boxes, or page references are consistent with the answer;
    • Process constraint reward: whether the model reasons step by step as required, avoids irrelevant repetition, and states uncertainty when evidence is insufficient;
    • Safety and compliance reward: refusing inappropriate requests, avoiding leakage of sensitive information, and avoiding unsupported assertions.

    Reward design must prevent “gaming.” For example, rewarding only format can teach the model to use templates; rewarding only brevity can make it omit necessary steps; rewarding only long reasoning can induce verbose output. A more robust approach is to combine rewards, clip outliers, set upper limits on length and format, and retain an independent validation set.

    The core of reinforcement learning with curriculum sampling is to dynamically match training sample difficulty to the model’s current capability. Samples that are too easy provide little gradient, while samples that are too difficult almost always fail and produce excessive variance. Samples can be bucketed by task type and pass rate, increasing the sampling weight of “moderately difficult” samples while retaining a small number of easy samples to stabilize format and a small number of hard samples to encourage exploration. An illustrative pseudocode example is as follows:

    1. for iteration in training_rounds:
    2. stats = evaluate(policy, difficulty_bins)
    3. weights = update_weights(
    4. base_weight=task_balance,
    5. pass_rate=stats.pass_rate,
    6. target_band=[0.2, 0.8]
    7. )
    8. batch = sample(pool, weights)
    9. responses = policy.generate(batch)
    10. rewards = reward_function(responses, batch.answers, batch.evidence)
    11. update_policy(responses, rewards)
    12. log_reward_breakdown_and_length()

    Optimization details are equally important: use relative advantage to reduce variance, add policy constraints to prevent distribution drift, monitor entropy and generation length, balance reward scales across tasks, and conduct regular manual spot checks. The most common problem in reinforcement learning is not “no reward,” but inconsistency between reward and true quality; therefore, mechanisms for adversarial samples, held-out sets, and online failure case feedback must be established.

    7. Evaluation: Looking Beyond Leaderboards at Generalization and Reliability

    To evaluate models of this type, at least the following dimensions should be covered: natural image question answering, document and table parsing, OCR, charts and science questions, mathematical calculation, visual grounding, video temporal understanding, UI operation, multilingual capability, long context, hallucination rate, refusal quality, safety, and reasoning cost. In addition to benchmarks, small real evaluation sets with the same distribution as enterprise tasks should be constructed, with fixed prompts, decoding parameters, image resolution, and maximum output length, to compare answer accuracy, evidence hit rate, average latency, and failure modes.

    Conclusions that “small models approach or exceed larger models” need cautious interpretation. Different models may use different resolutions, tool calling, prompt templates, and training data, so single-leaderboard comparisons are not necessarily fair. More reliable conclusions should come from comparisons under the same input, same budget, and same evaluation protocol, with reported variance across repeated experiments. Test set contamination, answer leakage, and template memorization should also be checked. Cross-domain generalization is an important highlight: document reading capability may transfer to UI understanding, grounding capability may reduce hallucination, and video temporal capability may also feed back into multi-image reasoning; however, multi-task training may also produce negative transfer, so task losses and evaluation changes should be monitored during training.

    8. Open-Source Value and Engineering Deployment

    The value of an open-source model lies not only in whether the weights can be downloaded, but also in the license, training code, data recipe, evaluation scripts, known limitations, and degree of reproducibility. Opening only weights without training details still leaves high costs for reproduction and secondary research; opening complete evaluation protocols and failure cases is often more valuable in the long term than simply pursuing leaderboard scores.

    For engineering deployment, the model should be positioned as a “visual understanding and reasoning component,” not an all-knowing system. A layered architecture is recommended:

    1. Input preprocessing: image rectification, resolution selection, page splitting, and video keyframe extraction;
    2. Retrieval and context: multi-page document retrieval, image-text memory, and business rule injection;
    3. Model inference: structured output, brief evidence, and confidence;
    4. External validation: calculators, rule engines, database queries, OCR rechecking, or manual review;
    5. Fallback strategy: route low-confidence cases to humans or escalate to stronger models and toolchains.

    For cost-sensitive scenarios with clear tasks, small- and medium-parameter models are attractive for document extraction, UI assistance, visual question answering, and content moderation. For high-risk decisions, they must be combined with tool-based verification, permission control, and human review. Before launch, continuous monitoring should cover distribution drift, OCR failures, visual hallucination, long-tail languages, and safety refusal, and a failure sample feedback and retraining mechanism should be established.

    9. Conclusion: A Promising Route That Requires Calm Evaluation

    From a technical route perspective, combining high-quality multimodal data, structured chain-of-thought supervision, and verifiable reinforcement learning is indeed a reasonable direction for improving the general reasoning capability of small- and medium-parameter visual language models. Its contribution lies in demonstrating that parameter scale is not the only variable; data recipe, reward design, and curriculum sampling can significantly change model behavior. Its open-source value also provides a foundation for reproducing experiments, establishing baselines, and customizing scenarios.

    However, evaluation should not stop at “whether it exceeds some larger model.” More critical questions are: Is training reproducible? Are rewards reliable? Is performance stable in real business scenarios? Are hallucination and safety controllable? Is the open-source boundary clear? Does deployment cost match the scenario? Continuous verification of these questions is the key criterion for judging whether a class of visual reasoning models can move from demos to production.