From Visual Perception to Multimodal Reasoning: Technical Architecture and Practical Analysis of 9B Vision-Language Models
This article examines how 9B-scale vision-language models are evolving from basic image understanding to multi-step reasoning. It covers the roles of visual encoders, multimodal connectors, and language models; the data required for OCR, document understanding, localization, video analysis, and instruction tuning; and the use of supervised fine-tuning, multi-dimensional rewards, and curriculum-based sampling. It also discusses evaluation criteria, deployment value, practical limitations, and the importance of combining model capabilities with system-level controls.
From “Seeing” to “Thinking”: The Evolution of Vision-Language Models
Vision-language models (VLMs) are designed to process images, videos, and natural language simultaneously. They can perform tasks such as image captioning, visual question answering, classification, localization, document parsing, and video analysis.
Early models were primarily good at “describing what they saw.” They could generate relatively direct answers based on visible content, but often struggled with multi-step calculations, spatial relationships, causal judgments, and complex rules. Surface-level visual matching is rarely sufficient for these tasks.
For example, a model may need to do more than identify the objects in an image. It may also need to determine their relative positions, understand how they change over time, and infer the intent behind an action. In chart question answering, the model must first read the axes and legend before deriving a numerical result. In engineering drawings or software interfaces, it may need to locate a specific control and execute actions in a particular sequence.
These tasks share a common characteristic: visual perception is only the starting point. The final answer depends on multiple reasoning steps.
As a result, the focus of VLM development is shifting from simply connecting visual information to a language model toward enabling visual information to participate in a complete reasoning process. Chain-of-thought data, process supervision, reward models, and reinforcement learning have become important techniques for improving performance on complex tasks.
A Typical VLM Architecture: Vision Encoder, Connector, and Language Model
A typical vision-language model consists of three main components.
1. Vision Encoder
The vision encoder converts images or video frames into visual features. For static images, it needs to capture local textures, object shapes, scene structure, and text. For videos, it must also model changes between frames, action sequences, and event duration.
Vision encoders commonly process images as a collection of visual units or patches. Multiple network layers then establish relationships between local regions and the overall scene.
Higher resolution allows the model to retain more detail, but it also significantly increases computation and memory consumption. In practice, systems often use dynamic resolution, region cropping, or multi-scale encoding to balance accuracy and efficiency.
2. Multimodal Connector
The features produced by a vision encoder usually cannot be consumed directly by a language model. A projection layer, adapter, or another connector is needed to map visual features into a representation space that the language model can process.
The connector has a direct impact on how efficiently visual information is transmitted. Excessive compression may discard small text, table boundaries, and spatial relationships. Retaining too many visual tokens, however, increases context length and inference cost.
The connector therefore needs to balance information fidelity, sequence length, and training stability.
3. Language Model
The language model combines visual features with text instructions and generates the final response. For reasoning-oriented visual tasks, it must provide more than fluent language generation. It also needs to perform step-by-step analysis, mathematical calculations, logical judgments, and result verification.
The overall workflow can be summarized as:
Image or video input → visual feature extraction → multimodal feature mapping → language-context fusion → step-by-step reasoning → answer generation
This architecture is highly generalizable. It can support image question answering, visual grounding, document understanding, video analysis, and diagram reasoning.
The Data System Determines the Model’s Upper Bound
The capabilities of a multimodal model depend not only on parameter count. Data coverage, accuracy, and organization are equally important. A complete data system generally includes the following categories.
1. Image Captioning Data
Image-captioning data establishes the basic connection between visual content and natural language. High-quality captions should not merely list objects. They should also describe the scene, actions, quantities, attributes, relationships, and important details.
For example, a useful description of an experimental setup should do more than state that “multiple instruments are present.” It should identify how the instruments are connected, their possible operating states, and any labels shown in the image.
The more accurate these descriptions are, the stronger the foundation for later visual question answering and reasoning.
2. Interleaved Image-and-Text Data
Real-world information is rarely presented as a single image followed by one question. Web pages, technical documents, product manuals, and research materials often contain alternating images, titles, paragraphs, and tables.
Interleaved image-and-text data can simulate this structure and train models to understand relationships across pages, paragraphs, and images.
This type of data is especially valuable for long-document understanding. However, it must preserve content order, reference relationships, and the correspondence between visual elements and surrounding text. Otherwise, the model may learn superficial concatenation patterns rather than meaningful cross-modal associations.
3. OCR and Document Data
Text recognition is one of the foundations of visual reasoning. Invoices, tables, screenshots, exams, flowcharts, and scanned documents all require the model to read text accurately while preserving its spatial position on the page.
High-quality OCR data should include:
- Text content
- Text-box coordinates
- Reading order
- Page layout structure
For tables, the data should also represent row and column relationships, merged cells, and contextual headers.
Recognizing text while ignoring layout can easily lead to misaligned interpretations in complex documents.
4. Visual Grounding and Localization Data
Visual localization requires the model to identify where a target appears in an image. The output may be a bounding box, a point coordinate, or a regional description.
This capability is useful for object retrieval, image editing, interface operation, and robot interaction.
One challenge is that target granularity varies widely. The target may be an entire car or a specific button inside the car. It may be a single object or multiple objects that satisfy a particular condition.
Localization data should therefore cover different object scales, levels of occlusion, and spatial relationships.
5. Video Data
Video understanding is more than frame-by-frame recognition. It also involves event order, action changes, human interaction, and causal relationships.
Training data should represent questions such as:
- What happened?
- In what order did the events occur?
- Why did a particular action happen?
To control computational cost, videos are often processed through key-frame sampling, clip segmentation, and temporal information compression.
Sampling too sparsely may miss important actions, while sampling too densely creates a heavy context burden. The sampling strategy should therefore be adjusted according to the content of each video.
6. Instruction-Tuning Data
Instruction data converts basic visual capabilities into interactive behavior. It may include direct question answering, explanatory responses, multi-turn conversations, refusal examples, output-format constraints, and tool-oriented tasks.
For reasoning-oriented models, the completeness of the answer process is particularly important. The model should learn to extract information first, establish the relevant conditions, and then produce a conclusion, rather than memorizing fixed answers for specific question types.
Supervised Fine-Tuning: Teaching the Model to Reason Reliably
The goal of supervised fine-tuning is not simply to add more question-and-answer pairs. It is to establish stable and reusable reasoning patterns.
First, the training set should include high-quality chain-of-thought examples. Chain-of-thought does not mean that the model must always produce lengthy explanations. Instead, the key steps should be complete and the conclusion should be verifiable.
For a chart-related question, the reasoning process may need to include data extraction, the relevant calculation, and the final judgment. For spatial reasoning, it should identify the reference object, describe the directional relationship, and locate the target.
Second, response formats should be standardized. Different tasks may use structures such as:
- Analysis → Evidence → Conclusion
- Recognition result → Reasoning basis → Final answer
Standardized formats help reduce response drift and make automated evaluation and subsequent reinforcement learning easier.
Third, a data-quality management process is required. Common practices include rule-based validation, duplicate removal, answer-consistency checks, difficulty stratification, and manual sampling.
For mathematical, OCR, and localization tasks, programs can also automatically verify whether answers satisfy required formats and numerical constraints.
Reinforcement Learning: Improving Stability on Complex Tasks
Supervised fine-tuning teaches a model to imitate high-quality answers, but imitation alone does not guarantee stable reasoning. Reinforcement learning can further optimize performance on complex tasks.
1. Multi-Dimensional Reward Systems
Rewards for visual reasoning should not evaluate only whether the final answer is correct. They can also consider:
- Outcome reward: Whether the final conclusion is correct
- Format reward: Whether the output follows the required format
- Process reward: Whether the reasoning contains the necessary evidence
- Localization reward: Whether the predicted coordinates or regions are accurate
- Consistency reward: Whether repeated reasoning produces similar conclusions
- Safety reward: Whether the model avoids unsupported speculation when the information cannot be confirmed
Multi-dimensional rewards can reduce cases in which the answer happens to be correct even though the reasoning process is flawed.
However, reward design must avoid excessive constraints. Otherwise, the model may prioritize format compliance at the expense of natural and effective expression.
2. Curriculum-Based Sampling
Tasks vary significantly in difficulty. If training begins with large quantities of complex videos, long documents, and multi-step mathematical problems, the model may have difficulty converging. If training uses only simple examples, it may not develop stronger reasoning capabilities.
Curriculum-based sampling adjusts the proportion of examples according to the model’s current performance.
During the early stage of training, the dataset can emphasize basic perception and short reasoning chains. As capabilities improve, the training process can gradually introduce more cross-modal tasks, multi-step problems, and high-interference examples.
Tasks on which the model repeatedly fails can receive higher sampling weights, while simple tasks that have already been mastered can be sampled less frequently.
3. Optimization Stability
The reinforcement-learning stage requires monitoring several issues, including reward fluctuations, policy degradation, abnormal output length, and capability loss.
Performance should be tracked across different task categories rather than through a single aggregate score.
If reasoning performance improves while OCR, localization, or basic question answering deteriorates, the training objective may be biased. A more reliable approach is to maintain validation sets for individual tasks and run regular capability-regression tests.
How to Evaluate a Vision-Language Reasoning Model
Evaluation should not rely on a single leaderboard or accuracy score. A multidimensional evaluation system is more appropriate.
1. Basic Visual Understanding
This category includes image captioning, object recognition, attribute classification, and scene understanding. It measures whether the model has reliable visual perception.
2. Text and Document Processing
Important capabilities include small-font recognition, complex-layout analysis, table question answering, cross-page information integration, and long-document retrieval.
3. Mathematical and Scientific Reasoning
Evaluation should cover chart calculations, geometric relationships, formula understanding, experimental-phenomenon analysis, and multi-step problem solving.
The final answer is not enough. The intermediate derivation should also be checked for logical validity.
4. Video Understanding
Evaluation should examine event recognition, temporal order, action changes, relationships between people, and long-video summarization.
5. Spatial Localization and Interaction
This category includes target selection, coordinate prediction, regional reference resolution, and interface-element recognition.
In practical applications, localization error and task-completion rate may be more important than the quality of the model’s language expression.
6. Generalization
Whether a model can transfer to unseen domains is a key measure of general-purpose capability.
Test data should cover different image styles, document templates, question formulations, and resolutions. This helps prevent the model from merely adapting to the format of its training set.
Application Value and Limitations of 9B-Scale Models
Vision-language models with a moderate parameter count offer relatively manageable deployment costs, fast response times, and support for private deployment. They are suitable for scenarios such as document review, knowledge question answering, industrial inspection assistance, educational analysis, content retrieval, and interface automation.
However, a smaller model is not automatically better than a larger one across all tasks. Performance often depends heavily on the quality of the training data, the vision encoder, context length, and reasoning strategy.
In scenarios involving extremely high-resolution images, very long videos, complex domain knowledge, or multiple rounds of tool calls, a 9B-scale model may still suffer from:
- Missing visual information
- Spatial localization errors
- Calculation mistakes
- Unsupported inferences
For production deployment, it is therefore better to combine model capabilities with system-level constraints.
Examples include image tiling, OCR preprocessing, retrieval augmentation, result verification, human review, and confidence thresholds. These mechanisms can reduce uncertainty in individual inference results.
Conclusion: Visual Reasoning Is Not Just About Larger Models
The path toward more general-purpose vision-language models does not depend simply on increasing parameter count. It requires a complete training and evaluation loop.
First, the vision encoder and multimodal connector must ensure that images, videos, and text can be fused effectively.
Second, the data system must cover real-world tasks through image descriptions, OCR, localization, video examples, and instruction data.
Third, high-quality supervised examples should teach the model to produce a clear and verifiable reasoning process.
Fourth, multi-dimensional rewards and curriculum-based sampling can improve stability on complex tasks.
Fifth, task-specific, cross-domain, and real-world evaluations are needed to verify whether the model can generalize reliably.
The broader technical trend is toward process verification, explainable results, and controllable systems. A VLM should not only answer “What do you see?” It should also explain “What evidence supports the answer?” and “How was the conclusion reached?” When the available information is insufficient, it should express uncertainty instead of making unsupported claims.
For developers and enterprises, model selection should not focus only on parameter count or advertised benchmarks. Data types, latency requirements, deployment conditions, and the cost of errors are equally important. The most effective multimodal reasoning solution is usually built by combining the right model with the right system constraints for a specific business environment.
