Building Multimodal Vision-Language Models on a Text-Based Foundation
This article explains how to build a multimodal vision-language model (VLM) by extending a strong text-based large language model with a vision encoder, rather than training one from scratch. It covers the core three-stage architecture, high-resolution image handling and compute-cost control, mixed-data training to prevent catastrophic forgetting, the key dimensions for evaluating multimodal ability, and deployment and cost considerations, and closes with practical engineering recommendations.
1. Background and Problem Definition
In recent years, the reach of large models has been expanding from pure text understanding toward “making sense of the world.” A multimodal vision-language model (VLM) is a model that can process both image input and natural-language input, and reason jointly across the two. It must do more than answer “what is in this image”—it also has to handle tasks such as formula recognition, chart understanding, UI reproduction, and multi-turn visual dialogue.
A pragmatic technical path is to avoid training a multimodal model from scratch and instead extend a language model that already has strong text capabilities by adding a vision encoder. The core value of this approach is that the text foundation already carries substantial semantic understanding, reasoning, and knowledge, so multimodal extension should preserve and reuse as much of that capability as possible rather than rebuild it.
2. Core Architecture: Text Foundation + Vision Encoder
Most mainstream vision-language models adopt a three-stage structure:
- Vision encoder: Usually a Vision Transformer (ViT) that splits the input image into a number of patches and maps them into a sequence of visual tokens.
- Feature alignment and fusion layer: Maps the visual tokens into the same semantic space as the text, so that both can be processed uniformly by the language model.
- Language model foundation: Takes the concatenated mixed sequence of “visual tokens + text tokens” and performs joint modeling and generation.
In this structure, most of the parameters are concentrated in the language model foundation (for example, an open-source text foundation at around the 13-billion-parameter scale), while the vision encoder accounts for a relatively small share. The overall parameter count is therefore close to that of the foundation—still at roughly the ten-billion level—which is an important precondition for deploying it under limited compute.
One design choice worth highlighting is whether to introduce an extra specialized module such as a “vision expert.” Adding a vision expert can make the model stronger on specific visual tasks, but it significantly raises deployment and inference costs. A lighter alternative is to skip the extra vision expert and instead mix text data with image-text pair data during training, injecting multimodal ability step by step while maintaining text performance. This design trades away some peak visual accuracy in exchange for a lower deployment barrier and more balanced overall capability.
The following illustrative code shows the basic flow of this fusion-based inference:
# Illustration: inference flow of a multimodal vision-language model (pseudocode)def multimodal_forward(image, prompt, target_res=1120):# 1. High-resolution preprocessing: scale to target resolution while preserving aspect ratioimage = preprocess(image, target_resolution=target_res)# 2. Vision encoder extracts visual tokens (ViT)visual_tokens = vision_encoder(image)# 3. Downsample / pool to compress the visual sequence length and reduce computevisual_tokens = visual_pooling(visual_tokens)# 4. Align visual features to the language model's semantic spacevisual_embeddings = align_projector(visual_tokens)# 5. Tokenize the texttext_embeddings = tokenizer.embed(prompt)# 6. Concatenate the mixed multimodal sequenceinputs = concat([visual_embeddings, text_embeddings])# 7. The language model foundation jointly models and generatesreturn language_model.generate(inputs)
3. Handling High-Resolution Images and Controlling Compute Cost
The difficulty of visual tasks often concentrates on “details.” A web screenshot, a math problem, or a data chart frequently carries key information in the form of small fonts and dense layouts. If an image is forced down to a low resolution, the model can easily lose those details and fail to recognize them. Supporting high-resolution input (such as 1120×1120 and above) is therefore a key capability for handling complex visual scenes.
But high resolution brings a direct problem: compute cost grows quadratically. The self-attention complexity of a Transformer grows quadratically with sequence length—the more patches an image is split into, the longer the sequence, and the greater the memory and compute consumption. Common engineering countermeasures include:
- Downsampling and visual token pooling: Compress the token sequence after visual encoding to shorten the sequence length without significantly losing key information.
- Adaptive resolution: Dynamically adjust the input resolution based on task complexity—downsampling simple scenes while keeping high resolution for complex ones.
- Aspect-ratio-preserving scaling: Avoid stretching that would distort text and structure, thereby protecting the accuracy of OCR and chart understanding.
4. Training Strategy: Mixed Data and Capability Retention
The biggest risk in multimodal extension is “learning a new skill but forgetting the old ones.” If training uses only image-text data, the language model’s original text ability may be weakened. To address this, mixed-data training is a widely adopted practice in the industry: combine a large-scale text corpus with image-text pairs and image-understanding instruction data in set proportions, so the model keeps reinforcing its language and reasoning ability while learning visual understanding.
Such models support multi-turn dialogue in a bilingual setting, with capabilities spanning Chinese-English competency, perception and reasoning, text recognition (OCR), and chart understanding. Typical task performance includes:
- Formula and text recognition: Recognizing complex layouts such as mathematical formulas in images and answering questions based on their content.
- UI-to-code generation: Converting a web or UI screenshot into structured markup code.
- Chart information extraction: Understanding values and trends in bar charts, line charts, and tables, and making inferences from them.
5. Key Evaluation Dimensions for Multimodal Ability
Evaluating a vision-language model should be done across multiple dimensions rather than by looking at a single leaderboard score:
- General perception and reasoning: Whether it can correctly describe and reason about both natural scenes and abstract diagrams.
- Text recognition (OCR): Accuracy on Chinese and English text, formulas, and dense layouts.
- Chart understanding: Reading, comparing, and drawing trend judgments from chart data.
- Multi-turn dialogue coherence: Maintaining context consistency across continuous visual question answering.
- Bilingual ability: Quality of understanding and generation in mixed Chinese-English scenarios.
It is worth cautioning that claims like “comprehensively outperforming some advanced model” should be treated carefully. Evaluation results depend heavily on the specific test set, prompt design, and evaluation criteria, and are not directly comparable across different leaderboards. For engineering selection, a more reliable approach is to build a dedicated evaluation set based on your own business scenario and run reproducible comparative tests.
6. Deployment and Cost Considerations
For enterprise users, model capability is only half of the decision—the other half is cost and operability. When bringing multimodal ability into production at the ten-billion-parameter scale, the following general practices are worth considering:
- Inference optimization: Use common techniques such as quantization, KV caching, and batching to reduce the memory and latency cost of a single inference.
- Elastic compute: Combine container platforms with elastic scaling to handle the peaks and troughs of visual-task traffic.
- Service governance: Integrate logging and monitoring/alerting to track inference quality, latency, and error rates, forming a closed loop.
- Storage and distribution: Use object storage to manage model weights and image-text datasets, and pair it with a message queue to decouple the offline batch-processing pipeline.
7. Practical Recommendations and Conclusion
Overall, building a usable open-source multimodal vision-language model can follow a clear engineering throughline:
- Reuse rather than rebuild: Extend a strong text foundation with vision capabilities first, protecting existing semantic and reasoning ability.
- Balance resolution and compute: Use downsampling, visual token pooling, and adaptive resolution to serve both detail recognition and compute cost.
- Mixed-data training: Inject multimodal ability through image-text mixing to avoid “catastrophic forgetting.”
- Prefer a lightweight architecture: Under a “good enough” premise, introduce extra specialized modules cautiously to keep the deployment barrier low.
- Scenario-based evaluation: Build an evaluation set from real business data, and avoid being misled by a single leaderboard.
For developers and enterprises that want to bring multimodal ability into their own scenarios, understanding the architecture and training principles above matters more than chasing the parameters of any particular product. Mastering the “text foundation + vision encoder + mixed training” paradigm lets you choose a suitable technical approach based on your own data conditions and compute budget, and reliably integrate the ability to “understand images” into real business.
