Open-Source Multimodal LLM Selection: Architecture, Evaluation, and Deployment
This article explains how to evaluate and select open-source multimodal LLMs that combine a language model with a vision encoder. It covers representative projects, parameter-efficient adaptation methods, evaluation metrics and reproducibility requirements, a selection checklist, deployment risks, and compliance and safety considerations. Readers will learn what to verify before choosing a model and how to avoid common evaluation and deployment pitfalls.
1. Scope and Representative Projects
This article discusses open-source multimodal models of the “language model plus vision encoder” type. These models use a language model as the core. A vision encoder reads the image, and the image features are then fed into the language model to generate an answer. Model capability mainly depends on three things: the capability of the base language model, the resolution and representation quality of the vision encoder, and the design of the training data and training stages.
The table below lists several representative projects from China and other regions. The information comes from each project’s paper or official repository. For specific versions, parameter scales, and license terms, refer to the release page.
| Project | Main organization(s) | Base language model | Vision component | Training or adaptation features |
|---|---|---|---|---|
| LLaMA-Adapter V1 | Shanghai AI Laboratory et al. | LLaMA-7B | Mainly text-instruction focused; visual capabilities added by later work | Zero-initialized attention; adapter prompts injected at top layers |
| LLaMA-Adapter V2 | Shanghai AI Laboratory et al. | LLaMA | CLIP-style vision encoder | Bias tuning, early fusion, joint training |
| Qwen-VL | Alibaba Cloud | Qwen | ViT plus position-aware vision-language adapter | Multi-stage training; Chinese-English bilingual support |
| InternVL | Shanghai AI Laboratory et al. | Large language model with InternViT | Large-scale vision encoder | Progressive image-text alignment |
| CogVLM | Zhipu AI, Tsinghua University | Vicuna series | EVA2-CLIP series | Visual expert modules added to attention and feed-forward layers |
| LLaVA | University of Wisconsin-Madison et al. | Vicuna, LLaMA series | CLIP ViT-L/14 | Two-stage training: feature alignment and visual instruction tuning |
| MiniCPM-V | ModelBest, Tsinghua University | MiniCPM | SigLIP-style encoder | Optimized for on-device deployment |
Some descriptions label one method as a bilingual model. However, the LLaMA-Adapter series uses LLaMA, which is primarily English-centric, as its base. Chinese capability depends on whether the specific version was trained on Chinese data. If Chinese is a hard requirement, test Chinese question answering separately rather than relying on the description.
2. Parameter-Efficient Adaptation Mechanisms: Distinguish Three Things
Many introductions describe parameter-efficient methods as “concatenating a set of vectors before the input.” That description is inaccurate. The LLaMA-Adapter series works in three ways.
- Learnable adapter prompts. In several Transformer layers at the top of the base model, a set of learnable prompt vectors is added to each layer and participates in attention computation as additional keys and values. These prompts are not concatenated before the input text; they are injected layer by layer.
- Zero-initialized gating. The output of the newly added attention is multiplied by a learnable gate coefficient whose initial value is zero. At the start of training, the new branch produces no output, so the model behaves like the original base model. The gate value then gradually opens. This makes training more stable and reduces damage to existing knowledge.
- Bias tuning. The V2 version trains only the bias terms in linear layers and does not modify the weight matrices. The number of bias parameters is much smaller than that of weight matrices, so training and storage costs are lower.
The table below compares several common parameter-efficient methods.
| Method | What is trained | Extra inference overhead | Common use |
|---|---|---|---|
| LoRA | Low-rank matrices, which can be merged into weights | Almost none after merging | General fine-tuning; most mature ecosystem |
| Prefix Tuning | Virtual prefix key/value per layer | Longer sequence, with compute overhead | Generation tasks |
| Adapter | Small bottleneck modules inserted between layers | Extra compute | Multi-task switching |
| Bias tuning | Bias terms only | Very small | Scenarios with very few parameters |
| Zero-init attention | Top-layer adapter prompts and gates | Extra attention compute | Low-cost instruction adaptation |
Parameter efficiency does not mean inference is resource-efficient. Training updates fewer parameters, but inference still loads the full base model, and VRAM requirements are determined by the base model.
3. Evaluation Metrics and Reproducibility Conditions
A single sample image can show that a model understands an image, but it cannot show that the model is reliable on a class of tasks. The three examples discussed—writing a poem, identifying a washing machine, and asking a Chinese question—do not give the model version, input image, decoding parameters, or repetition count. They can serve only as demonstrations, not as performance evidence.
Common evaluation dimensions are as follows.
- Visual question answering: VQAv2 and GQA, reporting accuracy.
- General capability: MME, MMBench (including the Chinese version), SEED-Bench, and MMMU, covering perception, reasoning, and subject knowledge respectively.
- Text recognition and document understanding: OCRBench and DocVQA, suitable for receipts, tables, and screenshot scenarios.
- Hallucination: POPE, which measures the false-positive rate by asking whether an object exists in the image.
- Image captioning: COCO Caption and Flickr30k, commonly using CIDEr.
A credible evaluation report should specify at least the following.
- Model name, version number, and code commit ID, as well as the checksum of the weight file.
- Image input resolution and preprocessing method. The number of visual tokens grows with resolution, directly affecting speed and VRAM.
- Prompt template, system prompt, and number of examples. The same model may score noticeably differently with a different template.
- Decoding parameters: temperature, top_p, and maximum generation length. Multiple-choice evaluation usually uses deterministic decoding.
- Hardware and inference precision, such as fp16, bf16, or int4.
- Number of runs, random seeds, and the range of result variation.
Use identical settings for comparisons. Scores obtained by different teams under different conditions cannot be directly ranked.
4. Selection Checklist and Deployment Risks
Before selecting a model, confirm the following five items one by one.
- Task type. Question answering, OCR, image-text generation, and video understanding have different requirements. Teams working on receipt recognition should prioritize OCR-related metrics rather than poetry-writing ability.
- Compute budget. Estimated by parameter count, a 7B model requires about 14 GB of VRAM for weights in fp16, and about 3.5 GB after int4 quantization. Actual usage also includes activations, KV cache, and the vision encoder.
- License. Before commercial use, confirm the license terms for the code, weights, and training data separately. Some models prohibit commercial use or require attribution.
- Language capability. When Chinese is a hard requirement, test with your own Chinese dataset rather than relying on English benchmarks.
- Context and resolution. Long conversations and multi-image inputs consume a large context window. High-resolution images produce more visual tokens, which increases time to first token.
Common deployment risks are as follows.
- Hallucination. The model may describe objects or attributes that are not in the image. Before launch, use POPE-style tests and manual spot checks to estimate the false-positive rate, and prompt users in the interface to verify results.
- Accuracy drop after quantization. After quantization, rerun evaluation; do not look only at inference speed.
- Framework differences. The same set of weights may have inconsistent sampling implementations or image preprocessing across inference frameworks. Before launch, run an end-to-end comparison with fixed samples.
- Version changes. After upstream repository updates, prompt templates or weights may change. Lock versions and keep a rollback path.
Common mistakes include judging capability from a few sample images, comparing with different prompt templates, and failing to re-evaluate after quantization or switching inference frameworks.
5. Compliance and Safety
- Content filtering. When providing services externally, set up moderation for both input images and output text, and comply with applicable generative AI service regulations.
- Privacy protection. Images often contain personal information such as faces, license plates, and addresses. Apply de-identification during collection, storage, and training, and define retention periods.
- Training data copyright. Confirm the source and license of instruction data and image-text data, and avoid data of unknown origin or data whose license prohibits training use.
- Security testing. Conduct dedicated tests for prohibited content, induced outputs, and prompt injection in images, and keep the test sets and results.
References
- Zhang R, et al. LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention. arXiv:2303.16199.
- Gao P, et al. LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model. arXiv:2304.15010.
- Bai J, et al. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv:2308.12966.
- Chen Z, et al. InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. arXiv:2312.14238.
- Wang W, et al. CogVLM: Visual Expert for Pretrained Language Models. arXiv:2311.03079.
- Liu H, et al. Visual Instruction Tuning. arXiv:2304.08485.
