Open-Source Multimodal LLM Architecture Selection and Deployment: A Guide to Chinese Models
This article provides a practical guide for developers selecting and deploying open-source multimodal LLMs from Chinese developers. It covers the model landscape, common architecture, parameter-efficient fine-tuning methods, reproducible evaluation, a cloud deployment checklist, and the boundaries of model selection, with the goal of helping readers make more reliable technical decisions.
Open-Source Multimodal LLM Architecture Selection and Deployment: A Guide to Chinese Models
This article is intended for developers who need to select or deploy multimodal models. One point should be clear before reading: model versions iterate quickly. The series and papers listed here are valid as of writing. Before formal selection, refer to the latest official model cards and repository documentation.
1. Define the Scope and Model List First
Multimodal models in the Chinese open-source community fall roughly into two categories. One is full multimodal LLMs, composed of a vision encoder, a connector, and a language model, which can be used directly for image-text question answering and instruction dialogue. The other is parameter-efficient fine-tuning methods. These are not complete models themselves; they are used to inject visual or instruction capabilities into an existing large language model. The two categories are often discussed together, but they should be evaluated separately during model selection.
The table below lists representative work covered in this article. The numbers are arXiv paper IDs.
| Name | Developer | Reference | Main modalities |
|---|---|---|---|
| Qwen-VL / Qwen2-VL / Qwen2.5-VL | Alibaba Tongyi Team | arXiv:2308.12966, 2409.12191, 2502.13923 | Image, text; some versions support video |
| CogVLM | Zhipu AI and Tsinghua University | arXiv:2311.03079 | Image, text |
| InternVL series | Shanghai AI Laboratory et al. | arXiv:2312.14238, 2412.05271 | Image, text; some versions support video |
| DeepSeek-VL | DeepSeek | arXiv:2403.05525 | Image, text |
| LLaMA-Adapter / LLaMA-Adapter V2 | Shanghai AI Laboratory et al. | arXiv:2303.16199, 2304.15010 | Image-text instruction fine-tuning methods |
The information in the table comes from each project’s public papers. It may be updated after release, so verify each item before use.
2. General Architecture: Encoder, Connector, and Language Model
A multimodal model usually has three stages. The vision encoder, commonly based on a ViT structure, splits an image into patches and maps them to vectors. The connector converts visual vectors into a space the language model can understand. Common approaches include linear projection, a small multilayer perceptron, and a resampling module with learnable query vectors. The language model receives text and visual vectors and generates answers.
The connector design directly affects quality and cost. More output visual tokens preserve more detail, but the context becomes longer, inference is slower, and memory usage is higher. Some models use dynamic resolution, splitting a large image into multiple sub-images. This improves detail but increases computation as the number of splits grows. During selection, first confirm the model’s maximum supported resolution and visual token count, then estimate latency for your own workload.
Bilingual output depends mainly on the language model base and the language ratio in the instruction data. A documentation claim of Chinese-English support does not mean consistent performance across all tasks. Validate with Chinese instructions and English output samples from your own business scenarios, and watch for language mixing and terminology translation errors.
3. Comparison of Parameter-Efficient Fine-Tuning Methods
Full fine-tuning is costly and can cause the model to forget pretrained knowledge. Common parameter-efficient fine-tuning methods are listed below.
| Method | Parameters updated during training | Inference overhead | Main risks |
|---|---|---|---|
| Prompt tuning | A small number of learnable vectors at the input | Almost none | Limited gains on complex tasks |
| Adapter | Small bottleneck modules inserted between layers | A small amount of additional forward computation | Module position and size require tuning |
| LoRA | Low-rank matrix bypass | None after merging back into the original weights | Poor rank and learning rate choices can cause underfitting or overfitting |
| Bias and normalization tuning | Biases and normalization parameters within layers | Almost none | Limited expressiveness; needs to be combined with other data |
Several LLaMA-Adapter design choices deserve separate mention. It inserts learnable adaptation prompts into high-level Transformer layers and uses zero-initialized gated attention. The gate starts at zero, so at the beginning of training the new module’s output is zero and the original model’s behavior is unchanged. As training progresses, instruction and visual information are gradually injected. LLaMA-Adapter V2 builds on this by unfreezing some bias and normalization parameters and integrating visual information into the language model earlier. The “linear layer bias adjustment” discussed in the source corresponds to this type of bias-parameter training.
There are three common ways to prevent forgetting. First, mix in a certain proportion of pure-text instruction data. Second, use a smaller learning rate and fewer training epochs. Third, after each training run, perform a pure-text benchmark regression test and observe whether scores on common-sense QA, code, or math tasks drop.
4. Evaluation Methods: Make Conclusions Reproducible
“Good performance” needs to be broken down into measurable indicators. The following steps are recommended.
- Build a fixed evaluation set. Stratify it by task, for example image-text QA, OCR, fine-grained recognition, and Chinese-English mixed instructions. At least 200 samples per category are recommended. With too few samples, single-run variance can mask real differences.
- Fix prompt templates and decoding parameters, such as setting temperature to 0. Otherwise, result differences cannot be attributed to the model rather than the settings.
- Choose metrics. Use accuracy for multiple-choice and classification tasks, and keep a confusion matrix to see which categories concentrate errors. For open-ended generation tasks, use human rubric scoring, with common dimensions such as factual correctness, description completeness, and language consistency. Aesthetic tasks such as poetry creation are difficult to measure with a single automatic metric and should rely mainly on human scoring.
- Set up control groups. At minimum, include the base model, the model before and after fine-tuning, and a pure-text regression test.
- Collect failure samples. Common issues include evaluating only image captions while ignoring common-sense reasoning; insufficient image resolution causing missed small text and details; and English outputs under Chinese instructions mixing Chinese terms or confusing languages.
Public benchmarks such as MMBench, MMMU, and OCRBench can serve as references for horizontal comparison, but their distribution usually differs from business data. Final decisions should be based on a self-built evaluation set.
5. Cloud Deployment Checklist
Confirm the following items before launch.
- Version locking: Record the model weight version number or hash, and also lock the tokenizer and inference framework versions.
- Memory estimation: Weight footprint is approximately the parameter count multiplied by bytes per parameter. Under FP16, this is about 2 GB per billion parameters. Add KV cache and visual token overhead. After estimation, stress test with real requests; do not rely only on theoretical values.
- GPU specifications: Use different resource pools for training and inference. After stress testing to determine acceptable concurrency, decide instance type and replica count.
- Storage and data: Put images and training data in object storage, and set lifecycle rules to clean up temporary files. Record the source and authorization status of training data.
- Input validation: Limit image size, resolution, and format; set request timeouts and rate limiting to prevent oversized images from overwhelming the service.
- Logging and privacy: Log request ID, latency, and token count. Do not save raw images by default; if retention is necessary, anonymize first. Images may contain faces, license plates, and other personal information, so processing must comply with applicable personal information protection regulations.
- Monitoring and alerting: Monitor GPU utilization, memory usage, P95 latency, and error rate, and set threshold alerts.
- Cost accounting: Calculate per-request inference cost based on GPU time consumed per request, and compare on-demand billing with reserved-instance total costs. Estimate peak-traffic costs separately.
- Canary and rollback: Route a small percentage of traffic to the new version first, and switch fully after metrics meet targets. Keep the previous weights and service image, write rollback steps into the runbook, and drill regularly.
- Access control: Service accounts should follow the least-privilege principle. Store API keys in a secret management service, not in code or logs.
6. Three-Step Selection and Applicable Boundaries
- Clarify the task and modality. Distinguish needs such as image-text QA, OCR, and video understanding, and determine the required model series and resolution requirements.
- Use a self-built evaluation set to run small-scale tests on two or three candidate bases, and rank them by accuracy and failure samples.
- Fine-tune only when the base model cannot meet requirements. Start with LoRA or bias-based methods where possible, and complete launch according to the checklist.
The methods in this article apply to image-text understanding and generation tasks. For high-risk scenarios such as long-video understanding, medical diagnosis, and judicial judgment, even if a model performs well on an evaluation set, professional review is still required, and decisions should not rely solely on model output.
