Open-Source Multimodal LLM Selection: Three Routes and Pre-Training Checks
This article presents a constraint-first approach to selecting open-source multimodal LLMs. It explains three common training routes—zero-initialized attention, bias-only fine-tuning, and visual projection—and provides pre-training checks for VRAM estimation, task evaluation, licensing, route decisions, and maintenance status.
1. Define Constraints Before Choosing a Route
Multimodal LLMs can process images and text, but open-source models vary widely in parameter scale, visual capability, Chinese-language support, and license terms. Selection failures usually fall into three categories: insufficient VRAM, unstable Chinese image understanding, and licenses that do not permit the intended use. All three can be identified before training. Therefore, it is more compute-efficient to define four constraints, decide on a technical route, and then compare specific models than to start with leaderboard rankings.
The four constraints are: available VRAM or GPU model; compute budget for training and inference; target tasks (such as Chinese OCR, chart question answering, and image-text retrieval); and commercial and distribution requirements. Put these on a single page and use it as the basis for every later step.
Model names, versions, and licenses change quickly. This article does not list specific model names or provide performance rankings. Before deployment, use the current versions in the official repositories and licenses.
2. Applicability of Three Technical Routes
Common open-source options can be grouped roughly into three routes. They differ in compute requirements, capability ceiling, and risk.
Zero-initialized attention: Insert a lightweight gated module next to the language model’s attention layers and initialize the gate parameters to zero. At the start of training, the model output is exactly the same as the original model; visual information then enters gradually. This route suits teams that want to preserve existing language capability and have medium compute. There are three main risks: it still requires a certain scale of image-text data; incorrect insertion positions or gating design can lead to insufficient use of visual information; and the original model license and the derivative model license must be checked separately. If compute is only enough for inference, or the task requires deep chart understanding, this route is usually not suitable.
Bias-only fine-tuning: Freeze most weights and train only bias terms or a very small number of parameters. VRAM and storage overhead are lowest, making this suitable for small data and rapid adaptation. The risk is limited expressiveness, which makes it difficult to learn entirely new visual features. If the visual encoder itself is weak at Chinese text recognition, bias-only tuning will hardly compensate. This route is not suitable for tasks that require high precision, such as Chinese OCR or fine-grained chart reading.
Visual projection: A general architecture of “vision encoder + projector + language model,” usually trained in two stages. In stage one, only the projector is trained so that visual features align with the language space. In stage two, the language model is instruction-tuned. This route has the highest capability ceiling and suits teams with sufficient image-text data and training budget. It also has the most risks: high training cost; alignment data quality directly determines results; two-stage configuration is complex; and catastrophic forgetting—degradation of language ability—can occur. Teams with only a few hundred examples or inference-only capacity generally should not choose it.
| Route | Trainable parameters | Suitable scenarios | Main risks | Pre-launch validation |
|---|---|---|---|---|
| Zero-initialized attention | Medium | Preserve original language capability; medium compute | Need for image-text data; gating design affects results | Pure-text capability regression test; image-text evaluation |
| Bias-only fine-tuning | Very low | Small data; rapid adaptation | Limited expressiveness; visual features hard to learn | Compare image-text accuracy with a baseline model |
| Visual projection | High | Strong visual understanding; sufficient data and budget | High training cost; catastrophic forgetting | Stage-wise alignment checks; bidirectional pure-text and image-text evaluation |
3. Five-Step VRAM Estimation
VRAM estimation is used to judge feasibility before renting GPUs or starting training. The formulas below are estimates; calibrate them with a small-batch run before a full run.
Step 1: Determine parameter count and weight precision. Weight memory equals parameter count multiplied by bytes per parameter. FP32 is 4 bytes, FP16 and BF16 are 2 bytes, INT8 is 1 byte, and INT4 is about 0.5 bytes. For example, a 7B-parameter model loaded in BF16 requires about 14 GB for weights.
Step 2: Add the vision encoder and projector. Calculate vision encoder parameters separately; it may be frozen during training but still occupies VRAM. The projector has few parameters, but its parameters must be counted as trainable parameters.
Step 3: Estimate inference cache or training overhead. During inference, KV cache size is approximately: 2 × number of layers × number of KV heads × head dimension × sequence length × batch size × bytes per element. During training, account for gradients, optimizer states, and activations of trainable parameters. With AdamW under common settings, optimizer states take about 8 bytes per trainable parameter. Schemes that freeze most parameters have much lower optimizer overhead.
Step 4: Plug in batch size and sequence length. Visual inputs are converted into many visual tokens, which count toward sequence length. A high-resolution image or multi-image input may produce far more tokens than the text itself. The estimate must include visual tokens, not only text length.
Step 5: Apply a redundancy factor and state the scope. Use a redundancy factor of 1.2 to 1.5 to account for frameworks, memory fragmentation, and temporary buffers. This is an empirical range, not a fixed standard. The final result must state whether it applies to a single GPU or multiple GPUs. For multiple GPUs, also explain the partitioning method—such as tensor parallelism, data parallelism, or sharding strategies such as ZeRO and FSDP—and communication overhead. Compare the estimate with the available VRAM on the target GPU and reserve a team-defined margin.
4. Evaluation Checklist
Evaluation must be completed before launch and use only the team’s own test set. Official scores for open-source models can be used as references but cannot replace results for the target scenario.
- Chinese OCR: Cover printed text, handwriting, Chinese in tables, and mixed text. Record character accuracy and line accuracy.
- Chart understanding: Include numerical reading, trend judgment, and legend recognition. Record numerical error and judgment accuracy separately.
- Image-text retrieval: Calculate recall; common metrics are R@1, R@5, and R@10.
- Visual question answering: Calculate accuracy using a unified scoring rule, and pay attention to whether answer formats are normalized.
- Hallucination and refusal: Provide information that is not in the image and check whether the model fabricates an answer and whether it can correctly answer “unable to determine.”
- Long and multiple images: Test inputs such as stitched long images and multi-page screenshots, and record behavior when the context length is exceeded.
- Inference latency and throughput: Record first-token latency, output tokens per second, and request throughput under concurrency.
- Pure-text capability: Run regression checks with the test set for the original language task and monitor catastrophic forgetting.
Use a consistent evaluation protocol: same hardware, same precision, same prompt template, same test set version; record the random seed and maximum generation length. The passing threshold for each item is determined by the team’s business requirements; this article does not set universal values.
5. License Checklist
License issues must be clarified before training. The checklist below is for organizing questions and does not replace legal advice.
- Commercial use: Is commercial use allowed? Are there revenue thresholds or user-count limits?
- Redistribution: Can weights be packaged and delivered to customers, or shared within the team?
- Derivative models: Do fine-tuned weights remain subject to the original license, and is there a requirement to open-source under the same license?
- Service use: Does offering the model as an API service require additional authorization?
- Patents and trademarks: Are there restrictions on using the model name or logo?
- Training data licensing: Does the training dataset license permit commercial use, and does it include data with restricted uses?
- Output usage restrictions: Some licenses restrict the use of generated content; confirm each item.
Keep records during the check: official repository URL, version number or commit ID, screenshot or hash of the license file, date of check, and reviewer. If any item cannot be determined, refer it to legal review and do not enter training before it is confirmed.
6. Decision Tree
The decision tree is executed in order. If any step does not pass, stop or pause.
Step 1: Is VRAM sufficient? After adding the margin, does the estimated requirement exceed the available VRAM on the target GPU?
- Yes: pause. Try a smaller parameter count, quantization, or lower resolution and sequence length, then re-estimate. If it still exceeds the available VRAM, do not select the model.
- No: proceed to step 2.
Step 2: Does the license permit the intended use? Are all three—commercial use, redistribution, and service use—permitted?
- No: do not select.
- Uncertain: refer to legal review and do not train before confirmation.
- Yes: proceed to step 3.
Step 3: Choose a route.
- Limited compute and need to preserve original language capability: prioritize evaluating bias-only fine-tuning or zero-initialized attention.
- Need strong visual understanding and have sufficient data and budget: evaluate visual projection.
- In between: try one route with a small-scale dataset, compare results, and then decide.
Step 4: Does evaluation meet the thresholds? On the team’s test set, do Chinese OCR and hallucination-and-refusal meet the preset thresholds?
- Any key item fails: stop training and do not select the model.
- All pass: proceed to training preparation.
Step 5: Is maintenance active? Have there been fixes and releases in the past year, and are issues being responded to?
- No: pause, or assess the cost of in-house maintenance.
- Yes: proceed to training.
7. Usage Notes
The formulas, checklists, and decision tree in this article are suitable for preliminary screening before training. Every step’s result must be confirmed with measured data. Model versions, license terms, and officially recommended deployment methods may change. Before a final decision, re-check the current versions in the official repositories and licenses, and record the date of the check in the project records.
