Open Source Multimodal LLMs in China: A Technical Landscape and Selection Guide
This article provides a practical technical guide for evaluating and adopting open source multimodal large language models (LLMs) originating from China. Instead of offering a static model list, it builds a capability matrix tied to real business scenarios, covering visual-language understanding, video understanding, native unified multimodal models, visual generation, perception and edge models, audio-full-modality exploration, and multimodal agents. It discusses architecture considerations, training data and pipeline design, evaluation methodology with private benchmarks, engineering deployment as a complete pipeline, fine-tuning strategies, license and compliance checks, and common pitfalls. The conclusion emphasizes a closed-loop process of capability matrix, private evaluation, and production-grade engineering.
From a Static List to a Capability Matrix
When discussing open source multimodal LLMs from China, two common mistakes are to simply enumerate model names or to treat public benchmark scores as evidence of production readiness. The ecosystem changes rapidly, and the degree of openness varies widely: some models release weights but lack training data and evaluation scripts; some release code but impose license restrictions on commercial redistribution; some support image understanding but not video temporal reasoning; some can generate images but with limited controllability. A more professional answer is not a static list but a capability matrix tied to business scenarios.
This article does not point to specific third-party names but organizes the landscape by transferable technical routes. For real projects, build your own candidate list based on five facets: public repositories, model documentation, license texts, reproducible experiments, and internal evaluation.
“Open source” should be verified at four levels: whether code is open, whether weights are open, whether training data and recipes are open, and whether the license permits the intended use. All four are necessary; “downloadable weights” alone does not equal full open source.
Major Technical Routes in the Open Source Multimodal Ecosystem
From publicly visible technical forms, open source multimodal LLMs in China can be grouped into the following categories:
| Category | Typical Goal | Architecture Focus | Selection Concerns |
|---|---|---|---|
| Visual-language understanding & document QA | Image QA, chart understanding, table parsing, multi-image comparison | Vision encoder + cross-modal connector + language backbone | High-resolution handling, OCR, coordinate reasoning, hallucination control |
| Video & long-sequence understanding | Video QA, event localization, action recognition, streaming analysis | Keyframe sampling + temporal aggregation + memory compression | Effective context, time-to-first-frame, keyframe strategy, temporal annotation |
| Native unified multimodal models | Interleaved image-text reasoning, cross-modal inference, unified generation | Early fusion + unified representation + joint training | Data mix, training cost, stability, cross-modal interference |
| Visual generation & editing | Text-to-image, video generation, localized editing, style control | Generative backbone + conditional control + safety filtering | Controllability, copyright risk, content safety, subjective evaluation |
| Perception segmentation & edge lightweight models | Segmentation, detection, localization, edge deployment | Multi-scale encoding + prompt-based decoding + distillation/quantization | Boundary accuracy, real-time performance, memory usage, device compatibility |
| Audio & full-modality exploration | Speech, image, text joint understanding | Modality-specific encoders + alignment fusion + long-audio processing | Synchronization quality, noise robustness, privacy protection |
| Multimodal agents & GUI operation | Screen understanding, task planning, tool invocation | Screenshot understanding + text state + action space constraints | Rollback, sandbox isolation, permission control, task success rate |
1. Visual-Language Understanding and Document QA
This is the most concentrated area for enterprise adoption. Typical tasks include image-based QA, receipt and form recognition, document layout understanding, chart trend explanation, multi-image comparison, and fine-grained localization. These models usually adopt a modular architecture: a vision encoder extracts image features, a cross-modal connector maps them into the language model’s representation space, and the language backbone performs reasoning.
When selecting, do not only look at general QA quality. Focus on:
- Whether small text, table lines, and legends in high-resolution images are readable;
- Whether table row/column relationships, reading order, and cross-page information are preserved;
- Whether output coordinates can be mapped back to the original image;
- Whether entities from different images are mixed when multiple images are input;
- Whether the model can refuse to answer when no answer exists, reducing hallucination.
Document tasks must avoid simple image scaling. Proper preprocessing often includes orientation correction, local cropping, text-region enhancement, layout structure extraction, and coordinate normalization. Many errors come from lost crop boundaries, broken table structures, or incorrect coordinate mapping rather than the model itself.
2. Video and Long-Sequence Understanding
Video multimodal processing is not just concatenating multiple images. It must handle event order, causality, action boundaries, and audio-visual synchronization along the time dimension. Common engineering paths include keyframe sampling, scene-change detection, extended temporal position encoding, cross-frame memory compression, and streaming caches. For long videos, the model’s nominal context window does not equal “effective temporal context.” What matters is key-event recall rate, time-localization error, and long-range dependency retention.
Selection considerations:
- Performance degradation curves for 5-minute, 30-minute, and multi-hour videos;
- Time-to-first-frame latency and total completion time;
- Support for streaming input, interrupted continuation, and event backtracking;
- Joint use of subtitles, speech, and on-screen text;
- Stability of fine-tuning when temporal annotations are scarce.
If the business only needs summaries, do not pursue complex temporal reasoning initially. But for alarm, forensics, or event localization, extra evaluation of timestamp accuracy and false-alarm cost is essential.
3. Native Unified Multimodal Models
Modular architectures connect a vision module to a language backbone; they are mature and iterate quickly. Native unified multimodal models emphasize joint training from the pretraining stage with mixed text, image, and video information through unified representations or interleaved data. The potential advantage is more natural cross-modal reasoning and better modality transfer, but training costs are higher and data mixing and stability requirements are stricter.
Key concerns when choosing a native unified route:
- Whether pure text capability degrades due to multimodal training;
- Whether image understanding and generation truly share representations or only expose a unified interface;
- Whether coreference resolution in interleaved image-text inputs is stable;
- Whether performance is sensitive to changes in modality sampling ratios.
In practice, start with a modular model for a pilot, then decide whether to move to a native unified architecture based on cross-modal complexity. Do not adopt “native” merely because it sounds more advanced without considering data and operational costs.
4. Visual Generation and Editing
Understanding models answer “what is in the image”; generation models solve “what to generate or modify.” Text-to-image, video generation, localized editing, and reference-image control typically require a generative backbone, conditional control modules, safety filters, and copyright policies. Evaluation is more subjective than understanding tasks. Rather than relying on single examples, build a review set covering subject consistency, instruction following, spatial relationships, text rendering, hand structures, style consistency, and content safety.
Production deployment also requires attention to:
- Whether generated content is traceable and whether necessary watermarks or logs are retained;
- Whether sensitive content blocking and minor protection are supported;
- Whether user input may contain other people’s portraits, copyrighted material, or sensitive information;
- Per-batch generation cost and failure retry cost.
5. Perception Segmentation, Localization, and Edge Lightweight Models
Segmentation, detection, and localization models are often used as the “visual perception base” in multimodal systems. They output masks, bounding boxes, keypoints, or region descriptions, which a language model then interprets and uses for decisions. Key metrics are not general QA scores but boundary accuracy, small-object recall, occlusion robustness, temporal consistency in video, and real-time frame rate.
For edge deployment, include compression, quantization, distillation, and hardware adaptation in early design. A common strategy is “small model resident + large model on demand”: run a lightweight model for initial screening, and escalate to a stronger model when confidence is low or the problem is complex. This reduces average cost but requires careful escalation rules to avoid tail latency issues from frequent fallback.
6. Audio and Full-Modality Exploration
Joint audio, video, and text understanding applies to meeting analysis, customer-service quality checks, content moderation, and equipment inspection. Technical challenges include modality synchronization, long-audio segmentation, noise robustness, and privacy protection. Evaluation must not only measure speech-to-text accuracy in quiet environments but also multi-speaker conversations, dialects, background noise, and far-field pickup.
If the business only involves “speech-to-text plus text analysis,” there is no need to adopt a full-modality large model. A staged approach is safer: first validate speech-to-text and text understanding, then gradually introduce image, video, and cross-modal alignment.
7. Multimodal Agents and GUI Operation
Agent applications must understand screenshots, UI elements, text states, and action history, then generate clicks, inputs, scrolling, and other actions. The core technical issues are not single-turn image recognition but state tracking, action planning, failure recovery, and safety constraints.
At least three safety lines should be set:
- Execute actions in an isolated environment, never directly on production systems;
- Require approval, second confirmation, and rollback for high-risk actions;
- Constrain the action space with a rule engine to prevent out-of-bounds actions.
Evaluation metrics include task success rate, average number of steps, recovery success rate, unauthorized access attempt rate, and human takeover rate.
Architectural Key Points: Distinguish “Marketing Features” from “Production Variables”
1. Vision Encoding and Cross-Modal Connection
Modular multimodal models consist of a vision backbone, a connector, and a language backbone. The vision backbone extracts local and global features; the connector maps visual representations into the language space. The connector’s form affects alignment efficiency, trainable parameters, and inference overhead. For enterprise developers, what matters is interface stability, structured input/output, and batch/async support.
2. Dynamic Resolution and Token Budgets
High resolution is critical for documents, industrial inspection, and remote sensing, but higher image resolution means more visual tokens and higher inference costs. Production systems often adopt “on-demand resolution”: first run a lightweight detection to judge text density or object size, then decide whether to crop, enlarge, or compress. For ordinary chat images, reduce cropping; for receipts and tables, preserve local details.
Be wary of models that used specific preprocessing in evaluation that cannot be reproduced in the business. Include preprocessing code, coordinate mapping, and image decoding in the reproducibility scope.
3. Long Context, Streaming Memory, and Temporal Position Encoding
Video and long documents require long context, but “long enough window” does not mean “no information loss.” Reasonable solutions include hierarchical summarization, keyframe caches, temporal retrieval, streaming memory, and important-segment replay. Position encoding extensions can alleviate long-sequence modeling, but if the training data lacks enough long sequences, inference may still be inaccurate.
4. Unified Generation and Understanding
Whether understanding and generation share a unified architecture is a major divergence. Unified architectures benefit multi-turn interaction and cross-task knowledge sharing, but training complexity, safety governance, and evaluation costs are higher. For most enterprise applications, understanding and generation models can be selected separately and combined through an orchestration layer, without forcing a single model to handle all tasks.
Training and Data: The Real Barrier in Multimodal Projects
1. Data Composition
Typical multimodal training data includes image-text pairs, interleaved image-text documents, tables, charts, OCR regions, visual localization boxes, video clips, audio transcriptions, and action trajectories. Data quality matters more than scale:
- Deduplicate, denoise, and de-template image-text pairs to avoid “title is the answer”;
- Preserve layout, reading order, table structure, and coordinates in document data;
- Annotate temporal boundaries, key events, and audio-visual synchronization in video data;
- Record state, action, result, and failure reason in agent trajectories.
2. Training Stages
A stable path is staged training:
- Basic alignment: establish correspondence between image, video, audio, and text;
- Multi-task instruction tuning: add QA, localization, document parsing, video summarization, etc.;
- Preference and safety optimization: combine human ranking, rule validation, and automatic evaluation to constrain output style and safety;
- Domain adaptation: fine-tune on low-risk business data while replaying general capabilities;
- Regression evaluation: full regression on old capabilities, edge cases, and high-risk inputs.
Preference optimization can use a combination of human ranking, rule filtering, and model-assisted scoring, but do not rely solely on one automatic scorer, as it may amplify scoring bias. Test-time augmentation such as multi-crop, prompt ensembling, retries, and self-consistency voting can improve robustness but increase latency and cost; enable them via gating mechanisms based on scenario.
3. Data and Evaluation Isolation
Multimodal data overlaps more easily with public evaluation sets. Establish a decontamination process to ensure training data does not contain evaluation images, questions, or approximate variants. For documents and charts, also prevent template-based samples from making the model “memorize layouts” rather than learn reasoning.
Evaluation and Selection: Build a Private Scenario Benchmark
We recommend a four-layer evaluation:
| Layer | Focus | Example Metrics |
|---|---|---|
| Basic capability | General visual QA, OCR, localization, video understanding | Task accuracy, localization error, temporal recall |
| Business capability | Receipt fields, chart conclusions, work-order screenshots, QC images | Field accuracy, end-to-end success rate, manual rework rate |
| Robustness & safety | Blurred images, adversarial noise, induced inputs, sensitive content | Refusal rate, out-of-bounds rate, hallucination rate, safety block rate |
| Engineering efficiency | Latency, throughput, memory, cost, maintainability | P95 time-to-first-token, per-request cost, failure rate |
The following pseudocode can be used for a scoring model:
candidates = read_candidate_list()for each model in candidates:quality_score = scenario_test(model, private_eval_set)robustness_score = adversarial_and_hallucination_test(model)latency_score = p95_time_to_first_token(model)cost_score = estimate_per_request_cost(model)compliance_risk = review_license_data_source_safety(model)composite = w1*quality_score + w2*robustness_score- w3*latency_score - w4*cost_score- w5*compliance_riskadd_to_pareto_frontier(model, composite)
The weights depend on business. Customer service may prioritize time-to-first-token and refusal ability; document extraction prioritizes field accuracy; video inspection prioritizes temporal recall and false-alarm cost; offline batch processing may relax latency requirements.
Engineering Deployment: Treat Multimodal as a Full Pipeline
A multimodal service is not just “pass an image to the model.” A stable production system usually includes:
- Access layer: authentication, rate limiting, request validation, tenant isolation;
- Preprocessing layer: decoding, orientation correction, on-demand cropping, text-region enhancement, video keyframe extraction, audio segmentation;
- Inference layer: dynamic batching, token budget control, visual feature caching, multi-model routing, async tasks;
- Post-processing layer: structured parsing, coordinate mapping, confidence judgment, citation and evidence generation;
- Observability layer: logs, metrics, tracing, quality sampling, cost accounting, and alerts;
- Security layer: bidirectional input/output filtering, sensitive-information masking, audit trails, and human review.
Deployment optimization directions:
- Dynamically determine the number of crops based on text density, avoiding maximum resolution for all requests;
- Cache visual encoding results for repeated images or pages;
- Use a lightweight model for simple questions and escalate complex ones to a stronger model;
- Use async tasks for videos, returning summaries first and filling in temporal localization later;
- Optimize prefill and decode stages separately, focusing on P95 latency rather than average latency;
- Set independent quotas for multi-tenant scenarios to prevent a single large request from degrading the whole service.
On the cloud, generic capabilities such as object storage, message queues, container platforms, logging, and monitoring can be combined. The key is not the tool name but closing the observable loop: “input governance — async tasks — model service — result write-back — quality sampling.”
Fine-Tuning and Deployment Example
Multimodal fine-tuning data should explicitly represent text, images, videos, regions, and expected outputs, avoiding mixing multiple types into an unparseable string. Below is a neutral pseudocode example:
sample = {"input": [{"type": "text", "value": "Explain the trends in the chart and point out anomalies."},{"type": "image", "uri": "object-store://image-1.png"},{"type": "video", "uri": "object-store://video-1.mp4", "time_range": [0, 12.5]}],"target_output": {"type": "text", "value": "..."},"region_annotations": [{"box": [0.12, 0.20, 0.35, 0.42], "label": "legend"}]}Training strategy:1. Freeze the vision backbone; train only the connector and a lightweight adapter layer.2. Add general instruction data for replay to prevent text capability degradation.3. For document tasks, add reading-order and coordinate supervision.4. For video tasks, add event temporal boundary and keyframe selection supervision.5. For agent tasks, add action constraints and failure recovery samples.6. Save regression baselines at every stage; roll back if metrics degrade.
Common pitfalls in fine-tuning include: using full-parameter training with very little business data, training and validation sets coming from the same template, inconsistent coordinate normalization, differing video frame sampling between training and inference, and optimizing only the final answer while ignoring intermediate localization. Archive preprocessing versions, model versions, and evaluation results together.
Licensing, Compliance, and Safety Governance
Open source model licenses vary greatly, and the authorization for weights, code, and training data is often independent. Before production deployment, at least verify:
- Whether commercial use, redistribution, model distillation, and derivative release are permitted;
- Whether attribution, preservation of notices, or restrictions on sensitive use are required;
- Whether training data contains privacy, copyright, or provenance risks;
- Whether output meets industry content-safety and audit requirements;
- Whether an acceptable use policy, risk description, and model card are provided.
Safety governance should cover input, inference, output, and operations. Input side: sensitive information detection and masking. Inference side: system constraints and permission boundaries. Output side: safety filtering and factuality checks. Operation side: audit logs, sampling records, and emergency rollback. Open source does not absolve enterprises from final responsibility for the service.
Common Misconceptions
- Treating open weights as full open source — ignoring data, training recipes, and license differences.
- Relying only on public benchmark rankings — ignoring preprocessing differences, benchmark contamination, and business distribution shifts.
- Testing only single-image QA — missing documents, tables, multi-image, video, and long-tail inputs.
- Blindly pursuing full-parameter fine-tuning — small data leads
