Open-Source Multimodal Large Models: Technical Paradigms and Selection Guide

    This article reviews how open-source multimodal large models have evolved from image-text retrieval and single-image question answering toward instruction-driven visual question answering and cross-modal deep reasoning. It explains the core technical components, including visual encoders and dynamic resolution, modality connectors, multimodal position encoding, and training objectives. It also covers data mixing, preference optimization, test-time augmentation, evaluation methods, a six-factor selection framework, memory estimation, deployment, fine-tuning, common pitfalls, and platform-level recommendations for teams evaluating or deploying these models.

    Open-Source Multimodal Large Models: Technical Paradigms and Selection Guide

    I. Why Open-Source Multimodal Models Need a Fresh Look

    Since 2023, the capability focus of open-source multimodal large models, commonly referred to as MLLMs or LMMs, has shifted twice. The first shift moved from image-text retrieval and image captioning toward instruction-driven visual question answering. The second shift moved from single-image question answering toward cross-modal deep reasoning, including multi-image input, high-resolution document parsing, long-video understanding spanning tens of minutes, and multilingual scenarios.

    This evolution makes open-source models a practical private-deployment alternative in many vertical use cases. At the same time, it introduces new engineering complexity: visual token counts grow rapidly, GPU memory and throughput become tightly coupled, long sequences create positional ambiguity, and image-text alignment competes with general text capability.

    For engineering teams, open-source multimodal models offer three clear benefits:

    1. Data can remain inside the organization and be deployed in a controlled compute environment.
    2. The model can be deeply fine-tuned with industry knowledge, document layouts, workflows, and other private priors.
    3. The model can be reduced through lower resolution, quantization, distillation, or other compression methods to bring inference cost into an acceptable range.

    To understand these benefits, teams need to break the model into analyzable components rather than treating it as a black-box API.

    II. Technical Paradigm Deconstruction: Four Critical Components

    2.1 Visual Encoders and Dynamic Resolution

    Early solutions often resized images to a fixed size, such as 224×224, before sending them to the visual backbone. This approach can erase small text in documents, tables, and receipts.

    Current open-source implementations commonly use dynamic resolution. Instead of forcing one fixed image size, the model splits the image into multiple tiles based on its original aspect ratio. It also keeps one global thumbnail image to preserve overall semantics. The visual features of each tile are then concatenated along the token dimension.

    1. # Illustrative pseudocode: dynamic-resolution tiling and visual-token budget
    2. MIN_EDGE, MAX_EDGE = 28, 1008 # Must be multiples of the visual patch size
    3. TOKENS_PER_TILE = 256
    4. MAX_VISUAL_TOKENS = 1536 # Total visual budget; directly affects memory and latency
    5. def plan_tiles(width, height):
    6. if max(width, height) * MIN_EDGE <= min(width, height) * MAX_EDGE:
    7. return [(width, height)] # Balanced aspect ratio: scale the whole image
    8. # Unbalanced aspect ratio: split into 2-6 columns and keep one global image
    9. cols = clamp(ceil(width / height), 2, 6)
    10. return grid_tiles(width, height, cols) + [(width, height)]

    Two practical rules follow from this design:

    • The number of visual tokens is approximately tile count × TOKENS_PER_TILE. This is the main variable determining GPU memory usage and time-to-first-token, while parameter count mainly determines weight footprint.
    • The same model can have very different costs in high-precision document mode versus high-throughput mode. The difference can be several times. This switch should be controlled at the business configuration layer rather than inside the model layer.

    2.2 Modality Connectors: Three Generations of Design

    • Linear projection, such as an MLP projector: This projects visual features into the language model’s embedding space. It is the simplest to implement and fastest to train, but the number of visual tokens cannot be compressed. Long images and long videos therefore create heavy context pressure.

    • Query-based resampling: This introduces a set of learnable queries and uses cross-attention to compress variable-length visual features into a fixed number of tokens. It keeps GPU memory more predictable, but it may lose fine-grained spatial information, which can hurt OCR-like tasks.

    • Native unified multimodal: The visual encoder and language model are jointly trained under the same pretraining objective. Images and text are handled as homogeneous token streams. This usually performs better on cross-modal reasoning and fine-grained grounding, but training is expensive and iteration cycles are longer.

    The selection implication is straightforward:

    • For document parsing and receipt recognition, prioritize dynamic resolution with linear projection.
    • For long-video understanding and image-text retrieval, resampling is often more suitable.
    • For complex chart reasoning or cross-image comparison, native multimodal models should be considered first.

    2.3 Position Encoding: Why Multimodal RoPE Matters

    Text is naturally a one-dimensional sequence. An image is a two-dimensional grid. A video adds time, making it a three-dimensional structure combining spatial layout and temporal order.

    If visual tokens are forcibly flattened into a one-dimensional sequence, the model may struggle to distinguish whether content appears in upper and lower columns or side by side. It may also struggle to determine the order of similar frames.

    Multimodal rotary position embedding, commonly referred to as M-RoPE, addresses this by decomposing position into time, height, and width components. Rotary encoding is then applied to each component before being used in attention.

    This has two important engineering benefits:

    1. Long-video temporal localization becomes more stable.
    2. When multiple images are provided, the relative positions between different images do not interfere with one another.

    2.4 Training Objectives: The Two-Stage Alignment Paradigm

    The mainstream recipe uses a two-stage approach: contrastive alignment followed by generative fine-tuning. The first stage uses image-text contrastive loss to pull visual representations into language space. The second stage uses autoregressive language modeling loss on interleaved image-text corpora.

    A recent trend is to unify both stages into a single autoregressive loss. This removes the contrastive branch and additional alignment module. The training pipeline becomes simpler, and it often still provides meaningful gains on image-text retrieval tasks.

    Understanding this distinction helps estimate the cost of secondary training. A single-objective autoregressive model can more easily continue pretraining on business instruction data. In contrast, a two-stage model that only fine-tunes the projection layer may degrade on out-of-distribution data.

    III. Training and Optimization: From Pretraining to Preference Alignment

    3.1 Data Mixing Determines Whether Text Ability Degrades

    Public recipes for native multimodal pretraining commonly mix text data and multimodal data at a fixed ratio. A typical setup uses about one quarter text and three quarters multimodal data.

    The purpose of this mixture is to suppress text capability degradation. If the model is trained only on image-text pairs, its reasoning and coding abilities can decline significantly.

    For enterprises that only fine-tune the visual branch, it is advisable to retain 20%-30% pure-text instruction data in the mixed corpus as an anchor.

    3.2 Preference Optimization: From Looking Like an Answer to Being an Answer

    Supervised fine-tuning helps a model imitate annotation style, but it does not effectively suppress hallucination or imbalance in refusal behavior.

    The preference optimization stage uses paired samples, typically a preferred response and a rejected response, to explicitly constrain the output distribution.

    Single-objective preference optimization converges quickly on general dialogue, but it may provide insufficient constraints for visual factuality. Mixed preference optimization constructs both visual relevance preference pairs and general preference pairs. On comprehensive multimodal evaluations, it often performs closer to closed-source models.

    The most common mistake in practice is constructing preference pairs incorrectly. The difference between the preferred and rejected responses must come from visual facts, such as reading 12.50 as 125.0. It should not be based only on simple length or formatting differences.

    3.3 Test-Time Augmentation: Trading Inference Compute for Accuracy

    For the same model, multiple sampling passes followed by majority voting, or retries for high-uncertainty samples, often produce stable improvements. The tradeoff is that latency increases significantly.

    A practical approach is to limit test-time augmentation to offline batch scenarios, such as bulk receipt verification or content moderation review. Online serving should keep single-pass decoding and use confidence thresholds to decide whether human review is needed.

    IV. Evaluation: Do Not Rely on a Single Score

    A single aggregate score for multimodal evaluation can be highly misleading. It is better to evaluate along at least the following dimensions:

    Dimension Typical Task Format Focus
    Visual perception Attribute counting, spatial relations, fine-grained recognition Small objects and occlusion
    Documents and charts Layout reconstruction, table structure, field extraction Small fonts and cross-column reading
    Video temporal understanding Event ordering, long-range dependency question answering Sampling strategy and frame rate
    Multi-image relations Cross-image comparison, image-set reasoning Image order and maximum number of images
    Instruction following Format constraints, structured output JSON validity and refusal rate
    Factuality Hallucination rate, no-answer recognition Overconfidence

    Teams should also be aware of public benchmark contamination and distribution shift. A more reliable engineering signal comes from self-built evaluation sets.

    A good starting point is 200-500 real business samples covering both high-frequency scenarios and long-tail cases. Random seeds and decoding parameters should be fixed. The evaluation set should be rerun after every model or prompt change, and results should be checked for consistency with human annotations.

    V. Selection Framework: Six-Dimensional Scoring

    1. Modality coverage: Does the model natively support images, video, and audio? Does video support long duration?
    2. Visual fidelity: Does it support dynamic resolution? What are the maximum input edge length and visual token limit?
    3. Context budget: Does the visual token count share the same context budget as text tokens?
    4. Deployment form: Are weights open? Does the license allow commercial use? Does it support quantization?
    5. Fine-tuning capability: Is a complete training recipe available? What are the minimum GPU memory and GPU type requirements for fine-tuning?
    6. Ecosystem maturity: How well does it work with inference frameworks? Is the community active? Are documentation and issue responses high quality?

    When scoring, weighted scoring is better than equal weighting. For document-heavy business scenarios, visual fidelity and context budget should account for more than 40% of the total weight. For dialogue-oriented scenarios, inference framework compatibility and throughput are more important.

    VI. Inference and Deployment: Memory, Parallelism, and Serving

    6.1 Rough Memory Estimation

    1. Weight usage ≈ number of parameters × bytes per parameter (FP16=2, INT8=1, INT4≈0.5)
    2. KV cache ≈ 2 × number of layers × attention head dimension × sequence length × batch size × bytes
    3. Activations and intermediate states ≈ visual branch tensors + attention intermediates (roughly linear with tile count)

    A common mistake in practice is to ignore the intermediate activations created by the visual encoder and dynamic resolution. This can lead to a situation where the estimated memory based on parameter count appears sufficient, but the service crashes with OOM during startup.

    A safer approach is to reserve 15%-20% extra memory and make the maximum number of visual tokens configurable.

    6.2 Parallelism and Batching

    When the parameter scale is not large, tensor parallelism often provides limited benefit. In such cases, it is better to keep the visual encoder resident and use dynamic batching during the decoding phase.

    Visual encoding is compute-intensive, while autoregressive decoding is more memory-bandwidth-intensive. Because of this difference, splitting them into independent services and connecting them with a message queue is usually more stable than a monolithic deployment.

    Visual features often repeat across questions for the same document. Caching visual encoding results by content hash in object storage or a local cache layer is one of the highest-value optimizations for long-document multi-turn question answering.

    6.3 Supporting Capabilities for Serving

    A mature serving stack needs more than the model itself. It should also include:

    • Containerized deployment and elastic autoscaling.
    • Logging that retains complete requests and responses.
    • Monitoring and alerting for time-to-first-token, throughput, and failure rate.
    • A separate metric for structured output validation failure rate.

    It is also advisable to apply input-side rate limiting for multimodal requests. Metering and throttling should be based on visual token count rather than request count alone, so that oversized images do not fill the GPU queue.

    VII. Fine-Tuning and Data Engineering

    For low-cost customization, parameter-efficient fine-tuning should be considered first. This usually means freezing the visual encoder and most of the backbone weights, while training only the projection layer and low-rank adaptation matrices.

    Three data-engineering lessons are important:

    • Collect at least 1,000 samples per task category, covering layout variants, shooting angles, lighting differences, and noise differences.
    • Include negative samples, such as “the field is not present in the image” or “the table spans multiple pages.” Otherwise, the model may forcibly invent answers.
    • Keep about 10% general capability data, including pure-text reasoning and general image-text question answering, to prevent catastrophic forgetting.
    1. # Illustrative pseudocode: minimum field structure for multimodal training samples
    2. sample = {
    3. "images": ["/data/doc/0001.jpg"], # Multiple images are supported
    4. "messages": [
    5. {"role": "user", "content": "<image>\nExtract the total tax-inclusive amount from the invoice and output only the number"},
    6. {"role": "assistant", "content": "1234.56"}
    7. ],
    8. "meta": {"resolution_mode": "high", "max_tiles": 6}
    9. }

    After fine-tuning, the self-built evaluation set must be rerun. Teams should also check regression items: has general question answering degraded? Can structured outputs still pass JSON Schema validation?

    VIII. Three Typical Deployment Scenarios

    • Document and receipt understanding: This scenario requires high dynamic resolution and layout awareness. The core metrics are field-level accuracy and structural completeness, not overall similarity. For fields such as amounts, dates, and order numbers, the model output should be treated as a candidate value and verified with regex or checksum validation.

    • Long-video content moderation and summarization: A hybrid strategy of uniform sampling and keyframe extraction is usually effective. Video-side position encoding that supports temporal modeling is critical. The output should first produce shot-level summaries and then aggregate them, avoiding the problem of putting too many frames into the context at once and diluting attention.

    • Industrial quality inspection and multi-image comparison: Fix the shooting station and lighting conditions, and provide multiple images as an input sequence under the same prompt. The evaluation should focus on the asymmetric cost of false positives and false negatives. In many cases, threshold tuning is more practical than replacing the model.

    IX. Common Pitfalls

    1. Resolution inflation: Increasing the number of tiles without control can make latency unpredictable. Resolution should be derived from the minimum readable font size required by the target fields.

    2. Text capability degradation: Training only on visual data can weaken reasoning and format-following abilities. Text data should be mixed into training.

    3. Benchmark contamination and overfitting: A model may improve on public benchmarks without improving real business results. Self-built evaluation sets are essential.

    4. Ignoring shared context: Visual tokens and text tokens share the same context budget. Multi-image scenarios can easily exceed the limit.

    5. High-density character layout assumptions: High-density character layouts, especially Chinese documents, often require higher resolution than strategies validated on English-only corpora. Directly applying the same resolution policy may not be sufficient.

    X. Engineering and Platform-Level Recommendations

    To make multimodal capabilities useful in production, teams usually need to build four capabilities beyond the model itself:

    • A unified model access and routing layer that sends requests to different model sizes based on task complexity.
    • An observability system that records input image size, tile count, token count, and latency percentiles.
    • A traceable data loop that automatically routes online hard cases back into the fine-tuning set.
    • Cost governance based on visual token metering, including quotas and budget alerts.

    If Baidu AI Cloud is used, teams can use the model service platform for model onboarding and elastic compute scheduling, use object storage to hold images and visual feature caches, use message queues to decouple visual encoding and decoding services, and use logging and monitoring to track latency, failure rates, and cost.

    Overall, selection for open-source multimodal large models has moved from “checking leaderboards” to “checking paradigm fit.” Requirements for visual fidelity determine the modality connector and resolution strategy. Requirements for long temporal understanding determine position encoding and context budget. Requirements for factuality determine the strength of preference optimization and output validation. By understanding these relationships and combining them with self-built evaluation and a cost model, teams can find a sustainable balance among quality, latency, and cost.