Visual Encoders for Vision-Language Multimodal Models: Evolution and How to Choose One

    This article looks at how visual encoders in vision-language multimodal models have evolved and what to weigh when selecting one. It starts from the common architecture (an LLM core, a visual encoder, and a projector or adapter), explains why ViT became the default visual backbone, and traces the shift in pretrained alignment objectives from global contrastive learning to pairwise sigmoid classification, including the engineering and quality trade-offs behind it. It also covers native multi-resolution support, why it matters for documents and OCR, and what it requires beyond the model itself, before closing with five practical selection criteria for choosing or evaluating a visual encoder.

    The Standard Architecture of Vision-Language Multimodal Models

    Almost every mainstream multimodal large language model (MLLM) follows the same basic pattern. A pretrained large language model (LLM) sits at the center, a dedicated visual encoder is attached to it, and a projector or adapter maps the visual features into the language model’s semantic space.

    In this setup, the visual encoder handles visual perception. It takes images as input and converts them into high-dimensional semantic features. Its capabilities therefore set the ceiling for how well the whole model can understand the visual world. If the vision side cannot see or interpret clearly, even a powerful language model is left reasoning over impoverished visual features.

    Why ViT Became the Default Visual Backbone

    Early visual feature extraction relied mainly on convolutional networks. In multimodal settings, however, the Vision Transformer (ViT) has become the de facto standard, for three main reasons:

    • Global modeling. Through self-attention, ViT can model long-range dependencies between any regions of an image, even in its early layers. This is critical for understanding spatial relationships and fine-grained local features.
    • A natural fit with language models. ViT outputs a sequence of tokens, which is structurally the same as the input format of a language model. This makes it easy to align the two modalities with a simple linear projection or an MLP.
    • Scalability. ViT has a regular, uniform structure that is easy to scale as model size and training data grow. This is in line with scaling-law trends.

    The names on the vision towers of multimodal models have varied from one generation to the next. Some are natively designed ViT variants, while others are structural evolutions of the standard ViT. All of them, however, are continuations and refinements of the ViT architecture.

    How Pretrained Weights Evolved: From Contrastive Learning to Sigmoid Alignment

    Visual encoders may look similar in architecture, but what really separates them is their pretrained weights and training objectives. These have gone through a clear paradigm shift.

    The Earlier Generation: Contrastive Image-Text Alignment

    For a long time, most multimodal models chose visual encoders pretrained with contrastive learning. These encoders typically shared a familiar specification: large-scale parameters, a 14-pixel patch size, a fixed input resolution of around 336×336, 24 transformer blocks, and a hidden dimension of 1024.

    Their popularity came from the training objective: a typical contrastive loss based on InfoNCE-style softmax normalization. For a batch of size N, the objective builds an N×N similarity matrix. Image-text pairs on the diagonal are positives, and all other pairs are negatives. Softmax normalization then maximizes the similarity of the positive pairs.

    Simplified pseudocode:

    1. # Contrastive loss, informally
    2. logits = image_feat @ text_feat.T * temperature # N x N similarity matrix
    3. labels = torch.arange(N) # diagonal pairs are positives
    4. loss = cross_entropy(logits, labels) # softmax normalized along rows

    This paradigm is simple and effective. It gives a visual encoder the ability to align with the text modality from the start. It also has an inherent drawback: it needs a global normalization over all samples in a batch. With small batches, there are too few negatives to align representations well. When training is scaled up to distributed setups, the resulting global communication across devices adds memory and bandwidth overhead.

    Today: Pairwise Sigmoid Independent Binary Classification

    In recent years, a growing number of multimodal models have adopted image-text alignment based on sigmoid loss. Its main difference from classic contrastive learning lies in the loss function:

    1. L = -log σ(t · x_i · y_i) - Σ_{j≠i} log(1 - σ(t · x_i · y_j))

    Instead of a global normalization within the batch, each image-text pair is treated as an independent binary classification problem: do this image and this text match or not?

    This looks like a simple change in mathematical form. In practice, it brings benefits on both the engineering and quality fronts:

    • Decoupled samples. Each image-text pair computes its loss independently, without relying on a global softmax, so all the computation can happen on a single device. This greatly reduces the communication and memory pressure of distributed training, making it well suited to large-scale training.
    • Less “negative pollution.” Under contrastive learning, similar samples in the same batch are wrongly treated as negatives, which introduces gradient noise. The sigmoid paradigm turns the task into independent classification of positive and negative pairs, so it naturally avoids this problem.
    • Better quality. At the same data scale, the sigmoid approach often trains higher-quality visual encoders. The difference shows up clearly in fine-grained understanding on downstream multimodal tasks.

    The Commonly Overlooked Capability: Multi-Resolution Support

    Beyond the evolution of training objectives, native multi-resolution support is another important advance in visual encoders.

    Early visual encoders almost all used a fixed input resolution, such as 336×336 or 384×384. Every image therefore had to be forced to the same size through resizing or cropping. For documents, charts, UI screenshots, and long images, which are full of small text and high-frequency details, a fixed low resolution causes severe information loss. The text blurs together, and the model cannot read it.

    Newer visual encoding approaches add native multi-resolution capability. The encoder can either process variable-size inputs directly or preserve detail through dynamic tiling, which splits a high-resolution image into several sub-tiles to encode separately and then combines them. This brings direct advantages:

    • Detail preservation. High-resolution input lets the model read small text in documents, as well as table structures and chart labels.
    • Compute that adapts to the input. Simple images can be processed quickly at low resolution, while complex ones are handled carefully at high resolution. This balances accuracy against cost.
    • Stronger document and OCR performance. This capability is key to whether a multimodal model can actually “read” a full page of a document.

    Multi-resolution support is not purely a matter of model architecture. It also requires enough high-resolution samples in the training data, together with appropriate designs for positional encoding and token-length management. Without these, performance can degrade.

    Practical Guidance for Choosing a Visual Encoder

    When choosing or evaluating a visual encoder for an MLLM, consider the following dimensions:

    1. Alignment paradigm. Prefer the newer generation of encoders trained with sigmoid loss, which generally outperforms early contrastive approaches on both scalability and alignment quality.
    2. Resolution capability. Be clear about your use case. If it involves documents, OCR, or chart understanding, choose an encoder that supports multi-resolution or dynamic resolution.
    3. Structural scalability. ViT remains the mainstream backbone. Check whether specifications such as the number of layers, hidden dimension, and patch size match the level of granularity your target task needs.
    4. Feature alignment quality. However good the visual encoder, it still needs a suitable projector to map visual tokens into the language space. The projector design (a single linear layer vs. a multi-layer MLP vs. cross-attention) also affects the final results.
    5. Training and inference cost. Higher resolution improves detail perception, but it also significantly increases the number of visual tokens. This lengthens the language model’s sequence and raises inference costs, so you need to trade off quality against cost.

    Summary

    The visual encoders in multimodal LLMs vary widely in naming and weights, but their underlying logic is highly consistent: a ViT backbone pretrained with image-text contrastive learning to produce features aligned with language. What really creates a generational gap is the evolution of the training paradigm, from global contrastive learning to pairwise sigmoid alignment, along with the introduction of multi-resolution support. Understanding this technical thread helps you focus on the key issues amid a crowded field of models and make more sound architecture decisions.