Lightweight Vision Encoders Explained: The Visual Backbone of Multimodal Models

    This article analyzes a typical 24-layer lightweight vision encoder used in a mainstream open-source inference framework, examining it across four dimensions: structural parameters, the preprocessing pipeline, token budget, and inference implementation. It explains why a "small but precise" vision tower is a rational design choice, how dynamic token budgets and 2×2 spatial merging keep cost under control, and how native integration lets a model decide "when to observe" and feed visual feedback into its reasoning. It also covers engineering practices, tuning recommendations, and common misconceptions for developers building multimodal applications.

    Multimodal large models are undergoing a shift in role. Early vision modules mainly answered “what is in this image”; today, more and more models need to answer “what should be done next.” A representative statement captures it well: visual programming is not just about processing images, but about extending the boundaries of what programming can reach. As a result, vision needs to be natively integrated into the model so that the model can decide for itself when to observe and use visual feedback to guide its subsequent actions.

    This means the vision encoder is no longer a one-off “pre-filter” but a perception component that can be invoked repeatedly within a single session. Take “visual programming” as an example: the model might first capture a screenshot to inspect the interface, then generate an operation script, run it, capture another screenshot to check the result, and correct accordingly. Each “observation” produces new visual tokens that enter the context and participate in reasoning.

    The vision encoder therefore faces a set of competing goals: it must be strong enough to provide reliable visual semantics, and light enough to stay low-cost and low-latency across multi-turn observation scenarios. This article takes a typical 24-layer vision encoder from a mainstream open-source inference framework as a sample and analyzes it across four layers: structural parameters, preprocessing pipeline, token budget, and inference implementation.

    One important trend on the vision side in recent years is “native resolution + visual scaling”: letting the model receive images at an economical number of tokens, then raising the resolution of local detail on demand. The dynamic token budget and 2×2 merging strategy in the sample below are a direct reflection of this idea.

    2. Positioning the Vision Encoder: Why “Small and Precise” Is the Rational Choice

    1. Breaking Down the Responsibilities

    The vision encoder’s work can be summarized in three steps:

    1. Semantic extraction: converting pixels into semantically meaningful feature vectors;
    2. Spatial compression: drastically reducing the number of tokens while preserving key spatial relationships;
    3. Dimension alignment: projecting visual features into the language model’s hidden space so they can be processed by the same attention mechanism.

    The common thread across these three tasks is that they are all “feature transformations,” not “complex reasoning.” The parts that genuinely require long chains of logic, code generation, and planning should be handled by the language backbone. Based on this division of labor, an all-encompassing vision tower is not necessary.

    2. Cost Structure Determines Priority

    In terms of parameter share, the vision backbone is about 0.56B, less than 1% of the entire model. Even when the overall model reaches the hundreds-of-billions parameter scale, this ratio remains small. In other words, the parameter scale of the vision side is not the bottleneck—the number of visual tokens is.

    The reason is straightforward: visual tokens enter the context and are computed together with all text tokens during attention in the prefill phase. The text side already has mechanisms such as grouped-query attention (GQA) to compress the key-value cache, while image tokens often number in the hundreds or thousands. So the first priority in optimizing the vision side is controlling the number of tokens, and the second priority is improving the expressiveness of the vision tower.

    3. What a “Good Enough” Vision Tower Looks Like

    The configuration of mainstream lightweight vision encoders usually falls within the following ranges:

    • Layers: 20–30;
    • Hidden dimension: 1024–1280;
    • Attention heads: around 16, 64 dims per head, no GQA;
    • Patch size: 14×14 pixels;
    • Spatial merge: 2×2.

    The structural parameters of the sample in this article are shown in the table below:

    Item Value
    Layers (depth) 24
    Hidden dimension 1024
    Attention heads 16 (64 dims each, no GQA)
    Patch size / merge 14 / 2×2
    Output dimension 4096 (aligned with the language backbone)
    Backbone parameters ~0.56B (less than 1% of the whole)
    Inference precision Fixed BF16, no FP8 quantization

    3. Structural Parameters, Item by Item

    1. 24 Layers, 1024 Hidden Dimension: A Middle-Ground Choice for Resolution-Sensitive Semantic Tasks

    The number of layers in a ViT directly determines the receptive field and the level of semantic abstraction. 24 layers is a configuration repeatedly validated in resolution-sensitive tasks such as document understanding, OCR, and GUI understanding: deep enough to complete hierarchical abstraction from texture to layout, yet not carrying the obvious inference overhead of 40+ layers.

    The hidden dimension of 1024, paired with 16 attention heads, gives a per-head dimension of 1024 / 16 = 64. This value matches the per-head dimension of mainstream language models, which brings the following benefits:

    • The attention implementations on the vision and language sides can reuse the same kernel;
    • A per-head dimension of 64 aligns well on GPUs, so matrix multiplication is efficient;
    • Parameter estimation is intuitive: the attention portion of each layer is about 4 × 1024² ≈ 4.2M parameters, and the feed-forward portion is about 8.4M at a 2× expansion, totaling about 12.6M. Across 24 layers, the backbone is about 0.3B; adding patch embedding, position encoding, and the output projection brings the total to the 0.56B range.

    2. Why the Vision Tower Does Not Use GQA

    The core value of GQA lies in compressing the key-value cache under long sequences. The vision tower’s input is typically the token sequence of one or several images after patching, usually a few hundred to a few thousand tokens in length, and it does not need to reuse the cache across multiple generations. Therefore:

    • The gains from GQA are minimal;
    • Standard multi-head attention (MHA) is simpler to implement and can directly reuse the optimized attention kernels.

    During inference, the vision tower runs a single forward pass, so the KV cache poses no memory pressure. This is also one of the reasons it opts for fixed BF16 rather than FP8.

    3. Output Dimension 4096 Aligned with the Language Backbone

    The last layer of the vision encoder projects the 1024-dim features to the language model’s hidden dimension (4096 in this case). This projection layer is usually a linear layer or a lightweight MLP with parameters in the tens of millions. It does more than upsample dimensions—it performs cross-modal space alignment: making visual features comparable to text embeddings within the same semantic space.

    In practice, this layer is often a focal point for fine-tuning multimodal capability. During pretraining, the vision tower can maintain a relatively high learning rate, while the alignment layer needs a more careful learning-rate schedule to avoid degrading text capability.

    4. Parameter Estimation and the Conclusion That “the Vision Tower Should Not Be Too Large”

    Plugging in the parameters above, the vision backbone is about 0.56B. If the overall model is at the hundreds-of-billions scale, the vision side accounts for less than 1%. The design philosophy conveyed by this number is: prioritize the compute budget for the language backbone and context length, and keep the vision tower “good enough and efficient.”

    This does not mean visual capability can be sacrificed. On the contrary, it shows that improving visual capability does not lie in “piling up vision tower parameters,” but in:

    • Better preprocessing strategies (dynamic resolution, token budget);
    • Better token merging methods;
    • Higher-quality joint training with the language model.

    4. Preprocessing Pipeline: From Pixels to Visual Tokens

    The input the vision encoder receives is not raw pixels, but a set of normalized patch sequences. The whole process breaks down into four steps.

    1. Resizing and Zero-Padding

    The image is resized to a target size and zero-padded to a multiple of 28. Why 28? Because a 2×2 merge follows, and 28 = 14 (patch size) × 2 (merge factor).

    Zero-padding rather than cropping avoids losing edge information. Padded regions are handled via masking or position encoding in the subsequent attention computation and do not contribute to valid token output.

    2. Patching

    The image is split into base patches of 14×14 pixels. For a 1024×768 image, this yields 1024/14 × 768/14 patches.

    3. 2×2 Merge: Reducing the Token Count to a Quarter

    Four adjacent base patches (2×2) are merged into a single token. This step is key to token compression:

    • Without merging, a 1024×768 image would produce 73 × 55 ≈ 4015 tokens;
    • After merging, the token count drops to about 37 × 28 ≈ 1036—exactly one quarter of the original.

    In long-context scenarios, this compression delivers considerable gains.

    4. Token Budget Control

    Each 28×28 pixel region corresponds to one visual token. The model controls the number of tokens a single image produces via min_image_tokens and max_image_tokens (default range of 16 to 8000), adapting to different context budgets:

    • When the budget is ample, resolution can be increased for finer OCR and layout information;
    • When the budget is tight, the image can be shrunk to keep the context from being overwhelmed in multi-image, multi-turn scenarios.

    This is a form of inference-time, controllable “visual scaling” that hands the trade-off between image quality and cost to the caller.

    5. Inference Implementation: Variable-Length Attention and Video Extension

    In an open-source inference framework, the implementation of this vision model follows a mature paradigm: from early 2D vision models, to document-oriented OCR variants, to the current general-purpose version supporting image-text and video, the overall structure maintains good continuity.

    The core forward logic can be summarized as the illustrative code below:

    1. # Illustrative pseudocode: vision tower forward pass
    2. def forward(self, hidden_states, grid_thw, **kwargs):
    3. # 1) Compute 2D/3D position encoding and variable-length attention segmentation
    4. position_ids = get_vision_position_ids(grid_thw, self.spatial_merge_size, ...)
    5. cu_seqlens, max_seqlen = get_vision_attention_seqlens(grid_thw, self.config, ...)
    6. # 2) Pixel patches → visual token embeddings
    7. hidden_states = self.patch_embed(hidden_states) # ① Conv3d patchify
    8. # 3) Broadcast position encoding + residual add
    9. position_embeddings = self.rot_pos_emb(position_ids)
    10. hidden_states = hidden_states + position_embeddings
    11. # 4) Layer-by-layer Transformer encoding (varlen attention)
    12. for layer in self.layers:
    13. hidden_states = layer(hidden_states, cu_seqlens=cu_seqlens, max_seqlen=max_seqlen)
    14. # 5) 2×2 spatial merge + projection to language model hidden dimension
    15. hidden_states = self.merger(hidden_states)
    16. return hidden_states

    A few key designs are worth noting:

    • Conv3d patching: A 3D convolution is used instead of 2D to cover the time dimension of video. For static images, the time dimension degenerates to 1, making it equivalent to a 2D convolution in implementation.
    • Variable-length attention: grid_thw records the patch grid dimensions (t, h, w) for each image or video segment. Within the same batch, different images have different resolutions and thus different sequence lengths. Using cu_seqlens (cumulative sequence lengths) and max_seqlen, the attention kernel can perform grouped computation over a single concatenated long sequence, avoiding the compute waste of large amounts of padding.
    • Position encoding: The vision side typically uses 2D or 3D rotary position encoding, letting the model perceive the positional relationships of patches on a 2D plane and even in space-time.
    • Output alignment: merger completes the 2×2 merge and dimension upsampling, with an output dimension matching the language backbone’s hidden size so it can be injected directly as an embedding.

    6. Coupling Vision Encoding with the Generation Model: Native Integration and “When to Observe”

    The phrase “decide when to observe” in the official statement touches the core of multimodal system design.

    1. From “One-Off Encoding” to “Multi-Turn Observation”

    Traditional multimodal models encode images once at the start of inference and no longer access visual information during the subsequent generation process. Native integration means the vision encoder can be invoked multiple times during a conversation: after generating a piece of code, the model can proactively initiate an “observation,” re-encoding a run result or interface screenshot into the context.

    This capability places demands on training data: it requires large amounts of interleaved image-text, multi-turn samples so the model learns “when it is worth taking another look.” Otherwise, the model may either over-observe (wasting tokens) or never observe (missing feedback).

    2. Typical Application Scenarios

    • GUI agents: screenshot → analyze layout → generate action → screenshot again to verify;
    • Document and form understanding: get a rough view of the whole first, then zoom in on specific fields;
    • Code and error troubleshooting: combine error screenshots and log screenshots to locate the problem;
    • Industrial and quality inspection: multi-view observation + rule-based validation.

    The common thread across these scenarios is: vision is not part of the input, but one link in the decision loop.

    3. Requirements on the Inference Framework

    Supporting “when to observe” requires the framework to have:

    • Dynamic injection of visual tokens (appending images during generation);
    • Flexible composition of the vision encoder and the language backbone (separable deployment, reusable cache);
    • Support for variable-length processing of multi-image, multi-frame sequences.

    7. Engineering Practice and Tuning Recommendations

    1. Setting a Reasonable Token Budget

    max_image_tokens is not “the larger the better.” From experience:

    • Document OCR and chart-reading tasks: raise it appropriately to retain more detail;
    • Interface operation and multi-image comparison tasks: control it moderately to avoid context bloat;
    • Video understanding: estimate the total as frames × tokens per frame, and subsample frames when necessary.

    It is advisable to run A/B tests in production to observe the trade-off curve between task accuracy, context length, and latency.

    2. Precision Choice: The Rationale for Fixed BF16

    Since the vision tower accounts for less than 1% of parameters, running fixed BF16 without FP8 quantization at inference is a pragmatic trade-off:

    • Limited gains: the vision tower’s compute share is small, so throughput improvements from quantization are not significant;
    • Non-trivial risk: visual features are relatively sensitive to quantization error, which can cause perceptible accuracy drops in high-resolution detail tasks;
    • Simple implementation: no need to maintain a quantization calibration pipeline, reducing the risk of online/offline inconsistency.

    3. Preprocessing Is a Hidden CPU Bottleneck

    Operations such as image resizing, padding, and patching are usually done on the CPU and can easily become a bottleneck under high concurrency. Consider:

    • Decoupling preprocessing from inference and scaling them independently;
    • Using GPU-accelerated preprocessing operators;
    • Caching embeddings for frequently repeated images.

    4. Monitoring and Observability

    It is recommended to track the following metrics:

    • Distribution of visual token counts;
    • Share of vision tower forward time;
    • End-to-end time to first token;
    • Context length growth in multi-turn observation scenarios.

    When deploying in the cloud, you can use a container platform to host the inference service, object storage to hold input materials and results, and log and monitoring/alerting services to achieve end-to-end tracing. Baidu AI Cloud’s related products can serve as one of the platforms carrying such general capabilities.

    8. Common Misconceptions

    1. Assuming a bigger vision tower is always better. The less-than-1% parameter share shows the bottleneck is not vision tower scale, but token count and training quality.
    2. Blindly raising max_image_tokens. This directly inflates context length and inference cost, and must be weighed against the task type.
    3. Overlooking preprocessing overhead. Under high concurrency, preprocessing is often the first link to become a bottleneck.
    4. Underestimating the context cost of “observation”. Multi-turn observation continuously accumulates visual tokens and needs to be paired with compression or summarization strategies.

    9. Summary

    The design logic of a lightweight vision encoder can be captured in three sentences:

    1. Use the smallest vision tower to perform semantic compression, leaving compute for the language backbone;
    2. Use a dynamic token budget to hand control of image quality versus cost to the caller;
    3. Use native integration to make vision a link in the decision loop, rather than a one-off input.

    From the structural parameters of 24 layers and 0.56B parameters, to the preprocessing details of a 28-pixel grid and 2×2 merging, to the engineering implementation of variable-length attention and BF16 inference, these choices together form a “good enough, controllable, and extensible” visual backbone. For developers building multimodal applications, understanding these details is more practically valuable than merely comparing parameter counts.