Vision Encoder Design in Lightweight Flagship Multimodal Models: Patch Merging, Token Budgets, and Inference Efficiency

    This article explains how lightweight vision encoders support multimodal models by converting images and video frames into compact visual token sequences. It covers input normalization, patch-based processing, 2×2 spatial merging, token budget control, a compact ViT-like backbone, inference flow, deployment benefits, and practical challenges such as high-resolution detail preservation, video frame selection, multi-image context management, and precision choices.

    1. Why Lightweight Vision Encoders Are a Key Module in Multimodal Models

    In practical multimodal model applications, the system receives more than text input. It must also process images, screenshots, tables, flowcharts, scanned documents, and short video clips. These are all forms of unstructured visual information.

    A traditional approach is to send the image to a separate vision model first, then pass the resulting text description to the language model. This method is straightforward, but it can lose fine-grained details and adds transfer cost between modules.

    A more modern approach is to integrate visual understanding directly into a unified model. In this case, the vision encoder is no longer just an “image-viewing tool.” It becomes part of the entire reasoning pipeline. It needs to complete three tasks:

    1. Convert images or video frames of arbitrary resolution into sequence representations that the model can process.
    2. Preserve as much task-relevant visual information as possible under a limited context budget.
    3. Provide high-quality visual feedback for downstream tasks such as text reasoning, code generation, UI understanding, visual question answering, or action planning.

    Therefore, the importance of a lightweight vision encoder is not only about having fewer parameters. It is about balancing performance, memory usage, latency, context occupancy, and task accuracy.

    2. Visual Input Normalization: From Arbitrary Sizes to Model-Readable Visual Sequences

    Multimodal models usually receive images with different sizes and aspect ratios. They may also receive multi-frame video input. To make the vision encoder process inputs stably, the input stage generally performs normalization, including resizing, cropping, or zero padding, so that the image dimensions satisfy the model’s internal patching rules.

    One common lightweight design requires the input image to align to a fixed multiple in both height and width. For example, if the base image patch size is 14×14 and a 2×2 spatial merge is performed afterward, each final visual token corresponds to a 28×28 region. As a result, image dimensions are usually padded to multiples of 28.

    The key point is: why 28?

    • 14 is the base patch edge length.
    • 2 is the merge factor.
    • 14 × 2 = 28.

    This design ensures:

    • The image is split evenly without residual boundaries.
    • Tokens after 2×2 merging maintain a clear correspondence with original spatial positions.
    • Subsequent positional encoding, visual sequence concatenation, and text alignment are more stable.

    This process can be understood as follows:

    1. Input image
    2. -> Resize
    3. -> Zero-pad to a multiple of 28
    4. -> Split into 14×14 base patches
    5. -> Merge each 2×2 base patches into one visual token
    6. -> Output visual token sequence

    For video input, frames are usually sampled first and then processed similarly on a per-frame basis. A video can be understood as multiple temporal image segments, and its visual token count is significantly higher than that of a single image. Therefore, the vision encoder must work with a context budget mechanism to avoid the visual sequence taking up too much space needed for text reasoning.

    3. Patch Splitting and 2×2 Merging: Balancing Detail Preservation and Token Count

    Vision encoders typically divide an image into local regions called patches. Smaller patches preserve finer spatial details, but they produce a large number of visual tokens. Larger patches reduce sequence length, but they may lose important information such as small text, thin lines, or icons.

    Lightweight vision encoders often adopt a combined strategy: “base patch + spatial merge.”

    For example:

    • Base patch size: 14×14
    • Merge method: 2×2
    • Region corresponding to one visual token: 28×28

    The advantage of this strategy is that it balances detail and efficiency. The base patch is small enough for convolution or embedding layers to capture local textures. After merging, the number of tokens is reduced to one quarter of the original, significantly lowering the computational pressure on subsequent Transformer layers.

    For example, suppose an original image is divided into a 28×28 base patch grid. After 2×2 merging, each visual token corresponds to a 28×28 pixel region. If the image size is 1120×1120, then before merging there are approximately:

    1. (1120 / 14) × (1120 / 14) = 80 × 80 = 6400 base patches

    After merging, there are approximately:

    1. (80 / 2) × (80 / 2) = 40 × 40 = 1600 visual tokens

    If no merging is performed, 1600 becomes 6400, which greatly increases context pressure. For multi-image conversations, document understanding, or interface screenshot analysis, controlling the number of tokens directly affects response speed, memory usage, and long-context capability.

    4. Token Budget Control: Adapting the Vision Encoder to Different Tasks

    In real applications, different tasks have very different requirements for visual detail. For example:

    • Understanding the overall layout of a screenshot may not require preserving every pixel-level detail.
    • Recognizing small-font tables in documents may require higher resolution.
    • Tasks such as code generation, UI replication, or image question answering may require zooming into local regions.
    • Video understanding tasks require controlling both frame count and total token count.

    For this reason, lightweight vision encoders usually introduce token budget control mechanisms, such as setting minimum and maximum visual token counts. A common budget range may be 16 to 8000 tokens per image, depending on the model design and application scenario.

    The core logic of token budget control can be summarized as follows:

    1. If the image is small, generate visual tokens directly according to the base rules.
    2. If the image is large, scale or chunk it according to the maximum token limit.
    3. If the task allows low-resolution understanding, reduce the number of tokens.
    4. If the task requires high-precision recognition, increase the number of tokens appropriately.
    5. For multi-image input, allocate a reasonable budget for each image to avoid exceeding the total context limit.

    Pseudo-code example:

    1. def allocate_visual_tokens(image, min_tokens, max_tokens, task_mode):
    2. base_tokens = estimate_tokens_from_resolution(image)
    3. tokens = base_tokens
    4. if task_mode == "high_detail":
    5. tokens = min(base_tokens * 2, max_tokens)
    6. elif task_mode == "fast_overview":
    7. tokens = max(base_tokens // 2, min_tokens)
    8. return tokens

    This mechanism makes visual encoding no longer a fixed process. Instead, it can be dynamically scheduled according to task goals. For interactive applications that require fast responses, a lower token budget can be used first. For offline tasks that require precise parsing, the budget can be increased.

    5. Lightweight Backbone: Visual Feature Extraction with Small Parameter Count

    Vision encoders usually adopt Transformer or ViT-like structures. A typical lightweight configuration may include the following parameters:

    1. Depth layers: 24
    2. Hidden dimension: 1024
    3. Attention heads: 16
    4. Head dimension: 64
    5. Base patch size: 14
    6. Spatial merge: 2×2
    7. Output dimension: 4096
    8. Backbone parameter count: ~0.56B
    9. Inference precision: usually BF16

    This configuration reflects several engineering tradeoffs.

    5.1 24 Layers: Balancing Expressiveness and Computational Cost

    If the vision encoder has too few layers, it may struggle to extract complex spatial relationships and semantic structures. If it has too many layers, latency and memory usage will increase significantly. A 24-layer configuration is a medium-depth lightweight design. It is suitable for converting images into visual tokens while keeping inference cost relatively low.

    5.2 1024 Hidden Dimension: Balancing Resolution and Stability

    The hidden dimension determines the representation width of each patch or merged token. A 1024-dimensional hidden size is a moderate scale in vision encoding. It can carry local texture, edge, color, and layout information without causing excessive memory pressure.

    5.3 16 Attention Heads with Head Dimension 64

    With 16 heads and 64 dimensions per head, the total dimension is 1024. This configuration helps capture local patterns, global context, and spatial relationships across different representation subspaces. Compared with more complex GQA-style attention configurations, a simpler attention setup is easier to deploy and debug, which makes it suitable for lightweight vision encoders.

    5.4 Output Dimension 4096: Aligning with the Language Model Backbone

    The output of a vision encoder is usually not directly generated as text. Instead, it is projected into the hidden space of the language model backbone. If the language model backbone has a hidden dimension of 4096, projecting the vision encoder output to 4096 dimensions helps with later fusion.

    5.5 ~0.56B Parameters: Lightweight but Still Expressive

    For a complete multimodal model, if the vision encoder accounts for only a small portion of the entire model, overall complexity can be reduced significantly. A vision backbone with about 0.56B parameters usually provides stable basic capabilities for tasks such as general image understanding, screenshot understanding, OCR support, and visual feedback.

    6. Breaking Down the Vision Encoder Inference Pipeline

    During inference, the core pipeline of a vision encoder generally includes: input preprocessing, patch embedding, positional encoding, attention computation, spatial merging, and cross-modal projection.

    It can be represented with pseudo-code as follows:

    1. def vision_forward(visual_inputs, grid_thw):
    2. # 1. Generate position information from image or video grids
    3. position_ids = build_position_ids(grid_thw, merge_size=2)
    4. # 2. Build attention metadata for variable-length inputs
    5. attention_meta = build_attention_meta(grid_thw)
    6. # 3. Convert input images into patch embeddings
    7. patch_embeddings = patch_embedding(visual_inputs)
    8. # 4. Add positional embeddings to preserve spatial structure
    9. embeddings = add_position_embedding(patch_embeddings, position_ids)
    10. # 5. Extract features using the vision encoder backbone
    11. encoded = vision_encoder(
    12. embeddings,
    13. attention_meta=attention_meta
    14. )
    15. # 6. Merge visual tokens and project them to the language model space
    16. visual_tokens = project_to_llm_space(encoded)
    17. return visual_tokens

    There are several important engineering points in this pipeline.

    6.1 Positional Information Cannot Be Ignored

    After an image is patched, its 2D spatial relationships need to be reinjected. Positional encoding helps the model understand the up-down and left-right relationships of tokens in the original image. Spatial position is especially important for tasks involving screenshots, forms, code interfaces, or document layouts.

    6.2 Variable-Length Attention Is Critical

    Different image sizes produce visual sequences of different lengths. If a fixed length is always used, computation may be wasted, or important information may be truncated. In practical inference, attention length information is usually built from the image grid so that the encoder can efficiently handle inputs with different resolutions.

    6.3 Patch Embedding Is Usually Implemented with Convolution

    A common approach is to use a small convolutional layer to map each 14×14 pixel patch to a hidden vector. The convolutional layer naturally preserves local spatial structure and reduces the input dimension.

    6.4 Spatial Merging Is Performed Before Output

    The merge operation reduces the number of tokens and makes the sequence easier for the downstream language model to process. Merged visual tokens usually represent larger spatial regions, which are suitable for cross-modal semantic alignment.

    7. Main Advantages of Lightweight Vision Encoders

    Based on the above design, the value of a lightweight vision encoder is not only reflected in parameter count. It comes from coordinated optimization across multiple engineering dimensions.

    7.1 Reducing Context Occupancy

    Visual inputs are often “heavier” than text inputs. A high-resolution image may correspond to thousands of visual tokens. If the encoder is too large, token generation can also become excessive, causing the context to expand rapidly. With 2×2 merging and budget control, the visual sequence length can be significantly reduced.

    7.2 Improving Inference Efficiency

    A smaller vision encoder means lower encoding latency per image. In interactive applications, users can see the first-round result faster. For multi-turn conversations, visual feedback loops, or agent tasks, low-latency visual encoding is especially important.

    7.3 Reducing Deployment Costs

    A smaller vision backbone is easier to run in environments with limited memory. It is also easier to deploy in cloud or edge environments. For offline tasks that require batch processing of images or videos, a lightweight encoder can significantly improve throughput.

    7.4 Easier Alignment with the Language Model

    The ultimate goal of the vision encoder is to generate high-quality visual tokens, not to independently complete all visual understanding. As long as the output representation can be effectively used by the language model backbone, the vision encoder can remain relatively lightweight. This leaves more parameter budget for the language model itself, which is used for reasoning, planning, and generation.

    7.5 Better Support for Multi-Turn Visual Interaction

    In multi-turn tasks, the model may need to observe images or screenshots repeatedly. For example:

    • The user uploads a UI screenshot, and the model generates code.
    • The model runs the code and observes the resulting screenshot.
    • The model modifies the code based on issues found in the screenshot.
    • The model observes the corrected interface again.

    This kind of “visual feedback” loop requires visual encoding to be fast, stable, and repeatable. A lightweight encoder reduces the cost of each observation and makes the overall task loop smoother.

    8. Common Challenges in Performance Optimization and How to Address Them

    Although lightweight vision encoders have clear advantages, several challenges still need to be addressed in practical use.

    8.1 How to Preserve Details in High-Resolution Images

    Directly reducing resolution can lose small text, thin lines, and complex icons. Possible strategies include:

    • Encoding large images in chunks.
    • Zooming into key regions locally.
    • Dynamically increasing the token budget based on task type.
    • Introducing visual question answering or region-focus mechanisms.

    8.2 How to Control Frame Count for Video Input

    Video understanding cannot simply send all frames to the model. Common practices include:

    • Sampling frames uniformly over time.
    • Reducing the frame rate for static scenes.
    • Prioritizing frames with key changes.
    • Using summary-style encoding for video clips.

    8.3 How to Allocate Context for Multi-Image Input

    If a single request contains multiple images, a reasonable token limit needs to be set for each image. Otherwise, one image may exhaust the context. In engineering practice, a total budget pool is usually used, for example:

    1. Total context budget = text reserve + total visual budget
    2. Total visual budget = number of images × per-image budget + video frame budget

    When the budget is insufficient, the system should automatically reduce resolution or skip low-priority inputs.

    8.4 Whether the Vision Encoder Output Is Stable

    The quality of visual tokens directly affects downstream reasoning in the language model. If positional encoding, spatial merging, or the projection layer is poorly designed, the model may misinterpret spatial relationships. Therefore, visual encoding quality needs to be validated through evaluation sets, manual spot checks, and task-level metrics.

    9. Engineering Recommendations

    If an organization wants to build similar multimodal capabilities, it can start from the following aspects.

    9.1 First Build General Visual Understanding, Then Strengthen Specialized Tasks

    A lightweight vision encoder is usually suitable for first achieving general image understanding, then optimizing for specific scenarios. For example, document parsing, screenshot understanding, product recognition, and form recognition can all be improved on top of general capabilities through task-specific fine-tuning or prompt design.

    9.2 Treat Token Budget as a First-Class Engineering Metric

    The number of visual tokens should be included in system performance evaluation. In addition to accuracy, it is also important to monitor:

    • Average tokens per image.
    • Total tokens for multi-image tasks.
    • First-response latency.
    • Vision encoding time per conversation turn.
    • Long-context truncation rate.

    9.3 Build a Tiered Visual Input Strategy

    Inputs can be classified according to task importance, for example:

    • Images that require high-precision processing.
    • Images that only need overview understanding.
    • Optional reference images.
    • Cached or repeatedly submitted images.

    This can improve the overall user experience under limited compute resources.

    9.4 Use Containerization, Logging, and Monitoring to Ensure Stability

    After a multimodal service goes live, the stability of the vision encoder is often closely related to resource fluctuations. It is recommended to deploy inference services through container platforms and combine them with logging services, monitoring alerts, and trace tracking. This helps observe image size distribution, encoding time, memory usage, and the proportion of abnormal inputs.

    9.5 Choose Inference Precision Carefully

    Some lightweight vision models may use BF16 during inference and avoid lower-precision quantization. The reason is that vision encoders can be sensitive to numerical stability, and premature quantization may cause detail loss. In actual deployment, testing should be conducted based on task accuracy requirements, hardware support, and performance goals.

    10. Summary

    A lightweight vision encoder is not simply about making a vision model “smaller.” It is a systematic optimization process around the practical needs of multimodal models, covering image normalization, patch splitting, spatial merging, token budgets, backbone structure, positional encoding, and inference efficiency.

    Its core value can be summarized in four points:

    1. Convert images into visual tokens at a relatively low parameter cost.
    2. Reduce context occupancy through 2×2 merging and budget control.
    3. Support complex interaction scenarios such as multi-image input, video input, and visual feedback.
    4. Work with the language model backbone to improve overall multimodal reasoning efficiency.

    For developers, understanding the design logic of vision encoders helps build more stable multimodal applications. Especially in scenarios such as code generation, UI understanding, document parsing, and visual agents, an efficient, lightweight, and manageable vision encoding module may determine the final user experience more than simply increasing model parameters.