Lightweight Visual Encoders for Multimodal Models: Design and Inference Optimization

    This article explains how lightweight visual encoders in multimodal models convert images into token sequences that a language model can process. It covers the conversion pipeline, a typical shallow vision transformer backbone with 2×2 spatial merging, preprocessing for arbitrary input resolutions, dynamic visual token budgets, two-dimensional position encoding, a simplified forward pass, and the trade-offs between accuracy, context length, memory use, and latency.

    In multimodal models, vision is more than reading image content. What matters most is that the model can actively use visual information when a task requires it, and feed visual feedback into later reasoning, generation, and tool calls. For tasks such as code generation, interface understanding, chart analysis, and video question answering, the visual encoder has two jobs. First, it must preserve as much useful information from the original image as possible. Second, it must compress the image into a token sequence that a language model can process efficiently.

    The core value of a lightweight visual encoder lies in mapping from pixel space to semantic space with fewer parameters and lower compute cost. Compared with large language models, the vision backbone usually accounts for only a small share of total parameters. However, it directly determines whether the model can accurately understand resolution, layout, text, graphics, and fine local details.

    1. How Visual Information Is Converted into Model Tokens

    Raw images consist of continuous pixels, while language models typically receive ordered sequences of discrete vectors. The visual encoding process can generally be abstracted into the following steps:

    1. Normalize the size of the input image or video frame.
    2. Split the image into regular local patches.
    3. Apply a linear projection or convolution to each patch.
    4. Extract spatial and semantic features with a visual attention network.
    5. Merge adjacent patches to reduce the number of visual tokens.
    6. Project the output dimension to the hidden dimension of the language model.
    7. Insert the visual tokens at specific positions in the text sequence for unified reasoning.

    The number of visual tokens is usually related to image area. If each token covers a larger image region, the sequence gets shorter, but detail loss may increase. If tokens are too fine-grained, recognition of text and small objects improves, but attention compute and context usage rise significantly.

    Therefore, the visual encoder should not simply pursue higher resolution at any cost. It must balance recognition accuracy, context length, memory consumption, and inference latency.

    2. A Typical Lightweight Vision Backbone

    A common design uses a relatively shallow vision transformer as the image encoding backbone. Its structure can be summarized as follows:

    • About 20 or more encoder layers
    • Hidden size on the order of 1,000
    • About 16 multi-head attention heads
    • Head dimension equal to the hidden size divided by the number of heads
    • Base patch size of 14×14 pixels
    • Adjacent 2×2 patches can be merged further
    • Output features projected to a language model hidden space of about 4,000 dimensions
    • Vision backbone size on the order of hundreds of millions of parameters

    The 2×2 merge is critical here. If the base patch side is 14 pixels, four neighboring base patches cover a 28×28 pixel region. After merging, each visual token roughly corresponds to one 28×28 image region. This preserves some spatial structure while reducing the visual sequence length to one quarter of its original size.

    This structure has a clear compute path, mature implementations, and compatibility with images of different aspect ratios. Its weakness is also clear. When an input contains small text, dense tables, or fine-grained interface elements, excessive compression can lose local information. For this reason, the number of visual tokens should not be fixed to a single value. It should be adjusted dynamically according to image content and task type.

    3. Preprocessing Strategies for Arbitrary-Resolution Inputs

    In practice, input images often have inconsistent widths and heights, and video frames may also vary in size. To let the vision backbone process these inputs stably, preprocessing typically includes the following steps.

    3.1 Resizing and Padding

    First, scale the image according to the target token range. Scaling should preserve the original aspect ratio as much as possible to avoid noticeable distortion of people, graphics, and text.

    After scaling, the width and height usually need to be padded up to an integer multiple of a fixed stride. For a structure with a 14-pixel base patch followed by 2×2 merging, the stride is typically 28. If the image dimensions are not multiples of 28, zero padding is applied to the right or bottom so that subsequent patch splitting aligns completely.

    Padding does not change the main content of the original image, but it introduces invalid regions. In practice, grid size information must mark the valid regions so the model does not mistake zero-padded areas for real visual content.

    3.2 Patch Splitting

    The padded image is divided into multiple 14×14 base patches. After passing through the embedding layer, each patch becomes an initial visual feature.

    In engineering implementations, patch splitting can be done with a convolution. The kernel size and stride are both set to the base patch size, so each output position corresponds to one patch. Compared with explicit loop-based splitting, the convolution approach makes better use of hardware parallelism and connects more easily with subsequent network layers.

    3.3 Spatial Merging

    The number of initial visual features is usually large. The model therefore concatenates or aggregates each group of four adjacent features, then maps the result back to the target hidden dimension with a linear layer. Merging reduces sequence length and gives each visual token a larger local receptive field.

    Spatial merging is not the same as simple average pooling. To preserve information from different sub-regions, common implementations concatenate the four features first, then perform fusion through normalization and a projection layer. This reduces the token count while retaining some directional, edge, and local layout information.

    4. Dynamic Control of the Visual Token Budget

    The number of visual tokens is an important resource in multimodal inference. Typically, a minimum and maximum number of tokens per image are set, for example a minimum of roughly 10 to 20 tokens and a maximum of several thousand tokens. The actual range should be determined by the model’s context length, memory capacity, and business requirements.

    Token budget control serves three main purposes.

    First, it limits context expansion from very large images. Raw screenshots, long web pages, or high-resolution scans can consume a large amount of sequence space if all details are kept, leaving less room for text input and model output.

    Second, it improves batch inference stability. In multi-image or video tasks, if the visual token count varies too much between samples, batching introduces substantial padding overhead. A reasonable upper limit helps control peak memory usage.

    Third, it selects a precision level based on the task. General scene classification can use fewer tokens, while table recognition, code screenshot analysis, and document understanding require higher visual resolution.

    A general strategy is to first estimate the initial token count from image area, then clip it to lie between the minimum and maximum values. For images with extreme aspect ratios, the longest side should also be limited to avoid generating an overly long one-directional visual sequence.

    5. Visual Attention and Position Encoding

    Visual encoders must handle both spatial position and sequence organization. Unlike one-dimensional text positions, images have two-dimensional coordinates by nature, and video adds a time dimension. Visual position encoding therefore often uses height, width, and time information.

    For images, each visual feature can be assigned two-dimensional coordinates. For video, the frame index can serve as an additional temporal coordinate. This allows the model to distinguish:

    • Horizontal and vertical positions within the same frame
    • Temporal order between frames
    • Spatial correspondence before and after merging
    • Boundaries and relative order among multiple images

    In implementation, position indices are usually generated from the grid size of each image or video, and the attention sequence length is computed from the valid regions. Because image sizes differ, batching cannot simply use a uniform length. Instead, cumulative sequence lengths must be maintained so that sample boundaries are correctly defined in variable-length attention computation.

    6. A Typical Forward Pass

    The forward pass of the vision module can be abstracted as the following pseudocode:

    1. def forward(pixel_values, grid_shape):
    2. position_ids = build_position_ids(
    3. grid_shape=grid_shape,
    4. merge_factor=2
    5. )
    6. sequence_info = build_sequence_metadata(
    7. grid_shape=grid_shape
    8. )
    9. hidden = patch_embedding(pixel_values)
    10. hidden = add_visual_position_information(
    11. hidden,
    12. position_ids
    13. )
    14. hidden = visual_transformer(
    15. hidden,
    16. sequence_info=sequence_info
    17. )
    18. hidden = spatial_merge(
    19. hidden,
    20. merge_factor=2
    21. )
    22. hidden = projection_to_language_space(hidden)
    23. return hidden

    The actual network may also include normalization layers, residual connections, rotary position embeddings, gated feed-forward layers, and cross-modal projection layers. The focus of this flow is not the specific interfaces. It shows how visual input is gradually transformed into a continuous vector sequence that the language model can consume.

    7. Inference Precision and Engineering