Lightweight Multimodal Models: Anatomy of the ViT Visual Encoder

    This article examines the ViT visual encoder used in a lightweight multimodal model. It explains why a small-parameter visual backbone can still dominate latency and cost, then walks through the full pipeline: image preprocessing, 14-pixel patchification, 2×2 merging, visual token budgeting, variable-length attention, and fusion with the language model. It also covers deployment trade-offs such as resolution planning, embedding reuse, BF16 precision, memory estimation, batching, observability, and autoscaling. The content is intended for developers evaluating or deploying multimodal systems.

    1. Introduction: Vision is moving from an add-on to a native capability

    Early multimodal large language models commonly adopted the combination of “visual encoder + projection layer + language model”. In this architecture, images are first encoded by an independent visual backbone and then mapped into the language model’s input space through a lightweight projection layer. This design is modular and easy to reuse, but the visual module mainly serves the role of “understanding the image”.

    As model capabilities expand into more complex scenarios such as AI agents, GUI operation, and visual programming, the role of vision goes beyond simple image understanding:

    • Vision expands the boundaries of coding and operation. When code execution fails, information such as error screens, rendered output, and interface layout is often only available as images. The model must be able to “see” in order to locate and fix the problem accurately.
    • Vision needs to be natively integrated into the model. Native visual integration means that visual input participates in modeling from the pretraining stage alongside text, rather than being attached later as an independent visual component. Native integration provides two key capabilities:
      • The model can decide for itself when it needs to look and what it needs to observe, invoking visual input on demand during inference.
      • The model can use visual feedback to guide subsequent actions, for example, revising code based on rendering results or adjusting the next operation based on a screenshot.

    These models are often positioned as lightweight flagship models. They are expected to remain competitive among compact models in coding, reasoning, and multimodal capabilities, while also being as efficient as possible in terms of inference cost, response latency, and throughput. As the first stage of the multimodal request path, the visual pipeline’s structure directly affects the overall quality and efficiency of the system.

    This article uses the visual encoder of a lightweight multimodal flagship model as an example. It follows the full path of “preprocessing → patching and merging → visual encoding → vision-language fusion → inference and deployment”. The code in this article is pseudocode written for explanation. Function names and call order only express the logical relationship between key steps, and the specific implementation may vary by version.

    2. Visual encoder architecture: few parameters, potentially decisive

    2.1 Fewer parameters do not mean lower overhead

    The structural parameters of the visual encoder are as follows:

    Configuration Value Engineering meaning
    Network depth 24 layers Medium-scale backbone, balancing representation capability and inference latency
    Hidden size 1024 Vector width of each patch
    Number of attention heads 16 Standard multi-head self-attention
    Per-head dimension 64 (1024 / 16, no GQA) Consistent dimensions across heads
    Patch size / merge factor 14 / 2×2 Each visual token corresponds to a 28×28 pixel area
    Output dimension 4096 Aligned with the language model hidden dimension
    Backbone parameter count About 0.56B Less than 1% of the overall model parameters

    In terms of parameter count, the visual encoder is almost a supporting component. However, developers often have a misconception in deployment: they assume that because the encoder has few parameters, multimodal inference overhead must also be small. This is not necessarily true, for three reasons:

    1. Compute depends on input size, not parameter count. If the visual encoder processes images at dynamic resolution, high-resolution inputs are split into more patches, and self-attention computation grows quadratically with sequence length. In many multimodal requests, the visual part can be more time-consuming than text prefill.
    2. The small parameter share is relative to the language model. In multimodal models, the language model usually accounts for the majority of parameters. “Less than 1%” describes parameter distribution, not the same proportion of inference latency.
    3. Precision choices require system-level trade-offs. The visual encoder is fixed to run in BF16 and does not participate in FP8 quantization, which will be discussed later. This shows that the designers prioritize the precision stability of visual inputs.

    2.2 Design considerations behind the key parameters

    • 24 layers, hidden size 1024: a medium-scale backbone. This configuration is comparable to the larger common visual Transformer backbones in the industry. It provides enough representation capability for tasks such as document understanding and interface recognition, while avoiding the serial-layer and memory overhead of an overly deep backbone.
    • 16 heads, per-head dimension 64, no GQA: matched to the computation characteristics of visual attention.
      • Grouped-query attention allows multiple query heads to share a set of key-value heads, and its main benefit is compressing the KV cache during autoregressive decoding.
      • The visual encoder performs one forward pass over the entire image and does not generate tokens one by one, so there is no reusable KV cache to compress. The benefit of GQA is limited.
      • Keeping the standard multi-head structure is simpler to implement and makes it easier to analyze the roles of different attention heads.
    • 14-pixel patch and 2×2 merging: a balance between detail and efficiency.
      • The 14×14 patch is consistent with the common configuration of mainstream visual backbones, making weights and implementation experience in the ecosystem easier to reuse.
      • The 2×2 merge increases the effective coverage of each visual token to 28×28 pixels.
    • Output dimension 4096: naturally aligned with the language model. The 2×2 merge concatenates four adjacent patches, each with 1024 dimensions, into a 4096-dimensional vector, which matches the common hidden dimension of language models. The fusion stage can then use a simple projection layer.

    3. Image preprocessing: the first step from pixels to visual tokens

    The visual encoder does not directly receive raw images. Before being fed into the model, images are first standardized by a dedicated multimodal processor. This step determines the number and quality of visual tokens, yet it is often overlooked.

    3.1 Resizing and zero padding

    A key rule of the processor is that after resizing, images are zero-padded so that both width and height are multiples of 28. This comes from the structure of the visual encoder: the patch size is 14, followed by a 2×2 merge, so 14 × 2 = 28.

    • If the width or height is not a multiple of 28, edge pixels cannot form complete 2×2 groups, and additional boundary handling is required during patching and merging.
    • Padding to a multiple of 28 ensures that inputs of arbitrary size can be cleanly partitioned.
    • Padding pixels do not carry valid information and are usually distinguished from valid patches when computing attention segment boundaries.

    Implementations may vary, but two common strategies are:

    1. Preserve aspect ratio: first scale the image proportionally to the target range, then pad to a multiple of 28.
    2. Scale by the long side: use the longer side as the target size and pad the shorter side.

    In practice, it is important to confirm which strategy the processor uses, because it directly affects how much detail is preserved for long images.

    3.2 Patchification and 2×2 merging

    The image is split into 14×14 pixel patches, followed by a 2×2 merge:

    1. Patching: each 14×14 patch is mapped to a 1024-dimensional vector.
    2. Merging: four adjacent patches are combined into one unit. A common implementation first concatenates these vectors into 4096 dimensions, then applies an MLP projection, so each 28×28 pixel region corresponds to one visual token.

    The merge is essentially a non-overlapping “space-to-channel” operation, similar to space-to-depth rearrangement in deep learning. It brings two benefits:

    • The sequence length processed inside the visual encoder is reduced to one quarter of the original, significantly lowering self-attention cost.
    • The number of visual tokens sent to the language model is also reduced, leaving more of the long-context window for text.

    3.3 Visual token budget: min_image_tokens and max_image_tokens

    The number of visual tokens can be constrained by the minimum and maximum token counts per image. Common implementations name these parameters min_image_tokens and max_image_tokens, with a default range of approximately 16 to 8000. The system dynamically determines the visual token count based on image area and limits it to this range. This mechanism is valuable for several reasons:

    • Context control: visual tokens and text tokens share the same context window. Setting an upper limit prevents a single high-resolution image from consuming too much budget.
    • Adjustable cost: token count directly determines visual encoding compute and language model prefill length. Adjusting the upper limit by business scenario allows flexible trade-offs between precision and cost.
    • Unified handling of multi-source inputs: screenshots, photos, and document scans can have very different resolutions. After unified constraints, batching and capacity planning become more predictable.

    4. Parsing the inference flow of the visual encoder

    In inference frameworks, the core entry point of the visual encoder is usually a forward method. The following is illustrative pseudocode. Function names and ordering only explain the logic and do not represent the implementation of any specific framework:

    1. def forward(self, pixel_values, grid_meta):
    2. # grid_meta describes each sample's grid information:
    3. # temporal frame count t, height h, width w (all in original patch units)
    4. position_ids = build_vision_position_ids(grid_meta, self.spatial_merge_size)
    5. cu_seqlens, max_seqlen = build_varlen_meta(grid_meta, self.config)
    6. # 1. Patch embedding: map pixel blocks to visual token vectors
    7. hidden_states = self.patch_embed(pixel_values)
    8. # 2. Generate positional embeddings
    9. position_embeddings = self.rope(position_ids)
    10. # 3. Multi-layer Transformer encoding
    11. # (2D rotary position encoding and variable-length attention)
    12. for layer in self.layers:
    13. hidden_states = layer(
    14. hidden_states,
    15. position_embeddings=position_embeddings,
    16. cu_seqlens=cu_seqlens,
    17. max_seqlen=max_seqlen,
    18. )
    19. # 4. 2×2 merging + projection: align with language model hidden size
    20. return self.merger(hidden_states)

    Several engineering details around this main path are worth attention.

    4.1 Visual position IDs: unified representation for images and video

    Visual position IDs usually describe each sample’s visual input with a grid shaped like [t, h, w]: t is the temporal dimension (1 for an image, the number of frames for a video), and h and w are the height and width of the grid.

    • 2D rotary position encoding separately encodes row and column positions, allowing the visual encoder to model spatial relationships.
    • The calculation of position IDs must consider the spatial merge factor. After merging, each token covers a 2×2 patch group, and the position IDs must remain consistent with the merged token order; otherwise, spatial relationships may be misaligned.
    • Because images and videos share the same representation, the interface layer does not need separate flows for the two input types. Video frames and image patches can follow the same encoding path.

    4.2 Convolutional patch embedding: efficient “blocking + linear mapping”

    Patch embedding is often implemented with a 3D convolution. When the convolution kernel and stride equal the patch size in spatial dimensions and are 1 in the temporal dimension, the operation is mathematically equivalent to:

    • Non-overlapping splitting of each 14×14 pixel block.
    • Multiplying the flattened pixel values by a linear transformation matrix.

    Compared with manually using reshape and matrix multiplication, the convolution implementation has more regular memory access and is easier to map to high-performance low-level operators. For video inputs, as long as the convolution kernel covers the corresponding number of frames in the temporal dimension, the same module can directly perform frame-block patchification.

    4.3 Variable-length attention: batching images of different sizes

    In real services, image sizes in a batch are often different. The visual encoder uses variable-length attention to handle this:

    1. It first computes the cumulative lengths of each sample’s effective patch sequences.
    2. During attention computation, boundaries are determined based on cumulative lengths, ensuring that each image or video segment only interacts with tokens inside itself, and different images do not interfere with each other.
    3. It also computes the maximum sequence length in the batch, which serves as a reference for operator scheduling and memory allocation.

    Variable-length attention based on cumulative lengths often requires sequence lengths to be aligned to a certain granularity, such as a multiple of 16. Therefore, reserved padding positions must be correctly excluded in boundary computation. This allows images of different resolutions to be efficiently batched and avoids the compute waste caused by padding everything to the maximum size.

    4.4 Fusion with the language model

    After the visual encoder produces its output, the visual representation needs to enter the language model:

    • Position placement: visual tokens are usually placed at specific positions in the dialogue, marked by visual placeholders, and later replaced by the framework with actual visual embeddings.
    • Projection alignment: the merged features are mapped into the language model hidden space by a projection layer, concatenated with text token embeddings, and fed into the decoder.
    • Unified decoding: the language model performs self-attention over both text and visual tokens in a unified sequence. Visual information participates throughout inference, providing the foundation for the model to decide on its own when to observe and to use visual feedback to guide subsequent actions.

    5. Deployment practice: key trade-offs in the visual path

    5.1 Resolution and token budget planning

    The number of visual tokens is approximately proportional to image area. Since each token covers 28×28 pixels, image area is roughly equal to token count multiplied by 784.

    • 1000 visual tokens correspond to about 780,000 pixels, close to a 1000×800 resolution.
    • A 1080×2400 full-screen phone screenshot has about 2.6 million pixels. If the default upper limit is used for the budget, its visual token count will approach 8000.

    This means high-resolution long images will be heavily compressed, and small text may become difficult to recognize. It is recommended to configure budgets by task type:

    Task type Typical characteristics Configuration suggestion
    General visual question answering Clear subjects, little text Keep the default upper limit to balance quality and cost
    UI understanding, visual programming Full-screen screenshots with dense control text Prioritize cropping key regions, or process in segments and combine results
    Batch moderation, coarse classification Large resolution variation, low detail requirements Appropriately lower the upper limit to improve throughput and reduce cost
    Video understanding Frame count × tokens per frame Prioritize optimizing frame sampling rather than blindly increasing per-frame resolution

    5.2 Reusing visual encoding results

    In multi-turn conversations or multi-step agent tasks, the same image or interface screenshot is often referenced repeatedly.

    • Reuse method: where the framework supports it, visual embeddings can be cached. Identical images, or images scaled to the same size, do not need to be re-encoded. Later turns can reuse them directly, and visual encoding cost is incurred only the first time.
    • Difference from KV cache: visual embedding reuse targets the entire encoder output, while KV cache stores key-value tensors at each decoder layer. They operate at different levels and should not be confused.
    • Context budget management: visual tokens also occupy the context window. For long-running tasks, it is advisable to periodically remove stale screenshots and retain only visual information relevant to the current decision.

    5.3 Precision and quantization strategy

    Compared with FP16, BF16 has the same exponent width and fewer mantissa bits. Its dynamic range is close to FP32, and it offers better numerical stability. The visual encoder is fixed to BF16 and does not participate in FP8 quantization, mainly based on three considerations:

    • Visual input distributions are complex: photos, screenshots, document scans, and charts have very different pixel distributions. In contrast, low-bit formats are more suitable for language model parts where weights and activations are stable.
    • Quantization gains are limited: the visual encoder accounts for less than 1% of model parameters and is executed only once per request. The memory and bandwidth savings from quantization are therefore relatively limited.
    • Precision risk is asymmetric: the visual encoder is at the front of the pipeline. Small degradations in detail recognition can propagate backward and eventually affect language model output.

    It is important to note that keeping the visual encoder in BF16 does not mean the entire multimodal model abandons low-bit inference. The language model portion can still adopt common low-precision inference schemes. The two are not contradictory.

    5.4 General deployment recommendations

    • Memory estimation: peak memory usage during the visual stage is directly related to image resolution. For high-concurrency scenarios, capacity should be reserved based on “maximum visual tokens × concurrency”, rather than estimating only from the language model.
    • Batching strategy: a visual encoder with variable-length attention can accommodate inputs of different sizes. During scheduling, small images can be batched together, while large images can be processed separately, avoiding a situation where one large image slows down the entire batch.
    • Observability: it is recommended to monitor and alert on the following metrics:
      • Visual token count per request, visual encoding time, and the ratio of visual encoding time to language model prefill time;
      • First-token latency and throughput;
      • Context usage and cache hit rate.
    • Elastic scaling: multimodal request load usually fluctuates more than pure text request load. Using container platforms for elastic scaling and continuously tracking the above metrics with logging and monitoring services helps maintain a balance between quality and cost.

    6. Conclusion and selection recommendations

    From resizing an image to a multiple of 28, to the 2×2 merge compressing every 28×28 pixel area into one visual token; from an adjustable per-image budget of 16 to 8000, to a 24-layer visual encoder with about 0.56B parameters fixed at BF16, the visual path of a lightweight multimodal flagship model makes clear trade-offs around being “small but precise”:

    • Lightweight parameters: the visual encoder accounts for less than 1% of the overall model parameters.
    • Simple architecture: standard multi-head self-attention, 2D rotary position encoding, and variable-length attention are used, without introducing a separate complex visual structure.
    • Strong controllability: the visual token budget can be adjusted by scenario, and the inference precision strategy is clear and stable.

    For developers, selection and implementation should focus on the following points:

    1. Validate native visual capability: confirm whether the model can autonomously decide when to observe and revise its output based on visual feedback. This is more valuable than simply comparing visual question answering accuracy.
    2. Match the visual token strategy: align resolution and budget configuration with business scenarios, especially for text-dense interface and document tasks.
    3. Run full-path load tests: use real business data to cover inputs of different resolutions and aspect ratios, and evaluate first-token latency, throughput, and context usage.
    4. Build a layered evaluation system: track both capability dimensions such as detail recognition, interface operation, and visual programming, as well as efficiency dimensions such as latency and cost.

    Although the visual encoder is only a small module in a multimodal model, it determines “how much the model can see and how clearly it can see it”. Understanding its design logic and boundaries is essential for using multimodal capabilities correctly and effectively in real business scenarios.