How Lightweight Vision Encoders Support Multimodal Models
In multimodal large models, the vision encoder links images and video to language. This article explains why a moderately sized, resolution-aware encoder often matters more than a deeper network, covering patch grids, 2×2 spatial merging, positional encoding, variable-length attention, token budgets, precision, and a typical forward path.
In a multimodal large model, the vision encoder connects the visual world to language space. It does more than turn an image into a set of features. It extracts representations that preserve spatial relationships, local detail, and semantic information from images and video, then passes those representations to the language model for understanding, reasoning, question answering, or code generation.
In recent years, vision-module design has shifted from “large and comprehensive” toward lighter, more efficient designs. The reason is straightforward: in a complete multimodal model, the language model usually accounts for most of the parameters and memory. The vision encoder is smaller, yet it strongly affects input length, time to the first token, memory use, and the cost of processing long images and video. A vision encoder with a moderate parameter count, stable outputs, and support for different resolutions is often more valuable in practice than simply stacking a deeper network.
1. What the vision encoder must do
A traditional image classifier usually outputs a class or a global feature. A multimodal language model needs richer information. When reading a technical screenshot, the model must do more than decide that “this is code.” It also needs code structure, indentation, symbols, table layout, and local text. When analyzing video, it must understand how frames change and the order in which actions occur.
A vision encoder typically works at three levels:
- Preserve visual content: recognize objects, text, color, texture, and local structure.
- Preserve spatial relationships: tell the model where regions sit in the image and how they relate to each other.
- Convert features into language-model input: compress 2D or 3D visual features into a sequence of vectors that can be fed alongside text tokens.
The overall process can be summarized as:
Raw image or video↓Resize and padding↓Local patch extraction↓Visual feature encoding↓Merge and compress neighboring features↓Map to the language-model hidden size↓Visual token sequence
The number of visual tokens produced at the end directly determines how much context pressure the language model faces next.
2. From arbitrary resolution to a regular grid
Images do not share the same aspect ratio. Scaling every image to one fixed size is simple, but it introduces clear geometric distortion. A wide screenshot forced into a square can warp text. Keeping the original aspect ratio exactly, however, makes a uniform batch pipeline harder.
A common approach is to resize the image while preserving its aspect ratio, then pad height and width with zeros so both are integer multiples of a chosen stride. Suppose the vision encoder uses 14×14-pixel base patches and later merges neighboring 2×2 patches. Then both height and width of the input are best divisible by 28.
This has two benefits:
- The visual feature grid has a regular size, which is easier to batch.
- Merging does not need special branches when a boundary is too small.
Padding does not rewrite the original content. New regions are usually filled with zeros or another fixed pixel value, and later stages reduce their influence through position information and the feature distribution. For text-heavy images, the resize policy matters even more: aggressive scaling can make characters unreadable. Practical systems therefore also set a minimum and a maximum number of visual tokens.
3. Patch splitting and spatial compression
After size normalization, the image is split into 14×14-pixel base patches. Each base patch is the smallest input unit of the vision encoder.
If the image size is:
height = Hwidth = W
the number of base patches is approximately:
(H / 14) × (W / 14)
Sending every base patch into the rest of the network would grow the token count quickly. To cut computation, the system usually merges neighboring 2×2 base patches. After merging, each visual token covers about a 28×28-pixel region of the original image, and the token count is approximately:
(H / 28) × (W / 28)
This step is spatial downsampling. It shortens the sequence the language model must process while trying to keep local structure.
A higher compression ratio is not always better. Strong compression often has limited impact on natural scenes, but on code, tables, scanned documents, and small text it can drop critical detail. The encoder has to balance information fidelity against token cost.
4. A typical lightweight visual backbone
A compact visual backbone can use a 24-layer network, a 1024-dimensional hidden representation, and 16 attention heads. Under that configuration, each head has dimension:
1024 / 16 = 64
Characteristics of this structure include:
- A moderate depth that keeps compute under control.
- A hidden size large enough to express complex visual semantics.
- A head count that matches the hidden size, so the layout stays clear.
- No extra grouped-query mechanism, which keeps the implementation relatively direct.
- After spatial merging, visual representations are projected to 4096 dimensions so they align with the language model’s hidden space.
The visual backbone is about 0.56B parameters. Compared with a full multimodal model, that is usually a small share. Its effect on system performance does not depend on parameter count alone. Input resolution, visual token count, the attention implementation, and the batching strategy all matter.
A small vision module does not mean its inference cost can be ignored. High-resolution images or long videos can produce many visual tokens and increase attention compute in the language model that follows.
5. Why visual positional encoding matters
One of the largest differences between images and text is that images have an explicit two-dimensional layout. Language-model tokens are usually ordered in one dimension, while visual tokens come from a two-dimensional grid. Without position information, the model has trouble telling “text in the upper left” from “text in the lower right.”
During visual encoding, the system typically builds positional encodings from the grid height, grid width, and any time dimension. For video, the frame index can be an extra time coordinate, forming a three-dimensional position structure:
Time: which frameHeight: which row in the imageWidth: which column in the image
In implementation, a common approach is to compute position indices from each sample’s grid size, then adjust the index range using the spatial merge ratio. That lets the model understand:
- Relative positions of regions within the same frame.
- Temporal order across frames.
- How visual tokens map back to original regions after compression.
If positional encoding is wrong, the model may still recognize local objects but fail on layout, direction, order, and timing.
6. Visual attention and variable-length inputs
Different images produce different numbers of visual tokens. Padding every sample to the same length wastes a lot of compute. Inference frameworks usually record the start and end of each sample’s visual sequence and build cumulative sequence lengths from those bounds.
The idea looks like this:
Sample A: n1 visual tokensSample B: n2 visual tokensSample C: n3 visual tokensCumulative bounds:[0, n1, n1+n2, n1+n2+n3]
Attention uses these bounds to separate samples so one sample’s visual tokens do not attend into another sample. For variable-length visual input, this lowers the padding ratio and improves memory use.
Video adds frame count. The number of visual tokens is roughly:
total tokens ≈ number of frames × visual tokens per frame
A video system therefore has to control frame sampling, resolution, and the maximum token count together. Otherwise the input length can easily exceed the context budget.
7. Why a visual token budget matters in practice
A lightweight vision encoder usually exposes a minimum and a maximum token count. These are not mere format limits. They are resource controls for the whole multimodal system.
1. Minimum token count
If an image is very small, or its content has been compressed too hard, there may not be enough visual tokens to express detail. A minimum keeps small images from being compressed until they are unusable.
2. Maximum token count
For very high-resolution images, a maximum stops a single request from consuming too much memory and context. In an online service, it also keeps a few complex requests from slowing the whole batch.
3. Dynamic allocation
A better plan is not one fixed budget for every image, but a budget that depends on the task:
- Ordinary visual question answering: a medium token budget.
- Document recognition and code screenshots: a higher budget.
- Video overviews: control frame count first.
- Fine-grained region analysis: crop the target region, then raise local resolution.
A strategy of low cost globally and high precision locally is often more efficient than simply raising resolution everywhere.
8. Precision and deployment
At inference, the visual backbone usually runs in a low-precision format such as BF16 to reduce memory and raise throughput. For visual features, low-precision inference is numerically stable in most cases, but that does not mean every operator can be quantized arbitrarily.
Deployment should pay particular attention to the following:
Whether image preprocessing matches training
Including resize method, padding rules, pixel normalization, and color-channel order.Whether positional encoding supports dynamic grids
A design that only fits a fixed size can hit boundary problems on long images or video.Whether the visual output dimension is aligned
Visual features must be mapped to the hidden size the language model accepts.Whether mixed precision covers every operator
Some normalization, positional encoding, or special projection layers may need higher precision to avoid numerical issues.Whether batching supports variable-length input
Otherwise a large number of padding tokens cancels the gains of a lightweight design.
9. A reference forward pass
From an engineering point of view, a typical visual forward pass can be sketched as follows:
def encode_visual_inputs(pixel_values, grid_shape):# 1. Convert pixel input into base image patchespatch_features = patch_embedding(pixel_values)# 2. Build position indices from the time, height, and width gridposition_ids = build_visual_position_ids(grid_shape,merge_size=2)# 3. Organize attention using variable-length sequence boundssequence_bounds = build_sequence_bounds(grid_shape)# 4. Run multilayer visual attention and feed-forward networksvisual_features = visual_transformer(patch_features,position_ids=position_ids,sequence_bounds=sequence_bounds)# 5. Spatially merge neighboring visual regionsmerged_features = spatial_merge(visual_features,merge_size=2)# 6. Project into the language-model hidden sizeoutput_features = visual_to_language_projection(merged_features)return output_features
This flow shows the main design choices: build local representations first, inject spatial position, model context with variable-length attention, then compress spatially and align dimensions across modalities.
10. Do not judge performance by parameter count alone
Evaluating a lightweight vision encoder means looking beyond model size:
- Recognition accuracy: whether text, objects, tables, and layout are preserved well enough.
- Detail recovery: whether small type and low-contrast content are easily lost.
- Long-image handling: whether ultra-wide screenshots and long documents are compressed too hard.
- Video scaling: whether latency and memory stay manageable as frame count grows.
- Token efficiency: how many visual tokens a task needs for the same quality.
- Deployment stability: whether behavior stays consistent across resolutions and batch sizes.
- End-to-end latency: whether vision encoding or language generation is a clear bottleneck.
In practice, a good lightweight design is not just fewer layers. Preprocessing, spatial merging, positional encoding, attention, and the language-model input have to form a closed loop. Only when fidelity and compute cost are balanced does the vision module actually serve multimodal reasoning.
Overall, a moderately deep visual backbone, regular patch splitting, 2×2 spatial merging, a dynamic token budget, and variable-length attention form a general path to a cost-effective multimodal system. That path keeps the vision branch’s parameter count in check and makes the compute cost of high-resolution images and video manageable, providing a stable base for question answering, document analysis, code understanding, and visual reasoning.
