Vision Encoders in Multimodal LLMs: How Images Become Tokens, End to End
This article breaks down how vision encoders in multimodal large language models turn arbitrary-resolution images—and even video frames—into token sequences that a language backbone can read uniformly. It explains why a lightweight Vision Transformer (ViT) is the component of choice, then walks through image preprocessing (resizing, zero-padding to multiples of 28, patch splitting, and 2×2 spatial merging), patch embedding, position encoding, per-image token budgeting, backbone parameters, and the decision to fix precision at BF16 rather than FP8. A complete forward pass is traced step by step, including variable-length attention segmentation and the final projection into the language model's hidden dimension. The conclusion distills practical principles around alignment, token budgets, layered precision optimization, and why a module under 1% of total parameters can still set the ceiling for vision quality and image-text alignment.
1. Introduction: Where the Vision Encoder Fits in a Multimodal LLM
In a large multimodal model, “vision” is never as simple as feeding an image into a network. The truly valuable capability is letting the model actively look at a scene when needed and use that visual feedback as the basis for subsequent reasoning and action. To support this, the visual signal has to be integrated natively into the language backbone, not bolted on as an after-the-fact analysis module.
This raises a core engineering question: how do you efficiently and losslessly convert an image of arbitrary resolution (or a sequence of video frames) into a token sequence that the language model can read in a unified way? The component that usually owns this job is a small but critical Vision Transformer (ViT) encoder.
A notable detail: the vision encoder often accounts for a very small share of total parameters. In the kind of design discussed here, the backbone vision network has roughly 0.56B parameters—less than 1% of the whole multimodal model. This “lightweight front end, heavyweight back end” architectural trade-off is exactly the entry point for understanding modern multimodal models.
2. Why a ViT: The Design Philosophy of the Vision Encoder
Traditional visual understanding relies on convolutional networks that progressively downsample. The core idea of a ViT is much simpler: split the image into small patches and treat them the way you treat words in natural language. Each patch is mapped to a vector, and a standard Transformer encoder then models them.
This design delivers two direct benefits:
- Architectural unification: vision and language share a similar Transformer backbone, which makes cross-modal feature alignment and joint training easier.
- Sequential processing: the image is expressed as a 1D token sequence that plugs straight into the language model’s attention mechanism.
One thing worth noting: so that inputs of any resolution can flow through the same pipeline, the encoder’s input stage has to do a lot of “normalization” work. That is the prerequisite for understanding the engineering details that follow.
3. Image Preprocessing: From Pixels to Standardized Input
Before reaching the encoder, an image goes through a standardized preprocessing stage. Two steps here directly determine how regular the resulting token sequence is.
3.1 Resize and Zero-Padding
Preprocessing first resizes the image to a suitable size and, when necessary, zero-pads it to a multiple of 28. This “28” is not arbitrary—it comes from the merge factor used downstream. The base patch size is 14×14, and there is a subsequent 2×2 spatial merge, so 14 × 2 = 28 lands exactly on an integer boundary of a “valid unit.”
The advantage of zero-padding over cropping or stretching is that it keeps the image content intact and lets the later splitting and merging run on a regular grid, avoiding alignment ambiguity at the edges.
3.2 Patch Splitting and Spatial Merging
The core preprocessing flow can be summarized as:
Original image→ Resize + zero-pad to a multiple of 28→ Split into 14×14 base patches→ Merge every adjacent 2×2 (4 in total) base patches into one unit→ Each 28×28 region corresponds to one visual token
The “split first, then merge” strategy has a clear motivation: using 14×14 as the granularity directly would produce a massive number of tokens, and such sequence lengths are unfriendly to attention computation. Merging 4 base patches into 1 via the 2×2 step cuts the number of visual tokens to a quarter of the original, striking a balance between preserving detail and controlling sequence length.
4. Patch Embedding and Position Encoding
Once inside the encoder backbone, the serialized patches are converted into vector representations.
- Patch embedding: typically implemented with a convolution layer. Notably, when the model needs to handle video (i.e., introduce a temporal dimension), it uses a 3D convolution (Conv3d) to patch across the “time × height × width” dimensions simultaneously. This expresses a video frame sequence as a unified token sequence that shares the same modeling logic as a static image.
- Position embedding: Transformers are inherently position-agnostic, so positional information has to be injected explicitly. On the vision side, position embeddings are usually generated from the 2D/3D grid coordinates of the patches, letting the model distinguish spatially close versus distant relationships.
5. Token Budget Control: Adapting to Context Length
Visual tokens are a double-edged sword: the higher the resolution and the richer the detail, the more tokens are consumed—and that directly eats into the language context window. To control this cost, engineering practice typically introduces a per-image token budget mechanism:
min_image_tokensandmax_image_tokensset a range for the number of tokens generated per image (a common range is roughly 16 to 8000);- the lower bound guarantees the model can “see” enough basic information, while the upper bound prevents very high-resolution images from exhausting the context.
A schematic version of the budget logic looks like this:
def compute_grid(image_h, image_w, unit=28, min_tok=16, max_tok=8000):# Each valid unit corresponds to one visual tokencols = image_h // unitrows = image_w // unittotal = cols * rows# Clamp to the budget range for context-aware dynamic resolutiontotal = min(max(total, min_tok), max_tok)return total
The value of this design is that the model can dynamically decide “how finely to look” based on the task and the context budget—simple image-text matching can run on a low token count, while complex visual programming or document understanding can use a higher one.
6. Backbone Parameter Breakdown
A typical parameter configuration for this kind of lightweight vision encoder is shown below:
| Item | Value | Notes |
|---|---|---|
| Depth (layers) | 24 | Encoder depth |
| Hidden size | 1024 | Vector dimension per token |
| Attention heads | 16 | Number of heads in multi-head attention |
| Head dimension | 64 | 1024 / 16, and no GQA is used |
| Patch / Merge | 14 / 2×2 | Splitting granularity and merge factor |
| Output dimension | 4096 | Must align with the language backbone’s hidden dimension |
| Parameters | ≈ 0.56B | Less than 1% of the total model |
One key alignment detail: the encoder’s output dimension must match the language model’s hidden dimension (in this example, the vision side maps to 4096 dimensions), otherwise it cannot plug seamlessly into the language backbone’s attention layers. In addition, “no GQA (grouped-query attention)” is a pragmatic trade-off—in a relatively shallow encoder with a modest head dimension, standard multi-head attention is sufficient, and introducing GQA would add complexity for limited benefit.
7. Precision Choice: Why BF16 Is Fixed
For inference precision, this kind of vision encoder typically fixes BF16 and does not go through FP8 quantization. The reasoning behind this trade-off is worth spelling out:
- The numerical distributions of the vision encoder (especially patch embeddings and normalized activations) are relatively sensitive to quantization noise; lower precision can introduce errors that get amplified downstream in image-text alignment.
- The compute during the encoding stage is small relative to the backbone, so even at BF16 its inference cost is a limited share of the overall multimodal pipeline.
In other words, this is a layered optimization approach: keep precision on the precision-sensitive small module, and only discuss quantization on the compute-heavy large module.
8. Breaking Down the Forward Pass
Chaining the pieces above together, a single forward pass through the vision encoder looks roughly like this:
def forward(self, hidden_states, grid_thw, **kwargs):# 1. Compute position indices from the grid dimensionsposition_ids = get_vision_position_ids(grid_thw, self.spatial_merge_size)# 2. Compute attention segmentation info for variable-length sequences (cu_seqlens / max_seqlen)cu_seqlens, max_seqlen = get_vision_attention_seqlens(grid_thw, self.config)# 3. Patch embedding (Conv3d, compatible with both images and video)hidden_states = self.patch_embed(hidden_states)# 4. Add position embeddingsposition_embeddings = self.rope_or_learned_positional_encoding(...)hidden_states = hidden_states + position_embeddings# 5. Pass through the Transformer encoder stack (with variable-length attention)for layer in self.layers:hidden_states = layer(hidden_states, cu_seqlens, max_seqlen)# 6. Spatial merge (2×2) and output projection, mapping to the language backbone's dimensionhidden_states = self.spatial_merge(hidden_states)return self.proj_out(hidden_states)
Two parts of this flow deserve special attention:
- Variable-length attention segmentation (cu_seqlens): because different images produce different token counts, multiple image sequences are concatenated into a single computation. By marking the boundaries of each image sequence with cumulative sequence lengths, the attention computation can correctly isolate different images within one batch and avoid cross-image interference.
- Spatial merge and output projection: the merge reduces sequence length while preserving geometric adjacency, and the final projection maps the visual features to the same dimension as the language model, completing the “vision-to-language” interface alignment.
9. Summary and Practical Takeaways
Back to the original premise: the value of a vision encoder lies in making “when to observe, how finely to look, and how to turn a scene into tokens that can be reasoned over” a controllable, native capability—not a bolt-on patch. The lightweight ViT approach outlined here distills into a few general practical principles:
- Alignment first: the padding factor in preprocessing and the merge granularity must be strictly self-consistent with the downstream splitting logic; any mismatch will corrupt the token sequence.
- Token budget as resource scheduling: using a min/max token range to achieve context-aware dynamic resolution is the core means of balancing detail against length.
- Layered precision optimization: keep higher precision on the numerically sensitive vision front end, and leave quantization gains for the compute-heavy backbone.
- Small modules can still drive the experience: at under 1% of total parameters, this module still determines the whole multimodal system’s “vision resolution ceiling” and the quality of image-text alignment.
Understanding this full “image-to-token” pipeline not only helps developers design more efficient vision front ends, but also provides a clear engineering reference for selecting, tuning, and deploying cross-modal models.
