Lightweight Vision Encoders: ViT Architecture and Image Tokenization for Multimodal LLMs
This article breaks down how lightweight vision encoders power multimodal large language models (LLMs). It explains why vision is increasingly integrated natively into model reasoning rather than bolted on as a separate pipeline, walks through a typical design (a 24-layer ViT with a 1024 hidden dimension, 16 attention heads, and roughly 0.56B parameters that make up less than 1% of the whole model), and details the image preprocessing flow, including 14×14 patch splitting, 2×2 merging, and adjustable token budgets. It also interprets the core ViT parameters, describes the forward pass using a unified grid_thw representation with variable-length attention, and offers practical guidance on design tradeoffs and deployment.
1. Why Multimodal LLMs Need “Native” Vision Encoding
The goal of a multimodal LLM is not simply to “look at an image and describe it,” but to let visual information genuinely participate in reasoning and decision-making. The industry has gradually reached a consensus: vision should not be a bolt-on, standalone pipeline (detect first, extract features, then feed them to the language model). Instead, it should be an organic part of the model’s capabilities—integrated natively. This means the model can autonomously decide when it needs to look at an image and how to use visual feedback to correct its subsequent actions, extending visual capabilities to more complex scenarios such as coding, interaction, and agents.
Under this approach, the vision encoder (typically a vision Transformer, or ViT) has one core responsibility: to compress an image or video of arbitrary resolution into a sequence of tokens that the language model can read. It needs to be lightweight enough not to significantly raise overall inference cost, while still being expressive enough to preserve critical visual details.
2. Positioning a Lightweight Vision Encoder
Take a typical lightweight vision encoder design as an example: the backbone is a 24-layer ViT with a hidden dimension of 1024, 16 attention heads, and a per-head dimension of 64 (i.e., 1024 / 16, without grouped-query attention, GQA). The backbone has roughly 0.56B parameters, usually accounting for less than 1% of the total parameter count of the entire multimodal model.
The point of this design is that the vision encoder is only a “translator,” responsible for translating the pixel world into tokens, while the heavy lifting is left to the larger language model backbone. The lighter the encoder, the smaller the extra end-to-end inference overhead, and the higher the deployment density per unit of compute.
One noteworthy engineering detail: this kind of vision encoder typically runs inference at a fixed BF16 precision rather than FP8 quantization. The reason is that the vision front end is relatively sensitive to numerical stability, and quantization error can be amplified stage by stage across image features, ultimately affecting downstream understanding quality. Keeping BF16 strikes a more reliable balance between cost and precision.
3. Image Preprocessing Pipeline
Before an image is fed into the ViT, it first goes through a normalization step (think of it as an “image processor” component). The whole flow can be broken down into the following key steps.
3.1 Normalization and Size Alignment
First, normalize the image by scaling its pixels to a suitable size and applying zero-padding so that the image’s side lengths become multiples of 28. The value 28 is not arbitrary: the patch size is 14, followed by a 2×2 merge, and 14 × 2 = 28. Aligning the side lengths to multiples of 28 ensures that no incomplete edge blocks appear when splitting and merging.
3.2 Patch Splitting and 2×2 Merging
Next, split the image into base patches of 14×14 pixels, then perform a 2×2 merge: combine 4 adjacent base patches into a single larger visual unit. As a result, each 28×28 pixel region ultimately corresponds to one visual token.
The value of merging is to reduce the number of tokens and shorten the sequence length, while using a larger receptive field to retain some spatial context—a common compromise between precision and efficiency.
# Preprocessing illustration (pseudocode)def preprocess(image, min_tokens=16, max_tokens=8000, patch=14, merge=2):unit = patch * merge # 28 = 14 × 2# 1) Resize to a suitable size# 2) Round up to a multiple of unit and apply zero-padding# 3) Split into 14×14 base patches# 4) Merge every 2×2 into a visual unit# 5) Each 28×28 region produces 1 visual tokentokens = merge_and_tokenize(image)return clamp(tokens, min_tokens, max_tokens)
3.3 Token Budget Control
For context-budget-sensitive scenarios, you can set upper and lower limits on the number of tokens allowed per image (a typical range is 16 to 8000). The upper limit prevents high-resolution large images from blowing up the context, while the lower limit ensures key images are not compressed to the point of being unrecognizable. This adjustable budget lets the same model flexibly adapt to different tasks such as question answering, document parsing, and fine-grained recognition.
4. Reading the Core ViT Parameters
| Item | Typical Value | Description |
|---|---|---|
| Depth | 24 | Determines representation depth |
| Hidden size | 1024 | Number of feature channels |
| Attention heads | 16 | Per-head dimension 64, no GQA |
| patch_size / merge | 14 / 2×2 | Split granularity and merge factor |
| Output dimension | 4096 | Aligned with the LLM hidden size |
| Backbone parameters | ≈ 0.56B | Usually <1% of the total model |
| Inference precision | BF16 | Usually no FP8 quantization |
Pay special attention to the point that “the output dimension of 4096 is aligned with the LLM hidden size”: the vision encoder’s output must be mapped into the language model’s hidden space so that visual tokens can be concatenated with text tokens into the same sequence and processed uniformly. Therefore, the end of the encoder usually needs a projection layer to align the ViT feature dimension to the language model dimension—this is the key bridge connecting the “vision” and “language” worlds.
5. Forward Inference Flow
The forward pass of this kind of vision encoder typically supports both images and video. The key is to handle different inputs with a unified grid description. Let grid_thw represent the patch grid across three dimensions (T, H, W): an image corresponds to T=1, while a video has T>1. From this grid you can derive position numbering, as well as the sequence boundary information required for attention computation (such as cumulative sequence lengths), thereby supporting variable-length, batched, efficient attention computation.
A schematic forward pass looks like this:
def forward(self, hidden_states, grid_thw, **kwargs):# ① Derive position numbering from grid informationposition_ids = get_vision_position_ids(grid_thw, self.spatial_merge_size, ...)# ② Derive metadata for attention, such as cumulative sequence lengthscu_seqlens, max_seqlen = get_vision_attention_seqlens(grid_thw, self.config, ...)# ③ Patchify: embed pixel blocks into features with a convolutionhidden_states = self.patch_embed(hidden_states) # Conv-like patch embedding# ④ Add position encodingshidden_states = hidden_states + self.position_embeddings(position_ids)# ⑤ Pass through multiple Transformer layershidden_states = self.encoder(hidden_states, cu_seqlens, max_seqlen)return hidden_states
There are two design points worth noting here:
- Position numbering is generated dynamically from the grid. Images and videos share the same logic, with video adding extra positional relationships along the time dimension. As a result, the position encoding can describe both spatial and temporal structure.
- Attention uses cumulative sequence lengths (cu_seqlens) together with variable-length attention. Each image and each video segment is treated as an independent subsequence, which both prevents incorrect cross-sample attention between different samples and allows multiple images to be packed into a single batch, thereby improving throughput.
6. Design Tradeoffs and Engineering Recommendations
Pulling the above together, the design of a lightweight vision encoder can be summarized into a few core tradeoffs:
- Lightweight vs. expressive. A 0.5B-class encoder that accounts for less than 1% of the total model controls cost while still ensuring sufficient visual representation capability through its 24-layer depth.
- Precision vs. cost. Using fixed BF16 instead of aggressive quantization buys more stable numerical behavior in the vision front end.
- Detail vs. context. The 2×2 merge and the adjustable token budget let developers freely trade off between “seeing clearly” and “fitting in” according to task needs.
- Unified vs. efficient. Combining grid_thw with variable-length attention unifies images and video, as well as variable-length inputs, into the same efficient inference path.
When it comes to engineering deployment, there are also a few practical recommendations:
- Plan the context budget in advance. Estimate visual token consumption based on typical image-text ratios, set reasonable upper and lower limits, and avoid long documents or high-resolution images crowding out the text space.
- Balance batching against GPU memory. While variable-length attention can improve throughput, large images and multi-frame videos can still significantly increase GPU memory usage, so you should run stress tests with respect to batch size and resolution.
- Verify precision consistency. If you later introduce quantization, evaluate the precision loss of the vision encoding branch separately rather than only looking at metrics from the text branch.
- Leverage general-purpose cloud capabilities on the deployment side. This kind of multimodal service can typically run on a container platform, paired with object storage for image assets, monitoring and alerting to track latency and throughput, and log services for request-level troubleshooting—building a stable, observable inference service with standardized cloud components.
In summary, the core idea of native vision encoding is to treat vision as part of the model’s reasoning rather than a bolt-on pipeline. A well-designed lightweight ViT, through reasonable splitting and merging, a controllable token budget, an output dimension aligned with the language model, and a unified and efficient image/video processing path, keeps quality high while pushing cost low enough—which is also a mainstream direction in the current engineering evolution of multimodal LLMs.
