Lightweight Vision Encoders for Multimodal Models: Visual Tokens, Preprocessing, and Cost Optimization
This article breaks down how a typical lightweight vision encoder—a 24-layer, roughly 0.56B-parameter design—turns pixels into visual tokens for multimodal models. It covers the tradeoff between bolt-on adapters and native integration, the parameter math behind a sub-billion-parameter vision tower, dynamic-resolution preprocessing with arbitrary tiling, the forward-pass flow, why the encoder stays in BF16 rather than FP8, and how visual token budgets drive cost. It closes with an engineering checklist for balancing accuracy, cost, and latency. Note that the example configuration is drawn from public technical materials to illustrate structure and methods, so actual parameters, precision, and preprocessing details should be confirmed against the corresponding model documentation.
In building multimodal applications, developers often care about three questions: How many parameters does the vision encoder have? How are visual tokens counted? How should dynamic resolution be chosen? Around these questions, this article takes a typical lightweight vision encoder—24 layers, about 0.56B parameters—as a sample and breaks down its design approach, preprocessing pipeline, forward path, inference precision, and cost optimization methods.
Note: The example configuration below is compiled from public technical materials to explain structure and engineering methods; actual parameter counts, precision, and preprocessing details should be confirmed against the corresponding model documentation.
1. Introduction: Vision Is More Than “Looking at Images”—It Expands the Decision Boundary
In multimodal models, the vision module is often understood as an “image translator”: converting pixels into a sequence a language model can read. But stopping at this level limits visual capability to “recognition and description.” A more complete view is that vision encoding expands the information boundary the model can reach, letting it read UI screenshots, interpret chart data, observe rendering results, and make the next decision based on visual feedback.
This leads to two design implications:
- Vision needs to be natively integrated into the model. The vision encoder should not be just a bolt-on recognition service but part of the model’s inference pipeline. Only then can the model decide on its own “when to observe” and “what to observe,” and use visual results to guide generation.
- Visual capability depends on long context and sequence mixing. In scenarios like code generation, web operation, and GUI automation, the model needs to form a closed loop of “observe—reason—generate—observe again.”
Different tasks place different demands on the vision encoder:
- Understanding tasks: Such as document parsing, table recognition, and UI reconstruction, prioritize “seeing clearly” and need to preserve layout, text, and spatial relationships.
- Interaction tasks: Such as GUI operation and web automation, prioritize “seeing fast”—high-frequency, low-cost observation often matters more than a single high-resolution close read.
- Generation tasks: Such as generating chart code, web structures, or vector graphics and then reviewing the rendering, prioritize “seeing accurately”—visual feedback is used to catch layout misalignment, text overflow, and similar issues.
2. Design Tradeoffs: From Bolt-on Adapter to Native Vision Backbone
There are two common approaches to connecting vision to a language model.
Approach 1: Bolt-on visual adaptation. Use a separately trained general-purpose vision model and map its visual features into the language model space through a projection layer or adapter module. This approach reuses mature components and starts quickly, but the vision encoder and language model are optimized separately, and cross-modal alignment relies mainly on a lightweight connection layer, which can lead to an imbalance of “strong language, weak vision.”
Approach 2: Natively integrated vision encoder. Make the vision encoder an important component of the model itself, jointly designed and jointly trained with the language backbone. Visual tokens and text tokens are processed uniformly within the same sequence.
Key characteristics of native integration include:
- Unified sequence space: Images are compressed into tokens isomorphic to text, reusing the same mechanisms for position numbering, attention masks, and context windows.
- Decision-making handed to the model: Whether a clearer view is needed, or whether to zoom into a region, is decided by the model itself based on the task.
- Natural visual output: The output side can support a “view—draw—review” closed loop, providing a foundation for GUI operation and visual self-correction.
3. Parameter Breakdown: Why It Can Be “Light Yet Sufficient”
The example configuration discloses a set of typical lightweight vision encoder parameters:
| Item | Value |
|---|---|
| Network depth | 24 layers |
| Hidden dimension | 1024 |
| Attention heads | 16 |
| Base patch size | 14 pixels |
| Merge scheme | 2×2 merge |
| Output dimension | 4096 |
| Backbone parameters | ~0.56B |
| Inference precision | Fixed BF16 |
The vision encoder’s main job is to convert pixels into semantic tokens and hand them to the language backbone. The language backbone holds the vast majority of parameters and is responsible for high-level semantic understanding and reasoning. The vision encoder’s goal is not to handle all understanding but to preserve key information at an appropriate granularity.
Estimating from parameters, for a Transformer structure with hidden dimension h = 1024 and a feed-forward intermediate dimension typically 4h:
- Self-attention projections: Wq, Wk, Wv, Wo each about h×h, roughly 4h² ≈ 4.2M per layer;
- Feed-forward network: two linear transformations about 8h², roughly 8.4M per layer;
- Per-layer total about 12.6M, and 24 layers about 0.30B.
Adding modules such as the merge projection and position-encoding-related structures pushes the total toward 0.5B+, on the same order as “about 0.56B.” This shows that a 24-layer vision tower with hidden dimension 1024 is itself a sub-billion-parameter structure, and its lightweight nature comes more from restraint in depth and width than from special structural tricks.
4. From Pixels to Visual Tokens: Dynamic Tiling at Arbitrary Resolution
Traditional vision encoders usually require a fixed input size, such as 224×224 or 336×336. This is unfriendly to non-standard inputs like screenshots, documents, and long images: forced scaling loses small text and details, while cropping can break the layout. So dynamic resolution and visual token budget control become critical.
The processing pipeline typically includes:
- Plan the grid: Estimate the target grid shape based on the original image’s width/height and the token budget.
- Scale proportionally: Resize the image so it does not exceed the target grid dimensions, avoiding distortion.
- Zero-pad to align: Pad the bottom-right of the resized image with zeros so it reaches the target size of cols×28 and rows×28.
- Tile and merge: The image is first split into 14×14 pixel base patches, then 2×2 merged; each 28×28 pixel region corresponds to one visual token.
Note that zero-padding should be done after the grid and scaled dimensions are determined, not by blindly padding the original image up to a multiple of 28.
PATCH_SIZE = 14MERGE_SIZE = 2STRIDE = PATCH_SIZE * MERGE_SIZE # 28def plan_image_grid(width, height, min_tokens=16, max_tokens=8000):# 1) Estimate target token count by visual tokens ≈ (W / 28) × (H / 28)estimated_tokens = max(1, int(round(width * height / (STRIDE * STRIDE))))target_tokens = max(min_tokens, min(max_tokens, estimated_tokens))# 2) Keep aspect ratio and allocate target tokens into a cols × rows gridratio = width / heightcols = max(1, int(round((target_tokens * ratio) ** 0.5)))rows = max(1, int(round(target_tokens / cols)))# 3) Get the target grid size; subsequent scaling and zero-padding follow thistarget_w = cols * STRIDEtarget_h = rows * STRIDEreturn (target_w, target_h), (rows, cols)
def resize_and_pad(image, plan):(target_w, target_h), grid = plan# Scale proportionally first, so the resized image does not exceed the target gridscale = min(target_w / image.width, target_h / image.height)new_w = max(1, round(image.width * scale))new_h = max(1, round(image.height * scale))resized = image.resize((new_w, new_h))# Zero-pad after scaling to keep both width and height multiples of 28pad_right = target_w - new_wpad_bottom = target_h - new_hpadded = pad_zero(resized, pad_right, pad_bottom)return padded, grid
When processing multiple images, each image can have a different grid shape. In the end, they are expanded into visual tokens within the same sequence by placeholders and handled uniformly through variable-length attention.
5. Forward-Pass Flow: Three Steps
The example configuration’s forward pass can be summarized in three steps: position numbering, variable-length attention boundaries, and 3D convolution tiling.
def forward(self, hidden_states, grid_thw, **kwargs):# grid_thw gives the (t, h, w) for each image or video segment# Note: t is the number of temporal patches, not the number of sampled framesposition_ids = get_vision_position_ids(grid_thw, self.spatial_merge_size)cu_seqlens, max_seqlen = get_vision_attention_seqlens(grid_thw, self.config)# 3D convolution performs tiling and embeddinghidden_states = self.patch_embed(hidden_states)# Position embedding and vision backbone computationposition_embeddings = self.rope_init(...)# Followed by attention, 2×2 merge, and projection to the language model dimension
1. Grid Shape grid_thw
Each static image or video segment produces a triplet: (t, h, w).
- Static image: usually t = 1.
- Video: t is the number of temporal patches, not the number of sampled frames.
- If the video’s sampled frame count is F and temporal_patch_size = 2, then t is usually obtained by grouping F along the temporal dimension. For example, when F = 8, t may be 4; if the frame count is odd, the specific implementation may handle it by padding, truncation, or skipping—confirm against the model implementation.
So you cannot treat the sampled frame count directly as the temporal dimension in grid_thw. The number of visual tokens is generally related to t, h, w, and the merge scheme. In common implementations with 2×2 spatial merge and no additional temporal merge, the number of visual tokens can be approximately understood as:
Visual tokens ≈ t × h × w / (2 × 2)
2. Position Numbering position_ids
Language models only understand a one-dimensional sequence, while images are naturally two-dimensional and videos three-dimensional. The role of position numbering is to map each patch’s coordinates into the language model’s one-dimensional sequence positions.
The key point: the flattening order on the vision side must match patch_embed and position_ids exactly. If the vision side flattens row-major while position numbering is computed column-major, the model may misread “vertical relationships” as “horizontal relationships.” Such problems are especially noticeable in tasks like tables, flowcharts, and UI reconstruction.
3. Variable-Length Attention cu_seqlens
If all images are padded to the same length, a lot of compute is wasted on invalid positions. Variable-length attention uses cumulative sequence lengths cu_seqlens to mark the start and end of each valid token segment, so inputs of different sizes can compute attention independently within the same batch without interfering with each other.
4. 3D Convolution Tiling patch_embed
Here a 3D convolution is used instead of a 2D convolution to cut patches. With a temporal kernel size of 2, it can correspond to the combination of two adjacent frames, letting images and videos share the same patch embedding logic. For videos, temporal grouping affects t in grid_thw and must be distinguished from the sampled frame count and temporal_patch_size.
5. Merge and Projection
- 2×2 merge: The merge may be concatenation or averaging. Concatenation makes the dimension 4h, requiring another projection back to h; averaging keeps the dimension unchanged and is more parameter-efficient.
- Projection into the language model space: The merged visual features are mapped by a projection layer into 4096 dimensions, aligned with the language model’s hidden dimension, and then inserted into the text sequence as visual tokens.
6. Inference Precision: Why the Vision Encoder Stays in BF16 Rather Than FP8
The example configuration states clearly: the vision encoder uses BF16 at inference and does not participate in FP8 quantization. This is a tradeoff between benefits and risks.
1. Limited Quantization Benefits
Estimating at 2 bytes per parameter, about 0.56B parameters correspond to about 1.1GB of weights. Compared with language backbones that are often tens of GB, this proportion is usually small. Even converting everything to 1-byte FP8 saves limited memory, with little impact on overall memory and throughput.
From a compute perspective, the floating-point operations for a single patch from 3D convolution to final projection can be estimated with the following formula:
FLOPs ≈ 12h² + 4h × d_out
Taking h = 1024 and d_out = 4096:
12 × 1024² + 4 × 1024 × 4096= 12,582,912 + 16,777,216= 29,360,128≈ 2.9 × 10⁷
Compared with the language backbone’s compute at the same token count, the vision encoder’s compute share is usually small. So “high precision for the vision tower, low precision for the language backbone” is often the more prudent division.
2. Higher Numerical Risk
The vision encoder is quite sensitive to pixel-level differences, and quantization error can be amplified stage by stage and propagate along the “vision → language” chain. The vision side includes 3D convolution, variable-length attention that changes per image, and multi-dimensional rotary position encoding—structures that are not fully equivalent to the regular matrix multiplications of the language backbone. Directly applying a low-precision quantization scheme makes it hard to guarantee both stability and benefit.
7. Visual Token Budget: The Key Lever Balancing Cost and Effectiveness
In multimodal requests, the number of visual tokens directly affects context length, prefill cost, and attention computation. An approximate formula is:
Visual tokens ≈ (image width / 28) × (image height / 28)
This formula is approximate; the actual result is affected by factors like cropping, scaling strategy, min/max token budgets, and video sampled frame count.
For example, a 1792×1792 screenshot:
1792 / 28 = 6464 × 64 = 4096
That is, about 4096 visual tokens. In batch and long-context scenarios, such tokens significantly raise cost.
| Visual tokens | Approx. equivalent resolution | Typical use cases |
|---|---|---|
| 16–64 | ~112×112 to ~224×224 | Icon recognition, rough UI state judgment |
| 256 | ~448×448 | Medium screenshots, single-region local zoom |
| 1024 | ~896×896 | UI reconstruction, single-page documents |
| 4096 | ~1792×1792 | High-resolution charts, long documents |
| 8000 | ~2528×2528 | Very large stitched images, long images |
In engineering practice, you can refer to the following strategies:
- Distinguish task modes: Use a small budget to “look a few more times” for interaction tasks, and a large budget to “see clearly in one look” for understanding tasks.
- Coarse read first, then fine read: First judge at low resolution whether a close look is needed, then locally zoom into key regions.
- Control video tokens: Video tokens ≈ “tokens per frame × sampled frame count,” and the sampling frame rate is an easily overlooked cost amplifier.
- Concatenate with placeholders: Represent images in the prompt with placeholders and annotate their source or order, helping the model establish image-text correspondence.
8. Engineering Practice Checklist
When building multimodal applications, you can refer to the following checklist:
- Preprocessing alignment: Use the same image processor for training and inference, ensuring consistent scaling strategy, padding method, and patch merge logic.
- Aspect ratio fidelity: Dynamic grids should prioritize preserving the original aspect ratio to avoid text distortion from forced stretching; extra-long or extra-wide images can be tiled first and then stitched.
- Position consistency check: Verify that the vision-side flattening order matches the position numbering; before launch, run regression tests with “vertical-structure” samples like tables and flowcharts.
- Memory planning: Memory can be estimated from four categories:
- Vision encoder weights: about parameters × precision bytes;
- Language model KV cache: related to sequence length, batch size, and number of model layers;
- In-batch image tensors: about batch size × peak visual tokens × hidden dimension × bytes;
- Variable-length attention buffers: including cu_seqlens, indices, masks, or temporary activations.
- Cache friendliness: When the same image repeatedly appears across interaction rounds, reuse the vision encoding result to reduce repeated computation.
- Fallback for effectiveness: For text-dense scenarios, add recognition checks; for structured output requirements, have the model provide coordinates or indices for programmatic verification.
- Observable cost: Record the ratio of visual tokens to text tokens for each request to locate the real cost source.
- Cloud deployment: Multimodal capabilities are typically provided in the cloud as API services, and can be managed with object storage, container-based inference platforms, and logging and monitoring services for images, inference requests, and token consumption.
9. Summary: The Vision Encoder Is “Light,” but Visual Capability Can Be “Heavy”
Through the analysis of a 24-layer, ~0.56B-parameter lightweight vision encoder, we can summarize several core principles:
- Native integration beats bolt-on assembly: Only when vision and language are processed uniformly in the same sequence can the model decide on its own “when to observe.”
- Dynamic resolution beats fixed scaling: Zero-padding alignment, 14-pixel patches, 2×2 merge, and token budget control let inputs of any size enter the model efficiently.
- Lightweight encoder with high-precision inference: The vision tower’s parameter share is small; quantization benefits are limited while risk is high, so fixed BF16 is a prudent choice.
- Token budget is the biggest cost lever: Allocating budget by “coarse read + fine read” is more effective than simply piling up parameters.
The “light” of the vision encoder is reflected in its parameter and compute overhead; the “heavy” of visual capability is reflected in the closed-loop value it connects across observation, reasoning, and generation. For developers, understanding this mechanism means being able to make more informed tradeoffs among effectiveness, cost, and latency.
