Multimodal Vision Encoding: Image Tiling and Visual Token Budget Control
This article explains how multimodal models convert images into visual token sequences. It covers image normalization, patch extraction, 2×2 feature merging, token count estimation, budget-based resizing, vision encoder architecture, inference implementation, and key engineering checks for reliable deployment.
1. Why Visual Input Must Become Part of the Model
A common approach in the past was to use a separate image recognition module to convert an image into a text description and then pass that description to a language model. However, this process can lose important information, including layout, color, small text, and spatial relationships. In code understanding, interface operation, and chart analysis, a model may need to revisit the image multiple times during reasoning and decide what to do next based on what it sees.
For this reason, an increasing number of multimodal systems place the vision encoder and language model within the same training framework. The image is encoded directly into a sequence of vectors that the language model can process. The model can then determine when it needs to inspect the image and which parts are relevant. Visual input becomes a formal component of the inference pipeline rather than an external preprocessing step.
2. Image Normalization: Resizing and Zero Padding
Before an image enters the vision encoder, it usually goes through a standardized preprocessing pipeline with two main steps.
2.1 Resizing
Images can have widely different resolutions. Preprocessing typically scales an image proportionally so that its dimensions remain within a reasonable pixel range. This preserves sufficient detail without allowing computational costs to grow uncontrollably.
2.2 Zero Padding to a Fixed Multiple
Vision encoders generally require the image height and width to be integer multiples of a base unit. If an image does not meet this requirement, zero-valued pixels are added to the right or bottom edges.
These padding pixels do not contain meaningful information, but they ensure that later splitting and merging operations do not require special boundary handling.
The base unit is determined by the encoder architecture. In a common configuration, the patch size is 14 pixels and the spatial merge window is 2×2. As a result, one visual token represents a region of:
14 × 2 = 28 pixels
Both the image height and width must therefore be padded to multiples of 28.
3. Patch Extraction and 2×2 Merging
After padding, the image is divided into non-overlapping square patches. For example, with 14×14-pixel patches, an image measuring 840×560 pixels produces:
60 × 40 = 2,400 patches
Each patch is transformed through a linear projection into a high-dimensional vector. Positional information is then added before the sequence is passed through Transformer layers.
A sequence of 2,400 patches can be expensive for a language model to process. Vision encoders therefore commonly perform spatial merging at the output side. Four adjacent patch features in a 2×2 window are concatenated and then compressed into a single vector through a projection layer.
As a result:
2,400 patches → 600 vectors
The sequence length is reduced to one quarter of its original size, while local information is retained through feature concatenation.
This design reflects a general trade-off. Smaller patches preserve more visual detail but increase computational cost. Spatial merging is a common way to balance detail against sequence length.
4. Visual Token Calculation and Budget Control
For an image whose dimensions have already been padded to multiples of 28, the number of visual tokens can be calculated directly:
Visual tokens = (height ÷ 28) × (width ÷ 28)
The 840×560 example therefore produces:
30 × 20 = 600 tokens
In real applications, context budgets can vary significantly between requests. One request may only need a small image for confirmation, while another may need to analyze a long screenshot containing a large amount of text.
Systems therefore commonly define minimum and maximum token limits for each image. For example, a configuration might use a range of 16 to 8,000 tokens:
- The minimum ensures that very small images still provide enough visual information.
- The maximum prevents an oversized image from consuming the entire context window.
When an image exceeds the maximum token count, its width and height are scaled down proportionally. When the token count is below the minimum, the image is enlarged as needed. The following example demonstrates this logic:
import mathPATCH = 14MERGE = 2UNIT = PATCH * MERGE # 28 pixels correspond to one visual tokendef plan_image(h: int, w: int, min_tokens: int = 16, max_tokens: int = 8000):"""Return target height, width, and visual token count."""grid_h = math.ceil(h / UNIT)grid_w = math.ceil(w / UNIT)tokens = grid_h * grid_wif tokens > max_tokens:s = math.sqrt(max_tokens / tokens)grid_h = max(1, math.floor(grid_h * s))grid_w = max(1, math.floor(grid_w * s))elif tokens < min_tokens:s = math.sqrt(min_tokens / tokens)grid_h = math.ceil(grid_h * s)grid_w = math.ceil(grid_w * s)tokens = grid_h * grid_wreturn grid_h * UNIT, grid_w * UNIT, tokensprint(plan_image(560, 840)) # (560, 840, 600)
With this approach, the final image dimensions and token count are determined before the image is sent to the encoder. The calling system can then estimate the cost of each request more accurately.
5. Vision Encoder Architecture and Parameter Scale
A vision encoder is typically built as a standard stack of Transformer layers. Its main components include:
- A patch embedding layer that maps image patches to vectors;
- Positional encoding that preserves two-dimensional spatial relationships;
- Multiple Transformer layers, each containing multi-head self-attention and a feed-forward network;
- A merging and projection layer that aligns the output dimension with the language model’s hidden size.
A general formula can be used to estimate the parameter scale. For a standard Transformer layer with hidden dimension d and a feed-forward expansion ratio of 4, the parameter count per layer is approximately:
12 · d²
For L layers, the total is approximately:
12 · L · d²
For example, with L = 24 and d = 1024, the main backbone contains approximately 300 million parameters. The actual size may vary depending on the feed-forward expansion ratio, whether grouped attention is used, and other architectural choices. However, this formula is sufficient for estimating the order of magnitude.
The vision encoder is usually much smaller than the language model. During deployment, its computation is therefore often not the primary bottleneck. Most memory and latency pressure still comes from the language model.
For inference precision, vision encoders commonly use half-precision formats such as BF16 to balance numerical stability and speed. Whether to apply lower-bit quantization should be determined through accuracy testing.
6. Key Inference Implementation Considerations
The main engineering concerns during inference include the following.
6.1 Constructing Positional Information
Each visual token needs two-dimensional coordinate information. When a request contains multiple images, position indices should be generated independently for each image so that coordinates from different images do not interfere with one another.
6.2 Batching Variable-Length Sequences
Images with different dimensions produce different numbers of tokens. A common approach is to flatten all images in a batch into one long sequence and use a cumulative-length array to record the boundary of each image. Attention can then be restricted to tokens belonging to the same image.
6.3 Concatenating with the Language Model Input
The visual vectors produced by the encoder replace reserved image placeholder positions in the input sequence. The number of placeholders must exactly match the number of visual tokens produced by the encoder. Otherwise, the sequence will become misaligned. This is one of the most common integration errors.
6.4 Video Input
A video can be treated as a sequence of frames sampled over time. A common method is to sample frames at a fixed rate, apply the image processing pipeline to each frame, and then adjust the sampling density according to the video duration and token budget.
Adjacent frames often contain substantial redundancy. Merging features across neighboring time steps is an effective way to reduce the total number of tokens.
7. Engineering Checklist
Before integrating visual input into a production system, verify the following items:
- Does preprocessing use the same base unit as the vision encoder, especially for padding?
- Are the minimum and maximum token budgets appropriate for the application’s context length and concurrency requirements?
- Do unit tests verify that the number of placeholders matches the encoder output length?
- During batching, is attention correctly isolated between different images?
- Is there a defined resizing strategy for oversized images, with resizing decisions recorded in logs to help diagnose reduced recognition quality?
- Have precision and memory usage been measured on the target hardware rather than estimated only from theory?
The cost of visual tokens can be calculated precisely when preprocessing rules are clearly defined. Encoding these rules in configuration and enforcing them in code is the foundation for stable multimodal deployments.
