Multimodal Vision Encoders: How Image Encoding Actually Works Inside a Vision-Language Model
The vision encoder is often reduced to "turning an image into a vector," but its real job is broader: deciding how much visual information a model gets, and whether it needs to look again. This article breaks down the design of a lightweight vision tower from a class of flagship multimodal models — a 0.56B backbone under 1% of total parameters, fixed BF16 inference, 24 layers, and a 1024-to-4096 output projection — and walks through the mechanics behind it: preprocessing and zero-padding to multiples of 28, 14×14 patchification with 2×2 spatial merging, the 16–8000 visual token budget, why the backbone stays small while the language model stays large, and how inference handles variable-length packing with cumulative sequence lengths and grid-based positional encoding. Practical guidance on matching resolution and token budgets to task type, scaling vision encoding independently, and building observability around token counts and encode latency rounds out the piece.
1. What a Vision Encoder Is Actually Responsible For: Not “Looking,” but “Deciding When to Look”
Multimodal model discussions tend to give all the attention to the language backbone, while the vision side gets flattened into a single action: “encode the image into vectors.” That framing undersells what the vision encoder actually does.
A well-designed vision pipeline has to answer three questions at once. First, how do you convert images and video frames of arbitrary resolution into a discrete sequence that the language model can read, without losing information? Second, how much information should a single image be allocated within a limited context window? Third, does the model need to “take another look” during inference — that is, issue a tool call or use visual feedback to make a second observation and correct a mistake in its first reading?
The third point matters most. The value of vision encoding doesn’t stop at mapping pixels to features; it extends vision into the task boundary “beyond encoding” — giving the model the ability to judge whether the information it has is sufficient, and to actively acquire new images when it isn’t. The vision module therefore has to be native to the architecture rather than a bolt-on component that could be removed at will. Only with native integration can the model learn the full decision chain: when to observe, which part to observe, and how the observation changes the next action.
From an engineering standpoint, all of these goals land on a fairly small vision encoder. Public information on one class of lightweight flagship multimodal models describes a vision tower roughly like this:
| Item | Typical value |
|---|---|
| Depth | 24 |
| Hidden size | 1024 |
| Attention heads | 16 |
| Head dimension | 64 (1024 / 16, no grouped-query attention) |
| Patch size / merge ratio | 14 / 2×2 |
| Output dimension | 4096 (aligned with the language model hidden size) |
| Backbone parameters | ~0.56B (under 1% of the total) |
| Inference precision | Fixed BF16 |
2. The Preprocessing Pipeline: Resolution Alignment and Patchification
Before an image enters the vision tower, it passes through a standardized image processor that resizes, zero-pads, and patches it. This step looks mundane, but it directly determines memory footprint and how stable the sequence length is.
2.1 Why Zero-Padding Rounds to Multiples of 28
Preprocessing resizes the image to a target size and zero-pads the width and height up to a multiple of 28. That 28 is not arbitrary — it is 14 × 2, where 14 is the pixel edge of a single patch and 2 is the spatial merge ratio.
Rounding up to a multiple of 28 removes runtime branches. If arbitrary sizes were allowed, patching would have to deal with incomplete patches in the last row and last column: more complex code, and sequence lengths that swing wildly between requests, which in turn hurts batching efficiency noticeably. Trading a small amount of zero-padding for well-formed shapes is a standard compromise in inference serving.
2.2 Patching and 2×2 Spatial Merging
The image is split into base patches of 14×14 pixels, and each patch is projected into a visual feature by a convolution layer. A 2×2 merge then combines four neighboring base patches into one unit, so each 28×28 pixel region ends up corresponding to a single visual token.
This step is the core lever between resolution and cost. Without merging, the visual token count would be four times higher: a 1024×1024 image yields roughly 5329 patches under 14×14 patching, dropping to about 1332 after merging. Merging aggregates information across a 2×2 neighborhood and compresses the sequence to a quarter of its length at almost no cost to layout information, while giving each token a more complete local semantics.
2.3 Aligning the Sequence with Text Tokens
The output of vision encoding has to live in the same vector space as the text embeddings before the language backbone can consume it directly. That requires the vision tower’s output dimension to match the language model’s hidden size exactly (4096 in the example above), with a modality projection applied before concatenation. Once aligned, visual tokens are equivalent to ordinary text tokens inside attention, so positional encoding, attention masks, and parallel sharding strategies can all reuse the language-side implementation — no separate vision path is needed.
3. Token Budgets: Turning Visual Information Into a Tunable Resource
The number of visual tokens a single image can produce is normally bounded by two parameters, with a default range of roughly 16 to 8000. These bounds aren’t arbitrary heuristics; they defend against two distinct failure modes.
The lower bound prevents information collapse. If an image containing dense text or a complex chart is encoded into only a dozen tokens, the details are flattened away and the model can only produce a generic description.
The upper bound prevents runaway budgets. An unconstrained high-resolution scanned document can generate tens of thousands of visual tokens, easily filling the context window and forcing conversation history and instructions to be truncated. Letting the caller set the bounds explicitly per scenario turns “information density” from an internal model constant into a configurable API parameter: raise the upper bound for document understanding to preserve text detail, lower it for scene description and visual question answering in exchange for throughput and latency.
This yields a practical tuning method: when a model answers questions about image detail inaccurately, first check whether the visual token budget is being capped by too low an upper bound, rather than rushing to swap the model.
4. Design Trade-offs in the Vision Backbone
4.1 Small Parameter Count, Broad Coverage
A 0.56B vision backbone accounts for under 1% of the overall model, and this “small vision tower plus large language backbone” division of labor has become mainstream. The reason is that the two sides have different computational characters. The language side must carry a large body of knowledge and reasoning ability; the vision side’s main job is to convert a continuous pixel signal into a structured discrete representation, which makes it fundamentally a high-throughput feature extractor. The ceiling on a vision encoder’s capability is determined more by its resolution strategy, data quality, and training approach than by parameter count.
So the marginal return on adding vision tower parameters decays quickly, while inference cost grows linearly. Leaving the budget to the language backbone is the more sensible allocation.
4.2 Fixed BF16 Instead of Low-Precision Quantization
The vision tower typically runs inference in BF16 and does not take a low-precision path such as FP8. The reasoning is straightforward: vision encoding sits at the front of the whole chain, so its errors accumulate layer by layer through residuals and attention and directly contaminate the language backbone’s input. The vision tower itself is a small share of total latency, so quantization buys limited speedup while the precision loss propagates into every downstream generation result. Between “save a little compute” and “protect input quality,” the latter is clearly the right trade.
5. Inference Implementation: Variable-Length Packing and Positional Encoding
The core difficulty on the vision side is that request shapes are highly irregular: different images produce different grid sizes, and video adds a time dimension on top. A schematic forward pass looks like this:
def forward(self, hidden_states, grid_thw, **kwargs):# 1) Reconstruct 2D/3D position indices from the grid dimensionsposition_ids = build_vision_position_ids(grid_thw, self.merge_size)# 2) Compute variable-length attention bounds: per-sample start offsets and max sequence lengthcu_seqlens, max_seqlen = build_attention_seqlens(grid_thw, self.config)# 3) Patch embedding: project pixel blocks into the hidden dimension (2D conv for images, plus a time axis for video)hidden_states = self.patch_embed(hidden_states)# 4) Positional encoding: inject relative positions according to the grid structurehidden_states = hidden_states + self.rope(hidden_states, position_ids)# 5) 24 transformer blocks, with attention computed inside each samplefor layer in self.layers:hidden_states = layer(hidden_states, cu_seqlens, max_seqlen)# 6) 2x2 spatial merge plus projection to the language model widthreturn self.merger(hidden_states) # 1024 -> 4096
Two implementation details are worth emphasizing.
The first is variable-length packed attention. Multiple images of different sizes are packed into one continuous sequence, with a cumulative sequence length array (cu_seqlens) marking where each sample begins and ends, so attention is strictly confined within a single image. This avoids cross-image information leakage and avoids the wasted computation that padding would introduce. It is the main reason the vision side achieves better throughput than a naive implementation.
The second is grid-based positional encoding. The positional relationships among visual tokens can’t simply be treated as a one-dimensional text sequence. Position indices have to be constructed from the rows and columns of the patch grid (and the time axis for video) so the model can correctly perceive “above, below, left, right” — which is what supports tasks that depend on spatial structure, such as reading charts, analyzing layout, and localizing objects.
6. What the Design Trade-offs Add Up To
- Shape alignment buys simpler scheduling: pad to a multiple of the patch size times the merge ratio, which eliminates tail-handling branches and keeps batch shapes stable.
- Spatial merging buys sequence compression: 2×2 merging delivers a 4× reduction at very low cost and is the precondition for scaling resolution.
- Explicit budgets buy controllable cost: make the lower and upper bounds on visual tokens tunable parameters rather than implicit constants.
- Dimension alignment buys path reuse: keeping the vision output identical to the language hidden size lets you reuse attention and sharding strategies in full.
- A precision floor buys reliable input: keeping the vision tower in BF16 prevents error from propagating into the language backbone.
- Packed attention buys throughput: variable-length batching is a necessary engineering technique for serving vision.
7. Deployment Recommendations
Three areas are worth attention when deploying multimodal capabilities like these in practice.
The first is matching resolution to budget. Set the lower and upper bounds on visual tokens based on task type rather than defaults: document tasks lean toward preserving detail, while general question answering leans toward throughput.
The second is how you scale the service. Vision encoding is compute-intensive but stateless, which makes it a good candidate for independent scaling. Running it with elastic container scaling plus load-balanced traffic distribution decouples the resource needs of encoding from those of language generation, so the two stop competing for the same capacity at peak times.
The third is observability. Bring visual token counts, encode latency, resolution distribution, and retry rates into your logging and monitoring/alerting systems; only then can you tell whether an inaccurate answer comes from an insufficient budget, too low a resolution, or the model’s own capability. If you host this kind of service on Baidu AI Cloud, these pieces can be composed from general-purpose building blocks such as a container platform, a logging service, and monitoring and alerting, while preprocessing and budget policies on the model side still need to be calibrated by your team per scenario.
