Understanding Lightweight Vision Encoder Design in Multimodal Large Models
This article breaks down how the vision side of a modern multimodal model is built around a lightweight vision encoder, and why it matters more than its small parameter share suggests. It walks through the full pipeline—from image normalization and patch splitting to 2×2 merging, positional encoding, variable-length attention, and projection into the language model's space—and explains the reasoning behind a typical configuration of 24 layers, a 1024 hidden size, patch size 14, and 2×2 merging. It then covers key design trade-offs, engineering practices such as keeping BF16 precision on the encoder, variable-length packing, continuous batching, chunked prefill, and preprocessing parallelism, plus deployment patterns and a three-layer validation approach. Readers will come away with a clear picture of how to turn images and video of arbitrary resolution into a controllable visual-token budget while balancing accuracy and cost.
1. Why the Vision Encoder Deserves Its Own Analysis
Today’s open-source multimodal models generally adopt a three-part architecture: a lightweight vision encoder, a large language model (LLM) backbone, and a connector layer. Recent cost-effective releases follow the same pattern. The core idea is to pay a relatively small cost on the vision side in exchange for strong image-text understanding, document parsing, and even code-related capabilities.
This article focuses on the vision side of a multimodal model—the part that determines what the model can “see,” how clearly it sees it, and how expensive that is. The encoder sits at the data entry point, and its patch granularity, token count, positional encoding scheme, and numerical precision directly determine the quality of visual information the downstream language model receives. Its inference cost also sits on the critical path: every request must pass through vision encoding first, before text generation can begin.
The value of vision in a multimodal model goes beyond “recognizing image content.” Many early industry solutions treated the vision model as a bolted-on module: first use OCR or a detection model to extract text and bounding boxes, then hand the results to the language model. This kind of pipeline has two inherent flaws:
- Misaligned objectives: The vision model’s optimization goals (character accuracy, detection mAP) do not match the language model’s task goals (whether an answer is correct, whether code passes), and errors accumulate stage by stage;
- No feedback loop: Once the text stage goes wrong, the vision stage cannot step in to correct it.
Mainstream models therefore integrate vision natively into the main network, allowing the model to decide on its own during generation “when it needs to look,” and to use visual feedback to guide subsequent actions. In intelligent coding tools, this is especially clear:
- Given a UI design mockup, the model first parses the layout and control structure, then generates code;
- After running, it compares a screenshot against the design; if the rendered result differs, it locates the discrepancy and fixes the styling;
- When faced with an error screenshot, it reads the stack trace and applies a targeted fix.
Vision is no longer a terminal “recognition” step, but an “observation” step at the start of and inside the loop.
However, native integration does not mean “bigger is always better.” The key to vision-side design is finding the right balance between accuracy and cost, and the analysis below revolves around this main thread.
2. How Vision Encoding Works: From Pixel Matrices to Language Model Tokens
2.1 The Overall Data Flow: Four Steps from Image to Visual Tokens
An encoder’s job can be summarized in one sentence: compress an image or video of arbitrary resolution into a token sequence the language model can read, while preserving semantic and spatial structure. The full chain usually has four steps.
- Image normalization: scaling, channel conversion, and zero-padding to a unified 3-channel input;
- Splitting and merging: cutting the pixel matrix into small patches as the basic unit, then merging adjacent patches into visual tokens;
- ViT backbone encoding: using multiple Transformer layers to perform cross-patch interaction for each patch, producing a visual representation with global context;
- Projection alignment: mapping the visual representation into the language model’s hidden space, then concatenating it with text tokens before feeding it to the language model.
Among these, steps 1 and 2 determine how many tokens a single image consumes, directly affecting the context budget; step 3 determines the semantic quality of the visual representation; and step 4 determines whether visual information and text can interact effectively within the same space.
2.2 Image Normalization: Why Align to Multiples of 28
The mainstream approach uses a 3D convolution layer to do the patching, i.e., patch embedding. This layer’s kernel size equals the patch size (14×14), with the same stride, an input of 3 RGB channels, and a stride of 1 along the temporal dimension. At this point, each image or video segment is treated as “1 frame,” making 3D convolution equivalent to 2D convolution. The benefit is that images and video can share the same set of parameters without extra training.
A 3D convolution requires both height and width to be divisible by the patch size, so a multimodal processor will typically align to a multiple of 28 directly during normalization. The reason lies in the vision-side merge ratio:
28 = patch size (14) × merge ratio (2)
The resolution algorithm in the normalization stage can be implemented like this:
PATCH, MERGE = 14, 2ALIGN = PATCH * MERGE # 28def smart_resize(h, w, max_pixels, min_pixels):"""Constrain the resolution to [min_pixels, max_pixels] while preservingthe aspect ratio, and align both sides to a multiple of 28."""def align(x):return max(ALIGN, round(x / ALIGN) * ALIGN) # round to the nearest multiple of 28ratio = w / h# Derive new width/height from the target pixel count, then align each sideh_new = align((min(max_pixels / ratio, max(min_pixels * ratio, h))) ** 0.5)w_new = align(h_new * ratio)# Secondary calibration: rounding after alignment may push the area out of range,# so scale down proportionally and align againif h_new * w_new > max_pixels:s = (max_pixels / (h_new * w_new)) ** 0.5h_new, w_new = align(h_new * s), align(w_new * s)if h_new * w_new < min_pixels:beta = min(w / h_new, h / w_new)h_new, w_new = align(beta * h_new), align(beta * w_new)return h_new, w_new
Reading it line by line:
- Round to nearest: the error is at most about 14 pixels, which is better than always rounding up.
- Proportional scaling during secondary calibration: shrinking only one side would change the aspect ratio, and aspect distortion is especially harmful for text and tables.
- Lower-bound calibration: ensures that low-resolution inputs still carry a meaningful amount of information.
“Padding” is just another optional implementation. A few processors preserve the original aspect ratio, pad with zeros up to a multiple of 28, and then use an attention mask so the padded blank pixels do not participate in computation. This avoids both aspect distortion and wasted tokens.
Compared with early schemes that required “scaling to a fixed size first,” a dynamic resolution strategy has clear advantages:
- For document screenshots, long web pages, and code screenshots, it retains more detail, which is friendlier to small fonts and table lines;
- For images with low information density, it does not waste tokens;
- It avoids the aspect distortion that fixed sizes introduce.
2.3 Splitting and 2×2 Merging: Carrying More Information with Fewer Tokens
After normalization, the pixel matrix is cut into 14×14 patches, and each patch is flattened into a 588-dimensional vector (3×14×14). Patch embedding then maps this to 1024 dimensions, producing a 1D sequence where each position corresponds to one patch.
Next comes 2×2 merging: each adjacent 2×2 group of base patches is concatenated and then reduced in dimension through a linear mapping, so four positions become one token. This yields the core formula for token count:
Visual tokens ≈ (H / 28) × (W / 28) # single frameTotal sequence length ≈ Σ_frames (H / 28) × (W / 28) # video / multiple images
Using the common default budget of 16–8000 visual tokens (estimated at 28×28 pixels per token):
| Visual token budget | Approx. pixels | Approx. equivalent resolution |
|---|---|---|
| 16 (lower bound) | 13K | 112×112 |
| 1024 | 800K | 896×896 |
| 4096 | 3.21M | 1792×1792 |
| 8000 (upper bound) | 6.27M | 2496×2496 |
2×2 merging brings three benefits:
- Length compression: the sequence length drops to 1/4. Since attention cost grows roughly with the square of sequence length, this cuts the encoder’s attention computation to about 1/16.
- Preserved granularity: adjacent patches are four separate positions during encoding, so attention can model them at fine granularity; merging happens after encoding, so information is not flattened prematurely.
- More natural dimension alignment: when the merged hidden dimension doubles (e.g., 512→1024), it can align with the backbone’s hidden size; the total dimension after merging equals the total before merging, and positional encoding can also be handled uniformly over the merged grid.
It is worth emphasizing that merging only saves the cost of the vision-encoding stage. Every visual token eventually enters the LLM backbone, participates in layer-by-layer self-attention alongside text, and occupies KV cache. For scenarios like very long documents or tens-of-seconds videos, the vast majority of input tokens come from the vision side, and the language model’s prefill is the real bottleneck. Therefore, the visual token budget is a global resource-planning decision, not a local parameter internal to the encoder.
2.4 Positional Encoding: Letting the Model Know Where Each Patch Is
Transformers are inherently position-agnostic and need explicit positional information. Multimodal scenarios have two special requirements:
- 2D spatial structure: an image is a 2D grid, so the position should encode both row and column; video also needs a temporal dimension.
- Arbitrary resolution: input sizes are not fixed, so the positional representation must expand dynamically. It cannot rely on interpolating a fixed-size positional embedding table trained at a specific resolution, as early ViTs did.
A typical approach builds a learnable 2D (or 3D) positional embedding table and looks up the corresponding vector for each patch by “frame–row–column.” During preprocessing, metadata such as positional indices (position_ids) is generated and cached alongside the image, so it need not be recomputed at inference time. This is also the prerequisite for reusing the same image across multi-turn conversations.
2.5 Variable-Length Attention: Packing Multi-Image, Multi-Resolution Samples
The number of visual tokens varies enormously across multimodal requests, from a dozen or so to several thousand. If you pad to the longest sequence in the batch and then compute attention, the many blank positions in short samples are computed needlessly, and they must be masked out—otherwise the results are polluted.
A more efficient implementation concatenates all visual tokens in the batch into a single 1D sequence, uses a cumulative sequence-length array (cu_seqlens) to record each sample’s boundaries, and pairs this with an attention operator that supports variable-length inputs. Attention computation is split into several independent segments, and segments cannot see one another, which both eliminates padding and naturally guarantees sample isolation.
The code below shows that the vision backbone’s forward pass follows a fixed pattern: derive positional indices and attention-segment boundaries from the grid_thw metadata, perform patch embedding, encode layer by layer, and finally project the output. Different models mainly differ in the number of layers, dimensions, and merging scheme.
def forward(self, pixel_values, grid_thw, **kwargs):# grid_thw: [num_samples, 3]; each sample is (num_frames, patches_along_height, patches_along_width)# pixel_values: flattened patch sequence, concatenated head-to-tail across samples,# each patch being 3×14×14 = 588 values# ① Generate positional indices and attention-segment boundaries from the grid metadata# (usually precomputed during image preprocessing and cached)position_ids = get_vision_position_ids(grid_thw, self.spatial_merge_size)cu_seqlens, max_seqlen = get_vision_attention_seqlens(grid_thw, self.config)# ② Conv3d patching: 588 → hidden_size, then flatten to [num_tokens, hidden_size]hidden_states = self.patch_embed(pixel_values)# ③ Add the positional embedding looked up by positional indexhidden_states = hidden_states + self.pos_embed(position_ids)# ④ Encode layer by layer: variable-length attention + FFN, with residual connectionsfor block in self.blocks:hidden_states = block(hidden_states, cu_seqlens, max_seqlen)# ⑤ 2×2 spatial merge (concatenate every 4 adjacent patches through a linear layer),# then split back into a list per sample# ⑥ Project to the language model's hidden dimension, concatenate with text tokens,# and feed into the language modelreturn self.merger(hidden_states, grid_thw)
2.6 Projection Alignment: From the Vision Hidden Space to the Language Model Space
The vision encoder’s hidden dimension is usually much smaller than the language model’s. Take a configuration of 24 layers with hidden size 1024; a rough parameter estimate is:
- Per-layer attention (the four projections Q, K, V, O) is about 4×1024²;
- Per-layer FFN (two linear layers, with a 4× expansion in the middle) is about 8×1024²;
- Total per layer is about 12.58M, so 24 layers come to about 300M;
- Adding patch embedding, positional embedding, the merging layer, and the output projection, the total parameter count is about 320M.
This magnitude is consistent with the published backbone parameter count for this kind of lightweight configuration. If the language model backbone is on the order of tens of billions of parameters, the vision side accounts for less than 1%.
At the end of the model, a “merge + project” module (the merger) does two things:
- Concatenation: 2×2 merging, which enlarges the hidden dimension by 4×;
- Linear projection: mapping to the language model’s hidden dimension to align with text tokens.
Some schemes also keep a two-layer MLP as an adapter. After alignment, visual tokens and text tokens are concatenated uniformly and fed into the language model. This “light encoder + lightweight connector” design lets the language model receive visual input in essentially the same way it receives text input, which helps transfer language capabilities to image-text tasks.
3. Key Hyperparameters and Design Trade-offs
The table below summarizes a typical configuration for this kind of lightweight vision encoder and explains the reasoning behind each choice:
| Item | Typical value | Design consideration |
|---|---|---|
| Depth (layers) | 24 | Vision tasks need less depth than language; each layer is about 12.58M params, and doubling the layers would directly raise parameters, memory, and latency |
| Hidden size | 1024 | Divides evenly by the number of attention heads; too large would increase parameters and memory-access cost |
| Attention heads | 16 | Per-head dimension of 64 (1024/16), close to a common efficient attention configuration |
| GQA | Not enabled | During decoding, KV cache grows linearly with history, and grouped-query attention can reduce cache; vision encoding uses bidirectional attention with no cross-token decoding, and the KV cache is released as soon as the sample ends, so the benefit is minimal |
| Patch / merge | 14 / 2×2 | Finer patches mean more positions and costlier attention; a higher merge ratio means greater detail loss |
| Output dimension | 4096 | Aligns with the language model’s hidden size, reducing connector parameters |
| Backbone params | ≈0.56B | Under 1% of the whole; but every visual token enters the backbone and participates in computation, so the actual cost should not be underestimated on this basis |
| Inference precision | Fixed BF16 | The vision side acts as the precision entry point; unified high precision reduces points where error is introduced (see 4.1) |
A depth of 24 is usually enough to cover multi-level visual features such as edges, textures, characters, layout, and semantic objects; going deeper yields diminishing returns while latency rises linearly. A hidden size of 1024 with 16 heads and a per-head dimension of 64 matches the per-head dimension commonly used in Transformer decoders, so mature efficient attention kernels can be reused, avoiding kernel degradation or extra blocking from oversized head dimensions.
This also reflects several core advantages of a lightweight vision encoder:
- Controllable high resolution: dynamic resolution plus a merging mechanism means high resolution only costs something on the samples that need it;
- Native integration: vision and text are aligned in a unified representation space, so the model can decide on its own during generation when to “look”;
- Balanced accuracy and throughput: the encoder keeps high precision while the backbone is quantized as needed;
- Decoupling from the backbone: the vision side can iterate and upgrade on its own without retraining the language model.
4. Engineering Implementation Points
4.1 Inference Precision: Why Stick with BF16 and Avoid Blind FP8
FP8 compresses weights and activations to 8 bits, with the main gains coming from doubled matmul throughput and halved memory bandwidth. It is usually applied to large, high-parameter LLM backbones. Whether the vision encoder should follow suit with FP8 requires weighing several factors:
- Limited benefit: the encoder accounts for less than 1% of parameters, and per-request computation is far smaller than autoregressive decoding, so the time quantization can save is limited.
- Higher risk: the vision side sits at the data entry point, and errors accumulate layer by layer through residual connections and LayerNorm, then get amplified all the way to the output through the language model’s attention. Fine lines and small fonts in documents, screenshots, and code interfaces are most likely to “blur” at low bit widths.
- Uncertain benefit: the gains from FP8 on small matrices also depend on hardware and kernel support. Some hardware’s FP8 only favors specific shapes, so introducing FP8 for small matrices may not pay off.
A pragmatic strategy, therefore, is:
- Fix the encoder at BF16;
- Quantize the language model backbone to lower bits as needed, evaluating the impact on task metrics layer by layer.
Mixed precision should be decided by measurement. It is recommended to run the full pipeline with both BF16 and FP8 on the same evaluation set, and to base the trade-off on changes in task metrics rather than indirect indicators like perplexity.
4.2 Scheduling and Batching: Reducing Idle Time and Waste
- Variable-length packing: use
cu_seqlensand variable-length attention instead of padding, which both saves compute and guarantees sample isolation. - Continuous batching: vision encoding usually takes only tens of milliseconds, while the language model’s prefill and per-token decoding last hundreds of milliseconds or more. The earlier the encoder finishes, the sooner the GPU enters the main task; the inference engine should support flexible batching across different requests.
- Chunked prefill for long inputs: very long documents or long videos can be encoded in chunks and prefilled in segments, lowering peak memory and reducing head-of-line blocking.
4.3 Preprocessing and Inference Pipeline: Hiding CPU Cost Behind the GPU
In multimodal serving, preprocessing—decoding, scaling, patching, and building positional indices—consumes substantial CPU, and this becomes more pronounced as resolution rises. If preprocessing and inference run serially, the GPU will visibly idle.
Practical measures:
- Run preprocessing processes or a thread pool in parallel with the inference process;
- Cache preprocessing artifacts (
pixel_values,grid_thw, positional indices) along with the corresponding text token sequences; - When the same image recurs across multiple turns of conversation, hit the cache directly;
- In video scenarios, reuse scaling results per frame.
4.4 Deployment Topology: Handling Elasticity and Observability with General Cloud Capabilities
Vision encoding and the language model have different resource profiles: the former is compute-intensive with short single-call latency; the latter is memory-heavy and long-running. In a monolithic deployment, the two can only share the same pool of resources. A more refined approach splits them at the service level:
- Vision encoding runs as a standalone preprocessing service that can scale flexibly by CPU/GPU ratio;
- The language model service focuses on decoding, making it easier to optimize quantization, KV cache, and batching independently.
When building this on the cloud, it can be assembled from general capabilities:
| Capability category | Role in the multimodal inference chain |
|---|---|
| Object storage | Storing weights, images and video assets, and evaluation sets |
| Container platform | Hosting preprocessing and inference services, scaling elastically with load |
| Message queue | Smoothing traffic spikes and asynchronously handling batch document and video parsing |
| Logging service | Recording request latency, token usage, and error codes to pinpoint bottlenecks by stage |
| Monitoring and alerting | Setting threshold alerts on GPU utilization, memory usage, and queue latency |
Baidu AI Cloud provides cloud services across these general categories, which teams can combine as needed to build a multimodal inference chain covering preprocessing, inference, and operations.
5. Quality Evaluation and Validation
After any vision-side change, validation is recommended across three levels.
Level 1: Functional correctness
- Use the formula to compute the expected visual token count and compare it with the actual sequence length fed to the language model;
- Check that the patch order within each row after merging matches the original grid; a misaligned merge makes the model “see” a scrambled image;
- For multi-image requests, confirm that attention is correctly masked across samples.
Level 2: Numerical consistency
- For a few representative inputs (high-resolution screenshots, low-resolution icons, multi-frame videos), compare the relative error of the encoding output between BF16 and FP32, and spot-check the logits distribution of the final output tokens;
- Whenever the quantization scheme changes, be sure to repeat this check.
Level 3: Task quality
- Detail: small fonts, table lines, and icon states—measured by character recognition accuracy and table reconstruction accuracy;
- Layout: component positions and hierarchical relationships—measured by the fidelity of UI descriptions and generated code;
- Temporal: action order and state changes in multi-frame videos—measured by event-ordering accuracy;
- Long input: multi-page documents and tens-of-seconds videos—measured by recall and citation accuracy under long context.
During evaluation, record token consumption and end-to-end latency together, to avoid “accuracy going up while cost spirals out of control.”
6. Common Pitfalls and How to Avoid Them
- Text outside the image getting cropped: when batch sampling, if different videos share one set of frame and size indices, low-resolution samples may fall outside the index range. If a frame can’t be retrieved and is simply skipped, that frame’s content never enters the model. Sample according to each item’s own frame count and dimensions.
- High-resolution inputs exceeding the range: the resolution algorithm must handle both “too large” and “too small” boundaries. When the area exceeds the upper bound, scale down proportionally and align again; when the area falls below the lower bound, scale up but not beyond the original size, avoiding meaningless interpolation.
- Extreme aspect ratios: for very long web screenshots or very flat tables, after the short side is aligned to 28, the long side can easily blow past the token limit. For such samples, separately limiting the long-side pixel count or encoding in segments is more reasonable than downsampling the whole image.
- Uncertainty from dynamic resolution: the same image gets different grids and token sequences under different budgets. Fix the budget in production, or at least rerun the task evaluation on version upgrades.
- Silent information loss: scaling, truncation, frame skipping, and merging all lose information. “Observation” actions during tool calls—capturing a screen, reading logs—should follow the same resolution and budget rules.
7. Summary
Although the vision encoder in a multimodal model accounts for less than 1% of parameters, it is the key component that determines whether the model “sees clearly” and “sees affordably.” Around a lightweight configuration of 24 layers, hidden size 1024, patch size 14, and 2×2 merging, this article has laid out three categories of engineering points:
- Principles: aligning dynamic resolution to multiples of 28, patch splitting and merging, and positional indices with variable-length attention together turn images and videos of any size into visual sequences with a controllable token budget;
- Trade-offs: keeping the vision encoder at BF16 while quantizing the language model backbone as needed balances accuracy and throughput;
- Engineering: pipeline parallelism, cache reuse, continuous batching, and chunked prefill, along with general cloud capabilities such as object storage, container platforms, logging, and monitoring/alerting, build an observable and scalable inference chain.
The value of native vision integration lies in letting the model autonomously “decide when to observe” during inference and use visual feedback to guide subsequent actions. Whether the vision-side engineering is solid directly determines how reliably this closed-loop capability can be delivered.
