Visual Encoder Design in a Lightweight Flagship Multimodal Model: From Pixels to Tokens
A practical walkthrough of how the visual encoder in a lightweight flagship multimodal model turns images and video into token sequences that a large language model can consume. Using an open-source inference implementation as a reference, the article breaks down the full pipeline—image preprocessing, the 24-layer ViT backbone, spatial merging, and cross-modal projection—along with key structural parameters, engineering tradeoffs, and practical tuning advice. It explains why images are aligned to multiples of 28 pixels, how token budgets (16–8000) keep visual cost in check, why the encoder stays in BF16 instead of FP8, and how variable-length packing avoids wasted compute. It is aimed at developers who want to understand, tune, and deploy multimodal models in production.
Introduction: The Visual Encoder in a Lightweight Multimodal Model
Recently, the capabilities of large language models (LLMs) have been expanding from pure text toward multimodality. Unlike early cascaded approaches that ran OCR first and reasoned second, a new generation of multimodal models lets visual signals participate directly in reasoning: a model can look at charts, UI screenshots, or code execution results and then adjust its next step accordingly. This capability is often summarized as “visual coding.” One way to frame it: vision is not just “processing images” but expanding the boundary of what programming can reach. More importantly, vision needs to be integrated natively into the model so that it can decide on its own when to look and use visual feedback to guide subsequent actions—for example, locating layout misalignment from a frontend screenshot, or judging from a GUI screenshot whether the previous action took effect.
Native visual integration places two requirements on the model architecture: first, map images and video at arbitrary resolution stably into a token sequence that the language model can consume; second, complete this mapping at as small a compute and memory cost as possible. This article uses the open-source inference implementation of a lightweight flagship multimodal model as a sample and dissects the design of its visual encoder—from preprocessing, structural parameters, and the forward pass to engineering tradeoffs.
1. From Image to Token: The Visual Encoding Pipeline
The overall pipeline has four stages: preprocessing, visual encoding, spatial merging, and cross-modal projection. The result is concatenated with text tokens and fed into the LLM backbone.
Raw image│ ① Resize + zero-pad to a multiple of 28▼Normalized tensor│ ② Split into 14×14 patches + Conv3d embedding▼Patch sequence (one vector per 14×14 region)│ ③ 24-layer ViT + variable-length packing attention▼Encoded features (one vector per 14×14 region)│ ④ 2×2 merge (Conv2d downsampling) + SwiGLU projection▼Visual tokens (one per 28×28 region) → concatenate text tokens → LLM backbone
Understanding this pipeline starts with one question: how fine-grained is the visual information when it enters the language model? This implementation gives a clear answer: each 28×28 pixel region corresponds to one visual token.
2. Image Preprocessing: Standardization at Any Resolution
Before visual encoding, every image goes through an image processor for standardization. Its job is to organize inputs of varying sizes into a regular tensor the model can consume.
2.1 Aligning to a Multiple of 28
- Resize and zero-padding: the image is first scaled to an appropriate size, then zero-padded along width and height so that both dimensions become integer multiples of 28.
- Why align to 28: a later 2×2 merge combines 4 adjacent base patches into one visual token. Each base patch is 14×14 pixels, so a 2×2 group is exactly 28×28 pixels. Aligning during preprocessing ensures the in-network merge never crosses a boundary.
This reflects a common engineering principle: preprocessing constraints should come from the model structure itself, not from arbitrary rules of thumb.
2.2 Patch Splitting and the 2×2 Merge
During preprocessing, the model computes the visual token grid and uses patch indices to record each token’s 2D position in the image. This position information is later used to build 2D rotary position encodings.
# Preprocessing pseudocode: illustrative of the computation onlypatch_size = 14 # base patch edge lengthmerge_size = 2 # 2×2 mergeregion = patch_size * merge_size # 28: pixel edge covered by one visual token# 1) Zero-pad to a multiple of regionpad_h = (-height) % regionpad_w = (-width) % regionpadded = zero_pad(image, pad=(0, pad_h, 0, pad_w))# 2) Compute the token grid and apply a count budgetgrid_h = (height + pad_h) // regiongrid_w = (width + pad_w) // regionnum_tokens = grid_h * grid_wnum_tokens = clamp(num_tokens, min_image_tokens, max_image_tokens)# 3) Record the patch position for each visual token (used for 2D positional encoding)position_ids = build_vision_position_ids(grid_h, grid_w, merge_size)
2.3 Token Budget Control
- Parameter range: the model controls the number of tokens produced per image via
min_image_tokensandmax_image_tokens, with a default range of 16 to 8000. - Purpose: clamp the row/column counts of the token grid between the lower and upper bounds so that a single model’s visual cost stays within a controllable range.
- Significance for long-context scenarios: visual tokens share the context window with text tokens. The upper bound ensures high-resolution images don’t exhaust the context; the lower bound prevents small images from being over-compressed and losing detail.
At the lower bound of 16, the grid is roughly 4×4 at minimum, corresponding to a 112×112 pixel region—so small icons and compact charts won’t collapse into a handful of tokens. The upper bound of 8000 means the model can allocate at most 8000 positions to a single image.
Token usage at different resolutions can be estimated as follows (assuming width and height are both multiples of 28):
| Input resolution | Grid (rows × cols) | Visual tokens | Typical use case |
|---|---|---|---|
| 224×224 | 8×8 | 64 | Small icons, simple screenshots |
| 448×448 | 16×16 | 256 | UI screenshots, document snippets |
| 1024×1024 | 37×37 | 1369 | High-resolution screenshots, infographics |
| 2240×2240 | 80×80 | 6400 | Fine-grained recognition near the default cap |
Token count grows roughly with the square of pixel area: double the resolution and tokens grow about fourfold. When tuning parameters on the business side, focus on the combination of resolution and token budget rather than simply chasing “higher resolution is better.”
3. The Visual Backbone: A 0.56B, 24-Layer ViT
The visual encoder uses a standard ViT structure. All dimensions are powers of two, which keeps memory access and compute regular and efficient across different hardware. Structural parameters:
| Item | Value |
|---|---|
| Depth (layers) | 24 |
| Hidden size | 1024 |
| Attention heads | 16 |
| Per-head dimension | 64 (1024 / 16, no GQA) |
| Patch size / merge | 14×14 / 2×2 |
| Output dimension | 4096 (aligned to the LLM hidden size) |
| Backbone parameters | ≈0.56B |
A few noteworthy details:
- Simple attention head structure: per-head dimension of 64, and no grouped-query attention (GQA) on the visual side. This likely trades some attention and memory efficiency in visual encoding for simpler implementation and scheduling.
- Narrow width, moderate depth: the 24-layer, 1024-dim configuration balances “receptive field and semantic abstraction” against “parameter count and latency.”
- Very small parameter share: at 0.56B, the encoder usually accounts for less than 1% of a multimodal model with tens or even hundreds of billions of total parameters. The bulk of the parameters still lives in the language model.
This division of labor—“small visual encoder + large language model”—is consistent with native visual integration: the encoder’s job is to translate pixels efficiently into visual symbols the language model can easily understand, without carrying complex semantic reasoning on its own. Higher-level reasoning, association, and planning remain the job of the larger language model backbone.
It’s worth stressing that a low parameter share does not mean low compute cost. The encoder’s compute is roughly proportional to the number of patches introduced by the input resolution; at high resolutions its forward pass can still take up a sizable portion of inference latency. When evaluating deployment cost, parameter count and activation volume need to be considered separately.
4. The Forward Pass: Six Key Steps
The visual model’s forward pass can be summarized as follows (pseudocode):
def forward(self, hidden_states, grid_thw, **kwargs):# 1. Compute the 2D (3D for video) position index for each patchposition_ids = get_vision_position_ids(grid_thw, self.spatial_merge_size, ...)# 2. Build cumulative sequence lengths for variable-length attentioncu_seqlens, max_seqlen = get_vision_attention_seqlens(grid_thw, self.config, ...)# 3. Conv3d patchify: pixels → vectorshidden_states = self.patch_embed(hidden_states)# 4. Axial RoPE: 2D positional encodingposition_embeddings = self.rotary_pos_emb(hidden_states, position_ids)# 5. 24-layer Transformer encodingfor blk in self.blocks:hidden_states = blk(hidden_states,cu_seqlens=cu_seqlens,...)# 6. Final normalizationhidden_states = self.post_layernorm(hidden_states)# 7. 2×2 spatial merge + downsamplinghidden_states = hidden_states.view(-1, 2, 2, C)hidden_states = hidden_states.permute(0, 3, 1, 2)hidden_states = self.downsample(hidden_states).view(-1, out_hidden)# 8. SwiGLU projection to the LLM hidden dimensionmerged = self.merger(hidden_states)return merged
4.1 Conv3d Patch Embedding
Patch embedding (patch_embed) uses a 3D convolution to map the image from pixel space into vector space. Using 3D rather than 2D convolution is typically meant to let a single implementation handle both images and video: an image can be treated as a degenerate input with only one time frame, while the frames of a video share the same set of convolution weights. The same visual encoding logic can therefore serve both static screenshots and sequences of video frames.
4.2 2D Rotary Position Encoding (Axial RoPE)
Standard ViTs often use learnable absolute position encodings, with a position table fixed to the training resolution; once the input resolution changes, interpolation is required, which easily introduces error. Here the model uses rotary position encoding (RoPE), split by axis (axial): width and height (and time for video) are each encoded for rotation independently, then combined into the attention computation. This has three benefits:
- Positional representations are generated dynamically from the grid size, so any resolution needs no interpolation;
- The 2D spatial structure is modeled explicitly, making attention more sensitive to relative “up/down/left/right” relationships;
- RoPE acts on attention in the form of relative positions, and typically extrapolates more smoothly to resolutions never seen in training.
4.3 Variable-Length Packing and Contextual Attention
Real-world photos, screenshots, and scans vary widely in aspect ratio. Padding to a uniform size for batching wastes a lot of compute on meaningless padded regions. This model uses variable-length packing:
- Within one image, only spatially adjacent patches can see each other;
- Patches from different images in a batch are concatenated into one flat sequence, with boundaries marked by cumulative sequence lengths (
cu_seqlens); - Attention is computed only within each image; combined with an attention operator that supports variable-length input, this avoids cross-image contamination and padding waste.
From the signature, the visual attention module also receives attention-related configuration from config; the implementation side typically uses this to select an appropriate attention backend. Details such as whether a causal mask is enabled, dropout, and the attention implementation should be confirmed against the specific version’s implementation.
4.4 24-Layer Transformer Encoding
The patch sequence passes through 24 encoding blocks, each performing multi-head self-attention followed by a feed-forward network. Attention is masked out over zero-padded regions and does not participate in computation. Unlike a language model, the self-attention here is bidirectional: image patches have no generation order and require global context modeling, so a causal mask is typically not applied.
4.5 2×2 Spatial Merge and Conv2d Downsampling
After encoding, the features of 4 adjacent patches are merged into one visual token. From the code, the implementation is:
- Reshape the sequence so each 2×2 neighborhood becomes one sample;
- Permute the dimension order so each sample has a “channel × 2×2” spatial layout;
- Perform downsampling via Conv2d;
- Flatten back into a 2D sequence.
Merging after encoding, rather than splitting into 28×28 blocks at the input, has an advantage: each base patch first interacts with broader context before being aggregated, reducing the risk of details being compressed away at the entrance. Also, the merge happens after the final layer and before final normalization, so the merged visual tokens no longer pass through any Transformer layer. The code confirms this ordering, but whether it is a deliberate design or an implementation simplification is not something the source itself concludes.
4.6 SwiGLU Projection to Align the LLM Hidden Dimension
The visual encoder’s output dimension isn’t necessarily the same as the language model’s hidden dimension. This project uses merger to align dimensions. Based on how the initialization parameters are sliced (gate, up, down arranged in sequence), it can be inferred that this module uses a projection layer with a SwiGLU structure. SwiGLU introduces nonlinear interaction through a gating mechanism, making it more expressive than an ordinary linear projection, and it is widely used in the feed-forward layers of mainstream language models. The output dimension is fixed at 4096, aligned to the language model’s hidden dimension, so visual tokens can enter subsequent computation as “ordinary tokens.”
5. Precision and Engineering Implementation: Inference-Side Tradeoffs
Parameter scale is only one side of the model; what really determines the online experience is the inference-side precision and implementation strategy. A few engineering tradeoffs worth learning from can be observed in the open-source implementation.
5.1 Fixed BF16, No FP8 Quantization
The visual encoder uses BF16 fixed at inference time and does not use FP8 quantization. This choice has its rationale:
- Limited benefit: the memory and bandwidth gains from FP8 quantization scale with parameter count. Quantizing the sub-1% visual parameters saves little memory.
- Sensitivity to risk: visual input distributions vary widely. Screenshots are dominated by large flat-color areas and sharp edges, while photos contain rich texture and noise; under extreme brightness or high-contrast scenes the activation range can be large. Low-bit quantization is sensitive to outliers, easily introduces error, and accumulates that error layer by layer through the encoding pipeline, ultimately affecting the quality of the visual tokens the language model sees.
- Implementation cost: to get stable benefits, an appropriate quantization backend must be matched to each operator. For such a small module, this engineering cost may not be worthwhile.
Overall, keeping precision on the visual side and concentrating quantization resources on the larger language model backbone is the safer strategy.
5.2 Reusing Mature Models and Evolving Along the Inheritance Chain
Structurally, the visual model follows the implementation of prior multimodal and OCR-specialized models, with differences between generations mainly in configuration parameters while core components—attention, positional encoding, merging—stay stable. This “reuse the model definition, differentiate generations by config” approach is very pragmatic:
- Existing operators and kernels need no repeated adaptation;
- Historical bug fixes can be inherited directly;
- Regression tests can reuse existing cases.
5.3 Clear Module Boundaries for Independent Evolution
The visual encoder exists as an independent module, exposing only one clear contract: “take pixels and grid information as input, output a token sequence aligned with the LLM.” This lets it evolve independently in operator optimization, precision strategy, and batching strategy without touching the language model.
6. Practical Advice: Key Parameters for Using the Visual Encoder Well
For users, a few things directly determine whether multimodal capability holds up in real business scenarios.
Resolution and token budget: not always “higher is better”
- High resolution helps with small-font text and detail recognition, but token count grows roughly with the square of pixel area, raising context usage and latency.
- Recommendation: set resolution and budget by task type—general Q&A, document recognition, and UI understanding can each have their own settings—and compare accuracy and latency on a small evaluation set.
Video input: control frame count first, then resolution
- Total video tokens ≈ number of sampled frames × tokens per frame, requiring a tradeoff between the time and spatial dimensions.
- Also check whether the visual positional encoding covers the time dimension. A 3D grid often means the implementation considered video scenarios, but the exact modeling should be confirmed against the documentation and implementation.
Output granularity: do enough contextual interaction before merging
- Since merging happens after encoding, fine-grained localization tasks should focus on preprocessing alignment and budget settings, rather than hoping to fix things in post-processing.
Precision strategy: steady on the visual side, frugal on the backbone
- Keep the visual side at higher precision such as BF16, and evaluate low-bit quantization on the backbone side according to hardware support.
- Before going live, compare key metrics before and after quantization on small business samples.
Serving deployment: watch memory and elasticity
- Multimodal inference demands noticeably more memory and compute than pure text, and visual tokens also lengthen the prefill stage. You can host the inference service on a container platform with elastic scaling to handle peaks; use logging and monitoring/alerting to track latency, memory usage, and error rates; and store input images in object storage, passing them by reference.
- As an example, enterprises can use general-purpose capabilities such as a model inference service on Baidu AI Cloud to complete deployment, then integrate the foundational capabilities above as needed.
Security and compliance: don’t let external content become instructions
- Visual inputs often come from external sources, such as web screenshots, user-uploaded files, and scanned documents.
- The service side should do input validation, size limits, and access control. For model actions that reference external content (such as calling tools or executing code), maintain least privilege and human confirmation to reduce prompt-injection risk.
7. Common Engineering Problems and Troubleshooting
- Width/height not a multiple of 28: this is normal; preprocessing zero-pads automatically. Note that extremely elongated images produce large invalid regions after padding, so cropping or scaling first is recommended.
- Out of memory (OOM): first check for high-resolution inputs and high concurrency. You can lower the per-image token cap, or rate-limit and queue visual encoding requests separately.
- Poor small-object recognition: try raising the resolution cap, or split the large image into multiple local regions and encode them separately; also confirm
min_image_tokensisn’t over-compressing the input. - Abnormal image or color cast: confirm the normalization parameters in preprocessing match the model version, and check whether image processor configs from different model generations are being mixed.
- High first-token latency: visual encoding is compute-intensive, so evaluate operator optimization and batching strategies, and observe visual encoding time separately in monitoring.
Conclusion
From the 28-pixel alignment in preprocessing, to contextual encoding by the 24-layer ViT, to the 2×2 merge and SwiGLU projection, this lightweight multimodal model’s visual encoder completes the “pixels to tokens” conversion through one clear, streamlined pipeline. At about 0.56B parameters—less than 1% of the full model—combined with a fixed-BF16 precision strategy, a variable-length packing attention implementation, and code organized to evolve along an inheritance chain, it reflects design tradeoffs oriented toward engineering feasibility. For developers, understanding this pipeline means more than reading code: only by knowing where tokens come from, how they’re compressed, and where they’re aligned can you make sound decisions in resolution tuning, cost control, and serving deployment.
