Lightweight Visual Encoders in Multimodal LLMs: From Pixels to Visual Tokens

    This article breaks down the lightweight visual encoder used in recent flagship multimodal LLMs, where a small vision tower is paired with a much larger language model. It walks through the full path from arbitrary-resolution images to visual tokens—covering 28-pixel alignment and preprocessing, 2×2 patch merging, token budget control, axial rotary position embeddings, variable-length attention, and SwiGLU projection for cross-modal alignment. It also explains why the vision tower stays in BF16 rather than FP8, and offers practical guidance on resolution and token-budget tuning, preprocessing consistency, inference optimization, and quality monitoring.

    Lightweight Visual Encoders in Multimodal LLMs: From Pixels to Visual Tokens

    A multimodal LLM’s visual ability depends not only on the language model itself, but also on how the “visual front end” turns an image into information the model can understand. A recent wave of lightweight flagship multimodal models all follow a similar pattern: pair a relatively small visual encoder with a much larger language model. The visual encoder usually accounts for only a tiny fraction of total parameters, yet it determines whether the model can see fine detail, how much content it can take in, and how much compute inference will cost.

    This article dissects the processing pipeline of such a lightweight visual encoder: how an arbitrary-resolution image becomes visual tokens, how the token budget is controlled, how visual tokens align with the language model, and why natively integrating vision matters. It closes with practical engineering recommendations.

    1. Visual Encoding: From “Reading an Image” to “Using Eyes to Complete a Task”

    Traditional vision tasks—image classification, object detection, OCR—mostly follow a fixed pipeline of “one input, one output.” Multimodal LLMs face a different situation: within a single inference pass, the model may need to analyze a screenshot, reconstruct a UI, read a chart, and recognize document structure—sometimes acting as an agent that autonomously observes a screen or page while it works.

    These tasks have two defining traits:

    • Intermediate results depend on visual observation. For example, the model first locates a UI element, then reasons about the layout, and finally generates code or performs an action.
    • Resolution must adapt. Small text in a document, table borders, and details in a code editor are unrecoverable once they are lost during scaling.

    For this reason, the official technical documentation stresses that visual encoding is not merely “processing an image” but extending the reach of what the model can encode; vision needs to be natively integrated into the model so it can decide on its own when to observe and use visual feedback to guide subsequent actions. This places three requirements on the visual encoder:

    1. Detail fidelity: preserve original-resolution information as much as possible; avoid aggressive uniform scaling.
    2. Budget control: the number of visual tokens must scale flexibly with the context budget.
    3. Closed feedback loop: visual output must live in the same representation space as the language model, so it can be inserted or replaced at any point in the reasoning chain.

    The encoder structure dissected below is designed around exactly these three points.

    2. Overall Architecture: Dividing Work Between a Lightweight Vision Tower and the Language Model

    2.1 Processing pipeline

    The whole visual pipeline can be summarized as:

    Raw image/video → standardized preprocessing (resize + zero-padding + patchify) → lightweight visual encoder → spatial merge and projection → concatenation with text tokens → autoregressive generation by the language model

    This division of labor brings three benefits:

    • Clear responsibilities: the vision tower focuses on extracting semantic features from pixels, while the language model handles cross-modal reasoning and generation. Each module can evolve and be optimized independently.
    • Training stability: once the vision tower’s output is aligned through projection, it is fed to the language model in the same form as text tokens, so multimodal training can reuse the language model’s mature training frameworks and parallelism strategies.
    • Engineering reuse: the visual encoder inherits the implementation of earlier models in the same family (a 4-channel vision encoder → an OCR vision encoder → a new-generation multimodal vision encoder). Operators, parallel sharding, and caching strategies in the inference stack carry over smoothly. This lowers the engineering adaptation cost of shipping a new model and is an important precondition for the “lightweight flagship” strategy to be viable.

    2.2 Structural parameters

    The vision tower of a lightweight flagship multimodal model has the following configuration (its publicly documented settings, shown here as an example):

    Item Value Engineering meaning
    depth (layers) 24 Moderate depth; enough receptive field and semantic abstraction
    hidden_size 1024 Small channel count; low per-token compute and memory cost
    attention heads 16 64 per head (1024/16); no GQA
    patch_size / merge 14 / 2×2 Base block of 14×14 pixels; every 4 blocks merge into 1 visual token
    output dim 4096 Aligned to the language model’s hidden dim; visual tokens concatenate directly
    backbone params ≈ 0.56B Under 1% of the full model

    A few key takeaways follow from this table:

    • The vision tower is deliberately small. A hidden size of 1024 is a medium-light configuration for a vision encoder; combined with 24 layers, it is enough to extract multi-level features such as texture, text, and layout.
    • Skipping GQA is reasonable. Grouped-query attention mainly compresses the KV cache during autoregressive decoding, but the vision tower is a one-shot encoding process with no incremental decoding. Standard multi-head attention is simpler to implement and better supported by inference frameworks.
    • Output-dimension alignment is a prerequisite. After projection, the tower’s output enters the language model sequence directly. Dimension alignment removes the need for an extra cross-modal adapter and makes it easier to arrange position encodings uniformly with text tokens.

    3. Preprocessing: Turning Arbitrary-Resolution Images into Tokens

    An arbitrary-resolution image (a picture or a video frame) must be standardized before entering the vision tower. The core logic is: without breaking the aspect ratio, convert the image into a grid of fixed-size “blocks.” This step solves three problems at once: uniform resolution, computable positions, and predictable token counts.

    3.1 Resize and zero-padding: why align to multiples of 28

    Preprocessing first resizes the image to an appropriate size and zero-pads it to a multiple of 28. The number 28 is not arbitrary:

    • 14: the vision tower uses a base patch of 14×14 pixels, matching the input granularity of many mainstream vision backbones.
    • 2×2: after encoding, every 4 adjacent base patches are merged into 1 visual token.
    • 28 = 14 × 2: aligning to 28 guarantees that both width and height after scaling are divisible by “14” and “2,” so no incomplete patch is cut and no merge block crosses a boundary.

    Illustrative code:

    1. import torch
    2. import torch.nn.functional as F
    3. PATCH = 14 # Base patch edge length (pixels)
    4. MERGE = 2 # Spatial merge factor
    5. ALIGN = PATCH * MERGE # Alignment granularity = 28
    6. def align_resize(img: torch.Tensor,
    7. min_tokens: int = 16,
    8. max_tokens: int = 8000,
    9. align: int = ALIGN) -> torch.Tensor:
    10. """img: (C, H, W), returns an image aligned to a multiple of `align` (illustrative)"""
    11. _, h, w = img.shape
    12. # 1) Scale proportionally to the token budget: one 28x28 region maps to one visual token
    13. tokens = (h / align) * (w / align)
    14. scale = 1.0
    15. if tokens > max_tokens:
    16. scale = (max_tokens / tokens) ** 0.5
    17. elif tokens < min_tokens:
    18. scale = (min_tokens / tokens) ** 0.5
    19. h_s = max(align, int(h * scale))
    20. w_s = max(align, int(w * scale))
    21. img = F.interpolate(img[None], size=(h_s, w_s),
    22. mode="bicubic", align_corners=False)[0]
    23. # 2) Zero-pad to a multiple of 28
    24. h_p = (h_s + align - 1) // align * align
    25. w_p = (w_s + align - 1) // align * align
    26. img = F.pad(img, (0, w_p - w_s, 0, h_p - h_s), value=0.0)
    27. return img

    Two points are worth noting:

    • The resize stage usually considers a target-resolution upper bound, a lower bound, and the token budget together. The implementation above keeps only the token-budget part to make the principle clear.
    • Zero-padding is merely an engineering stopgap. For images with extreme aspect ratios, padding can introduce a fair number of “blank” tokens. In practice you should constrain the single-image aspect ratio, or pair this with a tiling strategy for long images.

    3.2 Patchify and 2×2 merge

    The aligned image is cut into 14×14 base patches, after which every 2×2 group of 4 adjacent patches is combined into one “block.” The math is:

    1. Let H, W be the height and width after resize and padding
    2. Base patch count N_patch = (H / 14) × (W / 14)
    3. Visual token count N_token = N_patch / 4 = (H / 28) × (W / 28)

    The 2×2 merge matters because it:

    • Compresses the sequence length: after merging, the sequence is 1/4 of its original length, greatly reducing the visual tokens the language model must process.
    • Aggregates local information: before merging, the vision tower performs full-depth encoding on each 14×14 block, so small text and thin lines are already well modeled; the merge is just a spatial downsampling on top of that, not a raw pixel discard.
    • Reduces position-encoding pressure: a shorter sequence lowers both the attention compute and the difficulty of extrapolating position encodings.

    3.3 Token budget control

    Since each 28×28 region corresponds to one visual token, developers can constrain the number of tokens a single image produces via min_image_tokens and max_image_tokens (defaulting to 16 through 8000):

    • Upper bound: prevents one high-resolution screenshot from consuming a large chunk of context and truncating text instructions and prior conversation. For example, with a budget of 8000, a 1024×768 screenshot consumes roughly 1000 tokens, still leaving room for later multi-turn visual observations.
    • Lower bound: avoids scaling down so far that text and icon details are all lost.
    • Working with resolution constraints: the budget usually takes effect together with max/min pixel counts, long-edge limits, and other conditions; the scaling logic takes the intersection across these constraints.

    Token consumption at different resolutions can be estimated from the table below (using the 28-alignment rule):

    Typical scenario Original size Aligned size Visual tokens (approx.)
    Phone screenshot 1080×2400 1092×2408 39×86 ≈ 3354
    Desktop screenshot (1080P) 1920×1080 1932×1092 69×39 ≈ 2691
    Tablet screenshot 1024×768 1036×784 37×28 ≈ 1036
    Thumbnail / icon area 256×256 252×252 9×9 = 81
    Short video (5 s, 2 frames/s) 512×512 × 10 frames 504×504 × 10 frames 10×18×18 = 3240

    As the numbers show, visual-token consumption is almost directly proportional to the image’s pixel area. In multi-turn visual tasks, setting a sensible budget often improves context utilization more than simply upgrading the model.

    4. Inside the Encoder: From Pixel Patches to Visual Features

    The encoder’s overall flow can be summarized as: patchify → position encoding → 24 Transformer blocks → final normalization → 2×2 spatial rearrange → convolutional downsample → projection alignment. The forward pass looks like this (illustrative code):

    1. def forward(self, hidden_states, grid_thw, **kwargs):
    2. # grid_thw: (num_images, 3), each row is the (T, H, W) grid size of one image/video (unit: patch)
    3. position_ids = get_vision_position_ids(grid_thw, self.spatial_merge_size)
    4. cu_seqlens, max_seqlen = get_vision_attention_seqlens(grid_thw, self.config)
    5. hidden_states = self.patch_embed(hidden_states) # ① Conv3d patchify
    6. position_embeddings = self.rotary_pos_emb(hidden_states, position_ids) # ② Axial RoPE
    7. for blk in self.blocks: # ③ 24 x Block
    8. hidden_states = blk(hidden_states,
    9. position_embeddings=position_embeddings,
    10. cu_seqlens=cu_seqlens,
    11. max_seqlen=max_seqlen)
    12. hidden_states = self.post_layernorm(hidden_states) # ④ Final normalization
    13. hidden_states = hidden_states.view(-1, 2, 2, C) # ⑤ 2×2 spatial rearrange
    14. hidden_states = hidden_states.permute(0, 3, 1, 2) # (N, C, 2, 2)
    15. hidden_states = self.downsample(hidden_states).view(-1, out_hidden)
    16. # Conv2d downsample
    17. merged = self.merger(hidden_states) # ⑥ SwiGLU projection
    18. return merged

    4.1 Conv3d patchify

    The first step performs patchification with a Conv3d convolution: the input is a pixel tensor shaped (frames, channels, height, width), the kernel is 14×14 with stride 14 in the height and width directions and covers a single frame in the time direction (i.e., each frame is patched independently), and the output has 1024 channels.

    • For a static image, this is equivalent to a standard 2D patch embedding.
    • For video, the frame dimension reuses the same implementation, so images and videos share one encoding path with no need to design separate preprocessing or position-encoding logic for video.
    • Pixel values are normalized before being fed in, and zero-padded regions enter the network as 0, so they introduce no spurious texture information.

    4.2 Axial rotary position encoding (axial RoPE)

    The visual sequence uses axial rotary position encoding: each patch has coordinates along three axes—time, height, and width—and the rotation is decomposed and applied along each axis separately.

    • 2D images: the time coordinate degenerates to a constant, so position information is effectively determined by height and width, accurately encoding the relative relationships between different regions within the same image.
    • Video: the time axis participates in encoding, letting the model distinguish how the same position’s content changes over time.
    • Resolution-agnostic: because the position encoding is based on continuous coordinates rather than a fixed grid, inputs at any resolution get a continuous, consistent position representation.

    Illustrative code:

    1. def get_vision_position_ids(grid_thw, merge_size):
    2. """Generate (t, h, w) 3D position coordinates for each patch"""
    3. llm_pos, index = [], 0
    4. for t, h, w in grid_thw:
    5. h, w = h // merge_size, w // merge_size
    6. # Each merged unit outputs one token, anchored at its center-point coordinate
    7. h_idx = torch.arange(h).unsqueeze(1).expand(-1, w)
    8. w_idx = torch.arange(w).unsqueeze(0).expand(h, -1)
    9. # t, h, w 3D coordinates stay consistent within a merged unit
    10. llm_pos.append(torch.stack([h_idx, w_idx, h_idx], dim=-1).flatten(0, 1) + index)
    11. index += max(h, w)
    12. return torch.cat(llm_pos)

    These implementation details are illustrative. Different implementations may trade off between “anchoring on the merged block’s center” and “numbering each base patch individually,” but the design goal is the same: make the position representation of visual tokens both continuous and part of the same position system as text tokens.

    4.3 The 24 Transformer blocks and variable-length attention

    The body of the vision tower is a stack of 24 standard Transformer blocks, each containing multi-head self-attention and a feed-forward network. The key engineering point is variable-length attention:

    • Packing: multiple images in a batch (or multiple frames of a video) are concatenated into one long sequence.
    • Isolation: cu_seqlens (cumulative sequence lengths) records each image’s boundaries within the packed sequence, and max_seqlen provides an upper bound for the kernel. Attention is confined to a single image, so different images never “see” each other.
    • Payoff: this avoids the compute and memory waste of padding to the maximum size, while making it easy to batch-process multiple-image requests in high-throughput inference.

    One important property of the vision tower is that an image’s visual features depend only on the image content, the model weights, and the preprocessing parameters—not on the decoding state. So when the same image recurs in a conversation (for example, the same screenshot referenced across several turns), its visual features can be reused directly from cache with no need to re-encode. This is in the same spirit as KV-cache optimization in language models.

    4.4 Final normalization and SwiGLU projection

    Once encoding is complete:

    1. A final normalization stabilizes the feature distribution.
    2. Channels are rearranged into a 2×2 layout, organizing each merged unit’s 4 patch features into a 2D (C, 2, 2) form.
    3. A Conv2d downsample with stride 2 compresses this into a single feature vector. In terms of parameters, this is equivalent to applying a linear transform to each merged unit’s concatenated 4C-dimensional feature.
    4. A SwiGLU projection (the merger) maps the visual features to the same hidden dimension as the language model (4096 in this example). SwiGLU introduces a gating mechanism during projection, giving the output nonlinear selection ability that makes it better suited to cross-modal alignment than a plain linear projection.

    4.5 How visual tokens enter the language model

    Prompt text usually contains image placeholder markers, and during preprocessing these placeholders are expanded to the computed number of visual tokens. When the forward pass runs, the text embeddings at those placeholder positions are replaced one by one with the visual encoder’s outputs, forming a unified input sequence:

    1. [system instruction] + [user question] + [visual token × N] + [conversation history] + [new user input]

    To keep position encodings continuous, the language model typically treats these visual tokens as part of the continuous positions during computation. In effect, visual and textual information carry equal weight inside the language model: the model can “read” an image much as it reads text, and can refer back to visual details repeatedly within the reasoning chain.

    5. What the Lightweight Design Means for Performance

    Evaluating a lightweight visual encoder requires looking at both the “quality side” and the “cost side.”

    5.1 Quality side

    • Native high resolution: aligning to 28 and supporting any aspect ratio preserves small text, tables, and code details, giving an edge on visual-programming tasks such as document understanding, UI reconstruction, and chart reading.
    • Unified image, text, and video: images and videos share one encoding path, so the model behaves more consistently across multimodal tasks.
    • Autonomous observation: visual features interact with text in the same sequence, letting the model decide when to observe and what to observe, and adjust later steps based on visual feedback—well suited to agent-style applications.
    • Architectural continuity: the vision tower carries on from earlier designs in the same family, so training and inference behavior is relatively predictable and easy to validate and tune within existing engineering systems.

    5.2 Cost side

    • Low encoding overhead share: with the example parameters, the 0.56B backbone adds about 1.1 GFLOPs of encoding compute per visual token, so one image costs roughly “token count × 1.1 GFLOPs.” A 1024×768 screenshot is around 1.1 TFLOPs, typically just milliseconds to a dozen-plus milliseconds on common data-center GPUs; by contrast, a language model’s prefill at the same token scale often needs hundreds of GFLOPs to several TFLOPs, on top of per-token decoding cost. Visual encoding is not the main bottleneck of end-to-end latency.
    • Controllable context footprint: a single 28×28 region’s encoded information is compressed into one 4096-dim vector, sharply reducing the sequence length visual information occupies on the language-model side. Paired with the token-budget parameters, context growth in multi-turn visual tasks can be effectively constrained.
    • Low memory footprint: at BF16 weights, the 0.56B vision tower needs only about 1.1 GB of memory, freeing up resources for the language model without meaningfully increasing deployment cost.

    6. Inference Precision: Why Fixed BF16 and No FP8 Quantization

    The example model’s vision tower is fixed to BF16 at inference and does not take part in FP8 quantization, for three reasons:

    1. The output directly affects generation quality. Visual features enter the language model directly as part of the input. If the language model body uses FP8 while the vision tower uses lower precision, the mismatch between the two representations at the concatenation point gets amplified, hurting recognition of text and details.
    2. Limited quantization gains. The vision tower has few parameters and a low compute share, so the memory and bandwidth savings from quantization are relatively limited, yet they come with extra calibration and precision-validation costs.
    3. Visual features are more precision-sensitive. The tower’s features have high dimensionality and a wide value range, so low-precision quantization easily causes perceptible degradation on fine-grained tasks.

    A safer strategy is: keep the vision tower in BF16, and evaluate quantization schemes for the language model body based on the business scenario. Before launch, run before/after comparisons on fine-grained tasks such as document recognition, UI element localization, and chart reading—not just on general metrics.

    7. Engineering Practice Recommendations

    7.1 Choosing resolution and the token budget

    • Documents, code screenshots, UI reconstruction: prioritize resolution and raise max_image_tokens appropriately so small text is not flattened by scaling.
    • Long conversations, multi-turn visual observation: lower the per-image budget somewhat, or crop key regions before feeding the model, to leave room for historical context.
    • Video understanding: control the product of sampled-frame count and per-frame resolution; when necessary, reduce the frame rate rather than the resolution, since text and UI information depend more on spatial clarity.

    7.2 Preprocessing consistency

    Training and inference must use the same alignment rule, patch size, and merge factor; otherwise a train–inference mismatch appears. It is best to harden the preprocessing logic into a standalone component with unit tests, focusing coverage on edge cases such as non-multiples-of-28 sizes, extreme aspect ratios, very small images, and very large images.

    7.3 Inference performance optimization

    • Variable-length batching: replace padding with cumulative sequence lengths to improve GPU utilization.
    • Cache visual features: reuse encoding results for images that recur within the same session.
    • Parallelism strategy: the vision tower has few parameters and usually needs no separate sharding, so it can be deployed on the same instance as the language model; only in high-concurrency, multi-image scenarios should you consider a dedicated vision-encoding service.
    • Precision strategy: keep the vision tower in BF16, and quantize the language model as needed.

    7.4 Quality evaluation and monitoring

    • Per-task evaluation: break evaluation into document OCR, chart reading, UI element localization, and video temporal understanding, so a single aggregate score does not mask weak spots.
    • Online metrics: track the average number of visual tokens, the share of time spent on visual encoding, and the proportion of the context taken up by visual tokens, as the basis for capacity planning and budget tuning.

    8. Conclusion

    The value of a lightweight visual encoder lies in “trading a small cost for a large capability”:

    • Under 1% of the model’s parameters, yet it determines what the model can see.
    • Through 14×14 patches, 2×2 merging, and the 28-alignment rule, it balances detail and efficiency at any resolution.
    • Through the token budget, it makes the context cost of visual information controllable.
    • Through unified position encoding and same-dimension projection, it makes vision truly “native” to the language model’s reasoning process.

    For developers, using such models well is not just about calling the API—it is about understanding every step of “how pixels become tokens.” Only by applying the engineering levers of resolution, budget, precision, and caching correctly can you balance visual quality and inference cost in real workloads.