Visual Encoding Architecture for Lightweight Multimodal LLMs: From Seeing Images to Acting on Them

    This article examines how lightweight multimodal LLMs can move beyond image captioning toward native visual reasoning and action. It explains image preprocessing and token-budget control, describes a compact visual encoder configuration, walks through a simplified forward-inference pipeline, and summarizes practical engineering considerations for aligning visual representations with a language backbone while keeping visual compute and memory overhead under control.

    From “Seeing” to “Acting on Images”: The Core Goal of Multimodal Models

    In early multimodal attempts, visual capability was often understood as “converting image content into a text description.” But genuinely useful visual capability goes much further: it means the model can actively decide when to observe during reasoning and use visual feedback to correct subsequent steps. For example, when processing an interface screenshot, a code execution result, or a process diagram, the model needs to judge whether it should look at the image now, whether what it sees matches expectations, and how the next action should be adjusted accordingly. For the model to have this capability, vision cannot simply be attached as an external module. It must be natively integrated into the model structure and become a first-class input alongside language.

    This is the starting point for lightweight multimodal model design: use as little visual overhead as possible to obtain the ability to “understand and make good use of visual information.”

    How a Visual Encoder “Reads” an Image

    For a language model to consume an image, the first step is to discretize the continuous pixel matrix into a token sequence. The mainstream approach follows the idea of the Vision Transformer (ViT): split an arbitrary-resolution image or video frame into fixed-size patches, then compress and map them into vector representations acceptable to the language backbone. The pipeline usually includes the following stages.

    Image Preprocessing: Normalization and Alignment

    Before entering the encoder, the image first passes through a dedicated image processor for normalization:

    • Resizing and zero-padding: After the image is resized to an appropriate size, it is zero-padded to an integer multiple of a certain size. With a common 14×14 base patch and a 2×2 merge factor, the padding target is usually a multiple of 28—because 28 equals 14 (patch side length) × 2 (merge factor). This ensures that the subsequent merge operation is aligned and leaves no incomplete regions.
    • Splitting into patches: The padded image is split into 14×14-pixel patches and then subjected to a 2×2 spatial merge, combining each group of four adjacent base patches into a larger semantic unit. This preserves detail while compressing sequence length.
    • Token budget control: Each 28×28-pixel region corresponds to one visual token. Models typically expose minimum and maximum image token counts—for example, a configurable range from 16 to 8,000—as parameters for flexibly adjusting the token cost per image under different context windows and compute budgets. High-resolution images can be allocated more tokens to preserve detail, while low-information images can be compressed to save context.

    The purpose of this layer is to balance visual detail against compute savings. The number of tokens directly determines the proportion of visual information in the language model’s context and is a key adjustable knob.

    Structural Design of a Lightweight Visual Encoder

    A typical lightweight visual encoder may use a configuration roughly like the following:

    Item Typical value
    Depth 24
    Hidden size 1024
    Attention heads 16 (64 dimensions per head; no GQA)
    Patch size / merge factor 14 / 2×2
    Output dimension 4096 (aligned with the language backbone hidden size)
    Backbone parameters About 0.5B, less than 1% of the overall model

    This set of parameters reflects a pragmatic design philosophy: the visual encoder only “translates”; it does not “make decisions.” It does not use complex mechanisms such as grouped-query attention (GQA), relying instead on a regular multi-head structure to keep the implementation simple. Its output dimension is deliberately aligned with the language backbone’s hidden dimension (for example, 4096), so that visual representations can be seamlessly injected into the language sequence. The visual backbone parameter count is usually kept to a very small share of the total model, and inference generally runs at fixed BF16 precision to avoid losing visual detail through quantization. This controls memory and latency overhead while maintaining stable image understanding.

    Step-by-Step Breakdown of the Forward Inference Pipeline

    A simplified visual model forward pass clearly shows how data flows:

    1. def forward(self, hidden_states, grid_thw, **kwargs):
    2. # Compute position indices and variable-length attention metadata from the image grid size
    3. position_ids = get_vision_position_ids(grid_thw, self.spatial_merge_size, ...)
    4. cu_seqlens, max_seqlen = get_vision_attention_seqlens(grid_thw, self.config, ...)
    5. hidden_states = self.patch_embed(hidden_states) # ① Convolutional patch embedding: pixels -> vectors
    6. position_embeddings = self.rotary_pos_emb(hidden_states, position_ids) # ② Axial rotary position encoding
    7. for blk in self.blocks: # ③ 24 Transformer blocks
    8. hidden_states = blk(hidden_states, cu_seqlens=cu_seqlens, ...)
    9. hidden_states = self.post_layernorm(hidden_states) # ④ Final normalization
    10. hidden_states = hidden_states.view(-1, 2, 2, C) # ⑤ 2×2 spatial rearrangement
    11. hidden_states = hidden_states.permute(0, 3, 1, 2)
    12. hidden_states = self.downsample(hidden_states).view(-1, out_hidden) # Convolutional downsampling
    13. merged = self.merger(hidden_states) # ⑥ Projection alignment to language dimension
    14. return merged

    The design intent of each step is as follows:

    1. Convolutional patch embedding (①): A convolution kernel maps each image patch to a fixed-dimension vector. This is the entry point for pixels into the Transformer.
    2. Axial rotary position encoding (②): Axial rotary position embeddings (RoPE) inject positional information separately along the height and width directions. This lets the model perceive two-dimensional spatial layout while supporting length extrapolation.
    3. Multi-layer encoder blocks (③): Twenty-four standard Transformer blocks are stacked. Variable-length attention uses cumulative sequence lengths (cu_seqlens) to efficiently process images with different aspect ratios and sizes, without separately designing a structure for each resolution.
    4. Final normalization (④): A unified LayerNorm stabilizes the output distribution.
    5. Spatial rearrangement and downsampling (⑤): Reshape and convolution implement the 2×2 spatial merge described above, fusing four adjacent vectors into one and directly reducing the number of tokens sent to the language model.
    6. Projection alignment (⑥): A fusion module, often containing a SwiGLU activation, maps the visual dimension to the language backbone’s hidden dimension, completing the final “vision → language” step.

    Design Considerations and Engineering Practices

    Based on the analysis above, several general lessons are worth noting when building lightweight multimodal models:

    • Prioritize alignment over stacking: Aligning the visual output dimension with the language backbone and coordinating the positional encoding scheme can improve fusion more effectively than simply adding more visual layers.
    • Use token budgets to gain flexibility: Making the per-image token count a configurable parameter enables smooth switching between long-context scenarios and fine-grained recognition tasks.
    • Constrain the visual backbone’s share: Keeping the encoder lightweight—typically less than 1% of the overall model—and reserving most compute for the language backbone is a more cost-effective route.
    • Choose precision strategy carefully: The visual path is sensitive to detail, so fixed higher-precision inference is often more reliable than aggressive quantization.
    • Design for “acting with images”: The real value lies not in describing images, but in enabling the model to decide autonomously when to observe and to iterate based on visual feedback. The architecture should reserve interfaces for this closed loop.

    Overall, the key to integrating vision into large models is to use a structurally regular, cost-controlled encoder to reliably translate images into tokens that language can consume, and then rely on native integration to help the model learn to “see” and “use” visual information. Understanding preprocessing, token budgets, and the forward pipeline captures the main thread of multimodal model engineering.