Lightweight Multimodal Models: How the Visual Encoding Architecture Works and Why It Matters

    This article explains how lightweight multimodal models encode visual information and how they connect vision to language reasoning. It walks through the visual backbone configuration, the pipeline that turns images into language-readable tokens (resizing and padding, 14×14 patch splitting, and 2×2 spatial merging), dynamic visual token budgets, visual position encoding for images and videos, and dimension alignment with the language model. It also covers the trade-offs between inference precision and deployment efficiency, the core advantages of the architecture, and practical guidance for developers on tuning token budgets across tasks such as classification, document recognition, video understanding, and GUI operation.

    一、Design Goals of Lightweight Multimodal Models

    In recent years, multimodal models have evolved from simply “recognizing images” to “completing complex tasks based on visual information.” Scenarios such as image understanding, video analysis, interface operation, visual programming, and document processing require models to not only interpret content but also decide the next action based on visual feedback.

    The traditional approach treats the visual encoder as a standalone module attached to the language model: an image first passes through a vision network to extract features, which are then converted through a connector layer into vectors the language model can consume. This architecture is relatively simple to implement, but it has two problems:

    1. The connection between visual information and language reasoning is not tight enough;
    2. The higher the image resolution, the more visual tokens are produced, increasing inference cost and context pressure.

    The core idea behind lightweight multimodal models is to compress visual information as much as possible while preserving visual understanding, and to let the model dynamically decide the observation range and the amount of visual information according to task needs. In other words, the visual module is not only responsible for “encoding the image” but also for serving the subsequent reasoning, planning, and action.

    二、The Visual Backbone: Small in Scale, Complete in Structure

    These models typically use a relatively compact vision transformer as the encoder for images and videos. A typical configuration includes:

    Item Typical Configuration
    Network depth 24 layers
    Hidden dimension 1024
    Attention heads 16
    Per-head dimension 64
    Base image patch size 14×14 pixels
    Spatial merge 2×2
    Output dimension 4096
    Backbone parameters ~0.56B

    In terms of parameter count, the visual backbone accounts for only a small fraction of the full multimodal model, usually less than one percent of the total. This design delivers two direct benefits:

    • The visual encoding process itself is more lightweight, with lower memory and compute pressure;
    • The visual module can be combined with a large language model without significantly increasing overall deployment complexity.

    It is worth noting that fewer parameters in the visual encoder do not necessarily mean weaker visual capability. A vision model’s performance depends not only on its parameter count but also on training data, image resolution, position encoding, token compression strategy, the visual-language alignment method, and the reasoning ability of the downstream language model.

    三、How Images Are Converted into Tokens the Language Model Can Read

    A vision model cannot process raw images directly. Input images must go through normalization, patch splitting, feature extraction, and token compression, ultimately becoming a set of visual representations that can participate in attention computation together with text tokens.

    1. Resizing and Border Padding

    Input images usually have different widths and heights, while the visual backbone is better suited to sizes that meet specific stride requirements. Therefore, before entering the model, an image is first resized to an appropriate size and then padded with zeros to a multiple of 28.

    This value of 28 is not arbitrary; it is determined by the downstream structure:

    • The base patch size is 14×14;
    • Adjacent 2×2 base patches are merged;
    • Therefore, each final visual token corresponds to a spatial region of 28×28 pixels.

    Padding ensures that images do not produce unaligned boundary regions when split and merged. For the valid content of the original image, the padded regions are usually distinguished through masks or position information, preventing invalid pixels from noticeably interfering with the semantic representation.

    2. Base Patch Splitting

    After resizing, the image is divided into multiple 14×14 base patches. Each patch can be understood as the smallest spatial unit the vision model processes, similar to a base token in a language model.

    If the image size is height times width, the number of base patches is roughly:

    1. patch_count = (image_height / 14) × (image_width / 14)

    The higher the image resolution, the more base patches there are, and the higher the cost of subsequent attention computation. Therefore, simply increasing the input resolution does not improve results without limit; it must be paired with a reasonable visual token compression strategy.

    3. 2×2 Spatial Merging

    After the base patches are extracted, the model merges every four adjacent patches in a 2×2 arrangement. The merged region covers 28×28 pixels and corresponds to one visual token.

    This design significantly reduces the length of the visual sequence sent to the language model. Without merging, an image would produce a large number of base visual features; after merging, the number of tokens is reduced to about a quarter of the original, which lowers:

    • The compute of the joint attention between vision and text;
    • Context usage in multi-turn conversations;
    • Memory pressure from high-resolution images;
    • The accumulated cost of multi-frame input in video scenarios.

    It is worth emphasizing that spatial merging is not the same as simply discarding information. In practice, the model first extracts high-dimensional features through the vision network, then fuses local regions, so that the merged tokens preserve key information such as texture, edges, text, and local semantics as much as possible.

    四、Dynamic Visual Token Budgets

    Different tasks have different requirements for visual detail. For example:

    • Determining whether an image is indoor or outdoor requires only coarse-grained information;
    • Reading tables and code screenshots requires higher resolution;
    • Analyzing engineering drawings or complex interfaces may require a large number of visual tokens;
    • Video summarization focuses more on cross-frame information and should not let a single frame occupy the entire context.

    Therefore, models usually allow controlling the scale of a single image’s visual representation through a minimum and maximum token count. A typical default range can be set between 16 and 8000, with the actual number determined by the image size, aspect ratio, and task requirements.

    This mechanism can be understood as “visual budget management”:

    1. Input image
    2. ↓
    3. Estimate visual token demand based on resolution and aspect ratio
    4. ↓
    5. Clamp between the minimum and maximum values
    6. ↓
    7. Resize the image and perform patch encoding
    8. ↓
    9. Output visual tokens within the fixed budget range

    The core value of the dynamic token budget is balancing quality and cost. Too few tokens may lose detail; too many tokens slow down inference, increase memory usage, and take up text context space.

    In practice, strategies can be configured according to the task type:

    Scenario Recommended Strategy
    Image classification Use a lower token budget
    General question answering Use a medium budget
    Document recognition Increase the budget and preserve fine-grained spatial information
    Tables, code screenshots Prioritize higher resolution and a larger maximum token count
    Video understanding Control the per-frame budget and combine with frame sampling
    GUI operation Dynamically allocate the budget by interface region

    五、Visual Position Encoding and Variable-Length Sequence Handling

    Images and videos have one important difference from text: visual input has a two- or three-dimensional spatial structure.

    • Images typically contain two spatial dimensions, height and width;
    • Video adds a time dimension;
    • Different images may have different resolutions;
    • Different videos may have different frame counts.

    Therefore, a visual encoder cannot rely only on simple one-dimensional position numbering; it must generate position encodings according to the grid structure. For video input, the position information usually needs to reflect three dimensions—frame, row, and column—so that the model can distinguish:

    • Different spatial positions within the same frame;
    • The same position across different frames;
    • The trajectory of an object as it changes over time.

    At the same time, different samples in a batch may produce different numbers of visual tokens. To improve inference efficiency, engineering implementations usually concatenate the visual features of multiple samples for processing and record the boundary of each sample through cumulative sequence lengths. This supports variable-length inputs while avoiding a large amount of invalid padding.

    The logic can be abstracted as:

    1. position_ids = build_visual_positions(grid_shape)
    2. cu_lengths = build_cumulative_lengths(sequence_lengths)
    3. features = patch_embedding(pixel_values)
    4. features = add_position_information(features)
    5. features = visual_transformer(features, cu_lengths)
    6. features = spatial_merge(features)
    7. features = project_to_language_dimension(features)

    The key here is not the specific function names but the processing order: first complete patch embedding, then add spatial position information, then perform visual attention computation, and finally complete spatial merging and dimension mapping.

    六、Aligning Visual Features with the Language Model’s Dimension

    The feature dimension output by the visual backbone usually needs to be mapped to the language model’s hidden dimension. For example, the visual module can output a 1024-dimensional intermediate feature, which a projection layer then converts to 4096 dimensions so that it shares a unified representation space with text tokens.

    This step is very important. If the visual features and the language model features are not in the same semantic space, the language model will struggle to correctly interpret the visual vectors as text, objects, positions, or actions even after receiving them.

    A common alignment process includes:

    1. The visual encoder extracts local and global features;
    2. Spatial merging reduces the visual sequence length;
    3. A linear layer or multi-layer perceptron performs dimension projection;
    4. Visual tokens are inserted at designated positions in the text sequence;
    5. The language model then uniformly handles cross-modal attention and subsequent generation.

    This design allows visual information to participate in reasoning like text context, while visual tokens still preserve spatial order and image structure.

    七、Balancing Inference Precision and Deployment Efficiency

    During inference, the visual backbone uses low-precision formats such as BF16, which reduce memory usage and compute cost while maintaining good numerical stability. For vision networks, low-precision inference raises several concerns:

    • Numerical stability of patch embedding and normalization layers;
    • Error accumulation in attention computation for high-resolution images;
    • Precision loss when projecting visual features into the language space;
    • Tensor type consistency in mixed batching of images and videos.

    Not all modules are suitable for a more aggressive quantization strategy. The visual encoder contains many continuous feature transformations, and some layers are sensitive to precision changes. Therefore, a common engineering practice is to first adopt BF16 as a stable baseline, then decide whether to apply lower-bit quantization to some modules based on actual evaluation results.

    Deployment optimization can also be pursued from the following angles:

    • Batch process images of the same size;
    • Reuse visual features of repeated images or video frames;
    • Use keyframe sampling for video;
    • Pipeline the visual encoding and language generation in parallel;
    • Use region cropping and block recognition for extremely large images;
    • Dynamically adjust the token limit based on the task.

    八、Core Advantages of This Architecture

    Overall, the lightweight visual encoding architecture has the following main advantages.

    1. Lower Parameter Overhead

    The compact visual backbone does not noticeably increase the storage and deployment cost of the overall model, making it suitable for running in environments with limited memory.

    2. High Visual Token Compression Efficiency

    Through 14×14 base patches and 2×2 spatial merging, it reduces sequence length while preserving local information as much as possible, lowering the cost of multimodal attention.

    3. Support for Different Input Resolutions

    Input images do not have to be strictly limited to a fixed size; the model can adapt to different aspect ratios through resizing, padding, and dynamic token budgets, giving it a broader range of applications.

    4. Better Suited to Unified Image and Video Processing

    Three-dimensional grid position encoding and variable-length sequence handling enable the same visual backbone to process both images and multi-frame video input.

    5. Easy to Connect with Large Language Models

    After projection, visual features align with the language model’s hidden space, allowing them to directly participate in text generation, tool calling, interface operation, and multi-turn reasoning.

    九、Practical Considerations

    For these models, more tokens are not always better. A higher token budget helps recognize small fonts and complex details, but it also brings higher latency and memory usage. Actual deployment should establish a layered strategy:

    • Use a low budget first for a quick judgment;
    • Increase the budget only when reading details is necessary;
    • Use local zoom-in for documents instead of always processing the entire oversized image;
    • Use keyframes combined with local region analysis for video;
    • For interface operation, focus on control regions and regions that change.

    In addition, visual token compression may affect the recognition of very small text, thin line structures, and dense tables. Evaluation should not look only at overall accuracy; it should also focus on fine-grained capabilities such as text recognition, spatial localization, cross-frame tracking, complex chart understanding, and visual question answering.

    Conclusion

    The key to lightweight multimodal models is not just reducing visual parameters, but establishing a balance among visual information quality, token count, alignment effectiveness with language, and inference cost. A compact visual backbone of around 24 layers, 14×14 patch splitting, 2×2 spatial merging, dynamic token budgets, and dimension alignment with the language model together form a scalable visual encoding scheme.

    For developers, understanding this pipeline helps in judging a model’s real-world performance across scenarios: low-resolution tasks can prioritize speed, while high-precision document and code recognition should increase the visual budget; video analysis needs to control the frame count and per-frame tokens; and production deployment should further optimize overall performance with batching, caching, quantization, and memory management. Ultimately, a model’s effectiveness depends not only on the size of the visual backbone, but on whether visual information is fed into the language reasoning system at an appropriate granularity, in the correct positions, and at a controllable cost.