Visual Encoder Design and Token Budget: Building a Lightweight Vision Branch for Multimodal LLMs

    In native multimodal large language models, the visual encoder is the first gate between pixels and language—yet it is often overlooked while teams focus on the size and context length of the main model. Using a typical configuration as a worked example, this article breaks down the design logic of the visual encoder across six areas: why vision must be natively embedded to move from "reading images" to "deciding with images," the trade-offs behind parameters such as 24 layers, a hidden size of 1024, and 2×2 merging, the preprocessing pipeline that maps pixels to vision tokens, what happens inside a single forward pass, the design philosophy behind a small-but-precise vision branch, and a practical engineering and tuning checklist. Readers will learn how token budgeting, compression strategy, and unified input formats shape the cost and quality of vision in multimodal systems.

    Visual Encoder Design and Token Budget: Building a Lightweight Vision Branch for Multimodal LLMs

    In native multimodal large language models (LLMs), the visual encoder is the first gate connecting pixels to language. Many teams focus only on how large the backbone is or how long the context can be, while overlooking the visual branch. The result: a screenshot, a video segment, or a small button in a web page screenshot either loses all detail inside the model or consumes the entire context window in just a few frames.

    Using a typical configuration as an example, this article dissects the design logic of the visual encoder across six dimensions:

    • From “viewing images” to “deciding with images”: why vision must be natively embedded
    • The trade-offs hidden in the parameter table: why 24 layers, a hidden size of 1024, and 2×2 merging
    • The preprocessing pipeline: a deterministic mapping from pixels to vision tokens
    • Inference path analysis: what actually happens in a single forward pass
    • Design philosophy: why the vision branch is “small but right”
    • Engineering practice and a tuning checklist

    1. From “Viewing Images” to “Deciding with Images”: Why Vision Must Be Native

    1.1 Vision Is More Than Reading Images—It Expands the Boundary of What Can Be Encoded

    The official description of this vision branch emphasizes that visual encoding does not merely process images; it expands the boundary of what “programming” can reach. Rendered results, error screenshots, design mockups, terminal screens, and web layouts—information that originally could not be expressed as code—can all become part of the model’s reasoning.

    This means the vision module faces far more than “describing a photo.” It must:

    • Identify the position, hierarchy, and relative relationships of UI elements;
    • Judge whether a layout is aligned or whether there is overflow or occlusion;
    • Convert a design mockup into a structured page description;
    • Check a screenshot against a generated result to verify whether it matches expectations.

    Such tasks are far more sensitive to spatial detail than everyday image classification.

    1.2 Making Vision “Native”: Letting the Model Decide When to Look

    The same description notes that vision needs to be natively integrated into the model, letting the model decide autonomously when to observe and use visual feedback to guide its next actions.

    You can think of this as a closed loop of “observe—decide—act—observe again.” After generating code, the model judges for itself whether it needs to “take a look” at the running result; once it has the visual feedback, it decides what to change. This loop places three hard requirements on the vision pipeline:

    1. Low latency: visual feedback must return to the decision step as quickly as possible. The lighter the encoder, the smoother the interaction rhythm.
    2. Controllable token consumption: observation is “triggered on demand.” If a single observation consumes tens of thousands of tokens, the model will quickly be forced to compromise between “not seeing clearly” and “not fitting.”
    3. Alignment with linguistic semantics: image information must eventually be concatenated into the same sequence as text tokens for joint attention. The vision output must be aligned to the same hidden dimension as the language model.

    The “lightweight flagship” approach described above seeks a balance between the “upper bound of capability” and the “cost of a single inference.” The design of the visual encoder is the first place where this balance lands.

    2. The Trade-offs Hidden in the Parameter Table: Why 24 Layers, 1024 Dimensions, and 2×2 Merging

    This vision backbone follows the evolution path from earlier multimodal versions to OCR-specific models, and can be seen as a continuation of the same design paradigm on a lightweight flagship. Its core configuration is as follows (for illustrative reference only):

    Item Value Design intent
    depth (layers) 24 Extracting visual semantics needs sufficient depth; encoding cost grows linearly with the number of tokens
    hidden_size 1024 Feature width of vision tokens; a balanced configuration of depth and width
    attention heads 16 Standard multi-head attention
    per-head dimension 64 (1024 / 16, no GQA) Vision tokens do not need to be retained long-term, so grouped-query attention to compress the KV cache is unnecessary
    patch_size / merge 14 / 2×2 Small patches preserve detail; 2×2 merging reduces the token count to one quarter
    output dimension 4096 Aligned to the language model’s hidden dimension after projection
    backbone parameters ≈ 0.56 B Less than 1% of the entire model
    inference precision Fixed BF16, no FP8 quantization The result of balancing precision against benefit

    2.1 Depth and Width: 24 Layers × 1024 Dimensions

    Visual encoders typically adopt a “deep and narrow” structure: 24 layers deep with a hidden width of 1024, gradually abstracting from shallow edges and textures to deeper object and layout semantics.

    • Depth: the number of layers determines the level of semantic abstraction and is critical for recognizing small text, thin lines, icons, and other details. Encoding cost grows linearly with the number of vision tokens, so simply stacking depth raises the fixed overhead. A moderate depth is chosen in exchange for overall cost-effectiveness.
    • Width: 1024 dimensions is a fairly ample representation space for the thousands of tokens compressed from each image, and it also matches the downstream projection to 4096 dimensions.
    • Attention heads: 16 heads × 64 dimensions is a common configuration. Each head focuses on different spatial patterns or semantic cues—text lines, borders, color blocks—and grouped-query attention is not used.

    2.2 Why Not GQA: The Scenario Determines the Structure

    Grouped-query attention (GQA) lets multiple query heads share a set of key-value heads. It solves the problem of large KV-cache memory usage and high read bandwidth during the autoregressive decoding stage.

    A visual encoder, by contrast, processes an entire image in a single forward pass and produces no KV cache that needs to be reused across steps. Dropping GQA brings no memory benefit and may even weaken the representational capacity of vision tokens, so standard multi-head attention is the more reasonable choice.

    2.3 14×14 Patches and 2×2 Merging: A Compromise Between Detail and Token Count

    A patch size of 14×14 is chosen because smaller patches (below 16×16, for instance) preserve more detail but cause the token count to balloon. Applying a further 2×2 merge on top of 14×14 means each vision token covers a 28×28 pixel region. This brings three benefits:

    • Order-of-magnitude token compression: the token count drops to one quarter, directly saving context on the language side;
    • Preserved fine-grained information: compared with using 28×28 or larger patches directly, cutting small and then merging retains richer local texture;
    • Simple alignment: 28 equals exactly 14 × 2, so preprocessing, positional encoding, and merge logic can all be implemented with a single stride rule.

    2.4 A 4096-Dimensional Output: Aligning with the Language Model

    The output of the vision backbone must be mapped to 4096 dimensions through a projection layer to stay consistent with the language model’s hidden states. Only then can it be concatenated directly into the text token sequence and share subsequent attention computation with text, without an extra cross-modal attention module.

    2.5 About 0.56B Parameters: A “Cost-Effective Choice” Under 1%

    The backbone parameters make up less than 1% of the total, a ratio that reflects a clear sense of cost discipline:

    • Visual encoding is a mandatory step for every image input; even if the output has few tokens, the encoding itself carries a fixed overhead;
    • Keeping it lightweight can significantly reduce total inference cost in image-heavy scenarios (such as long documents or frame-by-frame video);
    • A small parameter count also makes it easier to train and iterate continuously on visual data without retraining the entire large model.

    2.6 BF16 Inference: Why Not FP8

    Inference is fixed at BF16 rather than FP8, for three reasons:

    • BF16 has the same exponent width as FP32 and a large numeric range. It can withstand the wide dynamic range and activation outliers common in visual inputs, and visual encoding precision usually determines directly whether text and symbol recognition succeeds;
    • Vision tokens typically number only from hundreds to a few thousand, so the memory and bandwidth benefits from quantization are limited;
    • Inference frameworks already offer mature quantization schemes for FP8, but a quantized vision branch would require additional calibration and precision validation, so the cost does not match the benefit.

    3. The Preprocessing Pipeline: A Deterministic Mapping from Pixels to Vision Tokens

    Before visual information enters the encoder, it must be standardized by the image processor. The goal here is very clear: to convert inputs of any size and aspect ratio into a fixed-length unit sequence that the encoder can process stably, while preserving the original spatial structure as much as possible.

    3.1 Resizing and Zero-Padding

    Images are first scaled by a rule, then zero-padded to a multiple of 28. Here 28 = 14 (patch edge length) × 2 (merge factor), which both ensures no incomplete edges appear when cutting patches and guarantees that each token after merging corresponds exactly to a 28×28 pixel region.

    Take a common 1920×1080 screenshot as an example:

    • Width 1920 ÷ 28 ≈ 68.6, rounded up to 69;
    • Height 1080 ÷ 28 ≈ 38.6, rounded up to 39;
    • After padding, roughly 1932×1092 pixels, giving about 69 × 39 = 2691 tokens.

    The padded regions are zero and contain no valid information. Together with positional encoding and attention masks, the model can distinguish real content from edge padding.

    3.2 Patch Extraction and 2×2 Merging

    The standardized image is cut into 14×14 patches, followed by a 2×2 merge: four adjacent base patches combine into a 28×28 unit, each corresponding to one vision token. The overall process can be described with the pseudocode below (which only illustrates the processing logic):

    1. def image_to_vision_tokens(image, min_tokens=16, max_tokens=8000):
    2. # 1) Resize to a multiple of 28 and zero-pad
    3. # 28 = patch_size(14) * merge_size(2)
    4. stride = patch_size * merge_size
    5. new_h = ceil(image.height / stride) * stride
    6. new_w = ceil(image.width / stride) * stride
    7. image = resize_and_zero_pad(image, new_h, new_w)
    8. # 2) Cut into 14x14 patches
    9. patches = to_patches(image, patch_size=14)
    10. # 3) 2x2 merge: combine every 4 adjacent patches into 1 vision token
    11. tokens = merge_2x2(patches)
    12. # 4) Token budget control
    13. n = tokens.numel()
    14. if n > max_tokens:
    15. # Downsample to the budget cap, prioritizing that the whole image stays visible
    16. tokens = downsample(tokens, max_tokens)
    17. elif n < min_tokens:
    18. # Upscale small images to avoid too little detail
    19. tokens = upsample(tokens, min_tokens)
    20. return tokens # shape approximately (N, 14*14*3)

    3.3 Token Budget Control: A “Scheduler” for Context Resources

    The model uses two parameters, min_image_tokens and max_image_tokens, to control the number of tokens a single image generates, with a default range of 16 to 8000. Its significance lies in:

    • Protecting the context at the upper bound: a single image’s token consumption can range from a few dozen to several thousand. The cap ensures that in long contexts containing multiple conversation turns, code snippets, and tool-call results, an image will not crowd out text space;
    • Guaranteeing detail at the lower bound: an extremely small icon or simple line drawing reduced to just one or two tokens can retain almost no recognizable information;
    • Adaptability to scenarios: tasks such as document understanding, UI analysis, and video frame extraction have very different needs for visual detail, so the budget can be tuned to match a specific context window and task type.

    Roughly speaking, 8000 tokens corresponds to an image on the order of 2500×2500 pixels (about three times the pixel count of a 1920×1080 screenshot). Most screenshots and design drafts fall within this range, so the default cap covers typical inputs without incurring excessive cost.

    4. Inference Path Analysis: What Actually Happens in a Single Forward Pass

    The entry function signature of the inference code (receiving parameters such as hidden_states and grid_thw) shows that the visual encoder’s execution path can be split into four stages.

    4.1 Positional Encoding: Spatial Cues in grid_thw

    The parameter grid_thw describes the grid size of each image (or video frame sequence) across three dimensions:

    • T: the temporal dimension of a video (number of frames), typically 1 for a single image;
    • H: the number of patch grids along the height;
    • W: the number of patch grids along the width.

    After the 3D grid is flattened into a 1D sequence, the model needs to know the original row and column position of each vision token, so it first computes the visual position IDs.

    4.2 Attention Boundaries: Variable-Length Sequences and Block-Diagonal Masks

    A batch of requests often contains images of different sizes, multi-frame videos, or interleaved image-text multimodal inputs. These are concatenated into a single long sequence sent to the encoder, and before computation, the cumulative sequence lengths (cu_seqlens) and maximum sequence length (max_seqlen) must be generated from the grid information.

    These two fields are used to build variable-length attention, which is equivalent to a block-diagonal mask: the vision tokens of each image (or each video segment) can only attend within their own group, and different images do not interfere with each other.

    Without this constraint, content in a long-document image within the same batch could “contaminate” the encoding of another small-icon image. In the multi-image scenarios of vision agents, such errors are especially subtle.

    4.3 Patch Embedding: Turning Pixel Blocks into Feature Vectors

    The patch_embed module projects the extracted patches into hidden vectors. In many implementations, this step is done with a 3D convolution:

    • The temporal kernel size is 1, ensuring frames are encoded independently;
    • The spatial dimensions correspond to the 14×14 patch;
    • The number of output channels equals the hidden width of 1024.

    Choosing a 3D convolution over a 2D one is precisely to support a unified input format for images and videos: an image is simply the special case of T=1, so the entire inference pipeline does not need to maintain two separate code paths for the two modalities.

    4.4 Backbone Encoding and Output Projection

    After the patch embeddings are added to the positional encodings, they pass through 24 layers of visual Transformer in sequence. Because the encoded content is static images, each layer can use bidirectional attention, letting every vision token see the context of the entire image at once.

    The backbone output is then merged and projected to combine adjacent 2×2 tokens and map them to 4096 dimensions, aligning with the language model. The overall flow can be summarized as:

    1. Original image / video frames
    2. │
    3. ▼
    4. Standardization: resize + zero-pad to a multiple of 28
    5. │
    6. ▼
    7. Patch extraction (14×14) → 2×2 merge → vision token sequence + grid_thw
    8. │
    9. ▼
    10. patch_embed (3D convolution) + positional encoding
    11. │
    12. ▼
    13. 24-layer visual Transformer (variable-length attention: visible within an image, isolated between images)
    14. │
    15. ▼
    16. Projected to 4096 dimensions, concatenated with text tokens and sent to the language model

    5. Design Philosophy: Why the Vision Branch Is “Small but Right”

    Three reusable principles lie behind this design.

    5.1 The Encoder “Sees,” the Language Model “Understands”

    The task of the visual encoder is to convert pixels into vector representations that carry spatial semantics, while recognizing the meaning of text, understanding UI logic, and inferring user intent are mainly the job of the language model.

    If the encoder is made to take on too many semantic responsibilities, the model must be scaled up continuously, and its capability boundary begins to overlap with that of the language model. Keeping the encoder lightweight instead encourages the language model to take on more semantic reasoning during joint training.

    5.2 The Compression Ratio Is a Tunable Resource, Not a Fixed Parameter

    The 28×28 token granularity, the 2×2 merge, and the 8000-token cap are all essentially a “price” for how much context an image occupies. Such parameters should be set according to the actual token distribution of the task, rather than copying the default values.

    5.3 A Unified Format Reduces Engineering Complexity

    • Images and videos share the same processing pipeline;
    • Encoder inference precision is decoupled from the backbone;
    • Positional encoding and variable-length attention handle multi-image and multi-frame inputs uniformly.

    A unified format reduces the maintenance cost and error rate caused by maintaining multiple code paths, a point often underestimated in multimodal systems.

    6. Engineering Practice and a Tuning Checklist

    6.1 Memory and Throughput: The Real Bottleneck Is on the Language Side

    The memory overhead of the visual encoder comes mainly from activations, which grow quadratically with resolution. But the real bottleneck usually lies elsewhere:

    • Once vision tokens enter the language model, they participate in the attention computation of all subsequent layers. Every additional 1000 vision tokens means 1000 tokens’ worth of KV cache and attention overhead across all decoding steps;
    • An image processed at the upper limit (about 8000 tokens) has a “footprint” in the context comparable to a technical document of several thousand words;
    • When images and long text are mixed as input, the proportion of vision tokens often exceeds intuition.

    Therefore, controlling the token budget is both a vision-side problem and a language-side inference-cost problem.

    6.2 Input Preprocessing: The Step That Most Directly Affects Results

    • Aspect ratio: extreme long images (such as long screenshots or long documents) waste tokens if padded passively with large blank areas; a more sensible approach is to first split the page into regions semantically, then encode them separately;
    • Clarity: upscaling a low-resolution image cannot “restore” detail and instead introduces interpolation noise; for scenes dense with small text and icons, you should increase the source image resolution rather than increase the token budget;
    • Video input: the cost of frame-by-frame encoding grows linearly with the number of frames, so a “keyframe extraction + frame-level token budget” strategy should be adopted rather than fully encoding every frame.

    6.3 Precision Strategy: Layered Handling

    BF16 inference for the visual encoder is the choice under the current configuration and does not mean quantization will never apply. If vision token counts grow significantly in the future, low-precision options can be evaluated first for the most compute-intensive parts (such as the attention matrix multiplications), with small-sample evaluations to confirm that sensitive tasks like text recognition have not regressed.

    6.4 Monitoring and Observability

    It is recommended to establish routine monitoring for the following metrics (using general-purpose cloud capabilities such as monitoring/alerting and logging services):

    • The distribution of vision-token counts per request (P50 / P95);
    • The proportion of vision tokens in the total context;
    • The share of visual-encoding time in end-to-end latency;
    • The ratio of inputs that are cropped or downsampled.

    These data tell you directly whether the current bottleneck is “not seeing clearly” or “not fitting.”

    6.5 Scenario-Based Tuning Checklist

    Scenario Recommendation Reason
    UI mockup to code Raise the cap appropriately and preserve the original aspect ratio Layout, alignment, and hierarchy depend on spatial precision
    Long-document / financial-report understanding Split by page + moderate budget Control total context while preserving table structure
    Multi-image comparison Set a per-image budget cap and enable inter-image isolation Prevent one large image from squeezing out other images’ context
    Video understanding Keyframe extraction + frame-level budget Cost grows linearly with duration
    General visual Q&A Use the default budget and observe the token distribution Most natural images are already sufficient within the default range

    7. Conclusion

    The visual encoder may seem like just a “small module” in a multimodal LLM, yet it directly determines what the model “sees,” how much it “sees,” and “at what cost” it sees it. From a lightweight backbone of about 0.56B, to the compression strategy of 14×14 patches with 2×2 merging, to a configurable token budget of 16 to 8000 and variable-length attention, every design choice answers the same question: how to pass enough visual information to the language model at the lowest context cost.

    For developers and enterprise users, what is truly worth taking away is not any specific set of parameters, but three general principles:

    1. Treat the token budget as a first-class citizen: the context cost of visual input must be managed explicitly;
    2. Make formats unified and boundaries clear: images, videos, and multi-image inputs share a single inference pipeline;
    3. Let observability drive tuning: use token-distribution and time-share data to guide resolution, budget, and splitting strategies.

    Do these three things, and whether you are deploying a vision agent in the cloud or integrating vision capabilities on the device, you can make “seeing images” both clear and affordable.